Explore Unity Technologies ML-Agents* Exclusively on Intel® Architecture

Abstract

This article describes how to install and run Unity Technologies ML-Agents* in CPU-only environments. It demonstrates how to:

• Train and run the ML-Agents Balance Balls example on Windows* without CUDA* and cuDNN*.
• Perform a TensorFlow* CMake build on Windows optimized for Intel® Advanced Vector Extensions 2 (Intel® AVX2).
• Create a simple Amazon Web Services* (AWS) Ubuntu* Amazon Machine Image* environment from scratch without CUDA and cuDNN, build a “headless” version of Balance Balls for Linux*, and train it on AWS.

Introduction

Unity Technologies released their beta version of Machine Learning Agents* (ML-Agents*) in September 2017, offering an exciting introduction to reinforcement learning using their 3D game engine. According to Unity’s introductory blog, this open SDK will potentially benefit academic researchers, industry researchers interested in “training regimes for robotics, autonomous vehicle, and other industrial applications,” and game developers.

Unity’s ML-Agents SDK leverages TensorFlow* as the machine learning framework for training agents using a Proximal Policy Optimization (PPO) algorithm. There are several example projects included in the GitHub* download, as well as a Getting Started example and documentation on how to install and use the SDK.

One downside of the SDK for some developers is the implied dependencies on CUDA* and cuDNN* to get the ML-Agents environment up and running. As it turns out, it is possible to not only explore ML-Agents exclusively on a CPU, but also perform a custom build of TensorFlow on a Windows® 10 computer to include optimizations for Intel® architecture.

In this article we show you how to:

• Train and run the ML-Agents Balance Balls (see Figure 1) example on Windows without CUDA and cuDNN.
• Perform a TensorFlow CMake build on Windows* optimized for Intel® Advanced Vector Extensions 2 (Intel® AVX2).
• Create a simple Amazon Web Services* (AWS) Ubuntu* Amazon Machine Image* (AMI) environment from scratch without CUDA and cuDNN, build a “headless” version of Balance Balls for Linux*, and train it on AWS.

Figure 1. Trained Balance Balls model running in Unity* software.

Target Audience

This article is intended for developers who have had some exposure to TensorFlow, Unity software, Python*, AWS, and machine learning concepts.

System Configurations

The following system configurations were used in the preparation of this article:

Windows Workstation

• Intel® Xeon® processor E3-1240 v5
• Microsoft Windows 10, version 1709

Linux Server (Training)

• Intel® Xeon® Platinum 8180 processor @ 2.50 GHz
• Ubuntu Server 16.04 LTS

AWS Cloud (Training)

• Intel® Xeon® processor
• Ubuntu Server 16.04 LTS AMI

In the section on training ML-Agents in the cloud we use a free-tier Ubuntu Server 16.04 AMI.

Install Common Windows Components

This section describes the installation of common software components required to get the ML-Agents environment up and running. The Unity ML-Agents documentation contains an Installation and Setup procedure that links to a webpage instructing the user to install CUDA and cuDNN. Although this is fine if your system already has a graphics processing unit (GPU) card that is compatible with CUDA and you don’t mind the extra effort, it is not a requirement. Either way, we encourage you to review the Unity ML-Agents documentation before proceeding. 

There are essentially three steps required to install the common software components:

1. Download and install Unity 2017.1 or later from the package located here.
2. Download the ML-Agents SDK from GitHub. Extract the files and move them to a project folder of your choice (for example, C:\ml-agents).
3. Download and install the Anaconda* distribution for Python 3.6 version for Windows, located here.

Install Prebuilt TensorFlow*

This section follows the guidelines for installing TensorFlow on Windows with CPU support only. According to the TensorFlow website, “this version of TensorFlow is typically much easier to install (typically, in 5 or 10 minutes), so even if you have an NVIDIA* GPU, we recommend installing this version first.” Follow these steps to install prebuilt TensorFlow on your Windows 10 system:

1. In the Start menu, click the Anaconda Prompt icon (see Figure 2) to open a new terminal.

   
   Figure 2. Windows* Start menu.

2. Type the following commands at the prompt:

   > conda create -n tensorflow-cpu python=3.5
> activate tensorflow-cpu
> pip install --ignore-installed --upgrade tensorflow

3. As specified in the TensorFlow documentation, ensure the installation worked correctly by starting Python and typing the following commands:

   > python
>>> import tensorflow as tf
>>> hello = tf.constant('Hello')
>>> sess = tf.Session()
>>> print (sess.run(hello))

4. If everything worked correctly, 'Hello' should print to the terminal as shown in Figure 3.

   
   Figure 3. Python* test output.

   You may also notice a message like the one shown in Figure 3, stating “Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2.” This message may vary depending on the Intel® processor in your system; it indicates TensorFlow could run faster on your computer if you build it from sources, which we will do in the next section.

5. To close Python, at the prompt, press CTRL+Z.
    
6. Navigate to the python subdirectory of the ML-Agents repository you downloaded earlier, and then run the following command to install the other required dependencies:

   > pip install.

7. Refer to the Building Unity Environment section of the “Getting Started with Balance Ball Example” tutorial to complete the ML-Agents tutorial.

Install TensorFlow from Sources

This section describes how to build an optimized version of TensorFlow on your Windows 10 system.

The TensorFlow website states, “We don't officially support building TensorFlow on Windows; however, you may try to build TensorFlow on Windows if you don't mind using the highly experimental Bazel on Windows or TensorFlow CMake build.” However, don’t let this discourage you. In this section we provide instructions on how to perform a CMake build on your Windows system.

The following TensorFlow build guidelines complement the Step-by-step Windows build instructions shown on GitHub. To get a more complete understanding of the build process, we encourage you to review the GitHub documentation before continuing. 

1. Install Microsoft Visual Studio* 2015. Be sure to check the programming options as shown in Figure 4.

   
   Figure 4. Visual Studio* programming options.

2. Download and install Git from here. Accept all default settings for the installation.
    
3. Download and extract swigwin from here. Change folders to C:\swigwin-3.0.12 (note that the version number may be different on your system).
    
4. Download and install CMake version 3.6 from here. During the installation, be sure to check the option Add CMake to the system path for all users.
    
5. In the Start menu, click the Anaconda Prompt icon (see Figure 2) to open a new terminal. Type the following commands at the prompt:

   > conda create -n tensorflow-custom36 python=3.6
> activate tensorflow-custom36

6. Run the following command to set up the environment:

   > "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"

   (Note: If vcvarsall.bat is not found, try following the instructions provided here.)
    
7. Clone the TensorFlow repository and create a working directory for your build:

   cd /
> git clone https://github.com/tensorflow/tensorflow.git
> cd tensorflow\tensorflow\contrib\cmake
> mkdir build
> cd build

8. Type the following commands (Note: Be sure to check the paths and library version shown below on your own system, as they may be different):

   > cmake .. -A x64 -DCMAKE_BUILD_TYPE=Release ^
-DSWIG_EXECUTABLE=C:\swigwin-3.0.12/swig.exe ^
-DPYTHON_EXECUTABLE=C:/Users/%USERNAME%/Anaconda3/python.exe ^
-DPYTHON_LIBRARIES=C:/Users/%USERNAME%/Anaconda3/libs/python36.lib ^
-Dtensorflow_WIN_CPU_SIMD_OPTIONS=/arch:AVX2

9. Build the pip package, which will be created as a .whl file in the directory .\tf_python\dist (for example, C:\tensorflow\tensorflow\contrib\cmake\build\tf_python\dist\tensorflow-1.4.0-cp36-cp36m-win_amd64.whl).

   > C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild /p:Configuration=Release tf_python_build_pip_package.vcxproj

   (Note: Be sure to check the path to MSBuild on your own system as it may be different.)
    
10. Install the newly created TensorFlow build by typing the following command:

    pip install C:\tensorflow\tensorflow\contrib\cmake\build\tf_python\dist\tensorflow-1.4.0-cp36-cp36m-win_amd64.whl

11. As specified in the TensorFlow documentation, ensure the installation worked correctly by starting Python and typing the following commands:

    > python
>>> import tensorflow as tf
>>> hello = tf.constant('Hello')
>>> sess = tf.Session()
>>> print (sess.run(hello))

12. If everything worked correctly, 'Hello' should print to the terminal. Also, we should not see any build optimization warnings like we saw in the previous section (see Figure 5).

    
    Figure 5. Python* test output.

13. To close Python, at the prompt, press CTRL+Z.
     
14. Navigate to the python subdirectory of the ML-Agents repository you downloaded earlier, and then run the following command to install the other required dependencies:

    > pip install .

15. Refer to the Building Unity Environment section of the “Getting Started with Balance Ball Example” tutorial to complete the ML-Agents tutorial.

Train ML-Agents in the Cloud

The ML-Agents documentation provides a guide titled “Training on Amazon–Web Service” that contains instructions for setting up an EC2 instance on AWS for training ML-Agents. Although this guide states, “you will need an EC2 instance which contains the latest Nvidia* drivers, CUDA8, and cuDNN,” there is a simpler way to do cloud-based training without the GPU overhead.

In this section we perform the following steps:

• Create an Ubuntu Server 16.04 AMI (free tier).
• Install prerequisite applications on Windows for interacting with the cloud server.
• Install Python and TensorFlow on the AMI.
• Build a headless Linux version of the Balance Balls application on Windows.
• Export the Python code in the PPO.ipynb Jupyter Notebook* to run as a stand-alone script in the Linux environment.
• Copy the python directory from Windows to the Linux AMI.
• Run a training session on AWS for the ML-Agents Balance Balls application.

1. Create an account on AWS if you don’t already have one. You can follow the steps shown in this section with an AWS Free Tier account; however, we do not cover every detail of creating an account and configuring an AMI, because the website contains detailed information on how to do this.
2. Create an Ubuntu Server 16.04 AMI. Figure 6 shows the machine instance we used for preparing this article.

   
   Figure 6. Linux* Server 16.04 LTS Amazon Machine Image*.

3. Install PuTTY* and WinSCP* on your Windows workstation. Detailed instructions and links for installing these components, connecting to your Linux instance from Windows using PuTTY, and transferring files to your Linux instance using WinSCP are provided on the AWS website.
    
4. Log in to the Linux Server AMI using PuTTY, and then type the following commands to install Python and TensorFlow:

   > sudo apt-get update
> sudo apt-get install python3-pip python3-dev
> pip3 install tensorflow
> pip3 install image

   Note: The next steps assume you have already completed the ML-Agents Getting Started with Balance Ball Example tutorial. If not, be sure to complete these instructions and verify you can successfully train and run a model on your local Windows workstation before proceeding.
    
5. Ensure your Unity software installation includes Linux Build Support. You need to explicitly specify this option during installation, or you can add it to an existing installation by running the Unity Download Assistant as shown in Figure 7.

   
   Figure 7. Unity* software Linux* build support.

6. In Unity software, open File – Build Settings and make the following selections:
   • Target Platform: Linux
   • Architecture: x86_64
   • Headless Mode: Checked
7. These settings are shown in Figure 8.

   
   Figure 8. Unity* software build settings for headless Linux operation.

8. After clicking Build, create a unique name for the application and save it in the repository’s python folder (see Figure 9). In our example we named it Ball3DHeadless.x86_64 and will refer to it as such for the remainder of this article.

   
   Figure 9. Build Linux* application.

9. In order to run through a complete training session on the Linux AMI we will export the Python code in the PPO.ipynb Jupyter Notebook so it can run as a stand-alone script in the Linux environment. To do this, follow these steps:

   - In the Start menu, to open a new terminal, click the Anaconda Prompt icon (Figure 2).
   - Navigate to the python folder, and then type Jupyter Notebook on the command line.
   - Open the PPO.ipynb notebook, and then click File – Download As – Python (.py). This will save a new file named “ppo.py” in the Downloads folder of your Windows computer.
   - Change the filename to “ppo-test.py” and then copy it to the python folder in your ML-Agents repository.
   - Open ppo-test.py in a text editor, and then change the env_name variable to “Ball3DHeadless”:
   - env_name = “Ball3DHeadless” # Name of the training environment file.
   - Save ppo-test.py, and then continue to the next step.

10. Once the application has been built for the Linux environment and the test script has been generated, use WinSCP to copy the python folder from your ML-Agents repository to the Ubuntu AMI. (Details on transferring files to your Linux instance using WinSCP are provided on the AWS website.)
11. In the PuTTY console, navigate to the python folder and run the following commands:

    > cd python
> chmod +x Ball3DHeadless.x86_64
> python3 ppo-test.py
If everything went well you should see the training session start up as shown in Figure 10.

    
    Figure 10. Training session running on an Amazon Web Services* Linux* instance.

Summary

In the output shown in Figure 10, notice that the time (in seconds) is printed to the console after every model save. Code was added to the ppo-test.py script for this article in order to get a rough measure of the training time between model saves.

To instrument the code we made the following modifications to the Python script:

import numpy as np
import os
import tensorflow as tf
import time # New Code
.
.
.
trainer = Trainer(ppo_model, sess, info, is_continuous, use_observations, use_states)
timer_start = time.clock() # New Code
.
.
.
Save_model(sess, model_path=model_path, steps=steps, saver=saver)
print(“ %s seconds “ % (time.clock() – timer_start)) # New Code
timer_start = time.clock() # New Code
.
.
.

Using this informal performance metric, we found that the average difference in training time between a prebuilt TensorFlow GPU binary and prebuilt CPU-only binary on the Windows workstation was negligible. The training time for the custom CPU-only TensorFlow build was roughly 19 percent faster than the prebuilt CPU-only binary on the Windows workstation. When training was performed in the cloud, the AWS Ubuntu Server AMI performed roughly 29 percent faster than the custom TensorFlow build on Windows.