Posts /

Installing and running Unity ml-agents on Ubuntu

Back to posts /
HOME POSTS
26 Jan 2019

As soon as I’ve found out about Unity ml-agents toolkit I got quite excited about it, and have been looking forward to checking it out.

It’s not really about reinforcement learning algorithm implementations, but more about the opportunity to use Unity as a tool for rendering and physics simulation. One might argue that there are other alternatives, such as MuJoCo, but Unity is free for personal use (vs 500$ for MuJoCo license), plus it looks like a better choice overall (flexible rendering, scripting capabilities, big community, etc).

Recently I’ve finally gotten to play with it, and I’m going to describe it in (hopefully) few following posts here. This one will be dedicated to the installation of Unity and ml-agents on Ubuntu 18.04, performing training and inference via provided tools and controlling the agents using external code.

Installing Unity

Let’s try to follow steps in Installation.md, which suggests to download Unity from https://store.unity.com/download page. At the time of writing, this page says Unity is available for Windows or Mac OS X, which is quite interesting - I was assuming that a chunk of ml-agents target audience would be using other OS :)

Anyway, turns out that Linux version is still in beta (even though apparently it was released on 26 Aug 2015), and should be downloaded from the Unity forum.

It wasn’t obvious, and I found out about it from this medium post, so credit for discovery goes to Tijmen Verhulsdonck. In fact, after finding this post, I’ve switched to following instructions from it, instead of Installation.md document. But turns out ml-agents is evolving quite fast, and some of the instructions from this guide are obsolete as well. So I will try to give as much detail as I can about the issues that I’ve had and solutions that I’ve ended up with, so it will hopefully save some time if you’re trying to do the same thing. Anyway, let’s get back to action.

On the Unity forum, go to the last page and download the installer from the link in the last post (which is to the version 2018.3.0f2 at the time of writing). It seems that installation via Unity Hub is now the preferred way, but it doesn’t seem to add any additional value, so I’ll go with a direct link to the installer.

After downloading, allow executing the installer and launch it:

alexey@laptop:~/Downloads$ chmod +x UnitySetup-2018.3.0f2 
alexey@laptop:~/Downloads$ ./UnitySetup-2018.3.0f2

During installation, accept the agreement, and choose the components you want to install (I selected only “Unity 2018.3.0f2”) and where to install them. It will download approximately a gig, and take 3.23 GB of your storage space. After download and installation are finished you might launch Unity to make sure it’s operational (but it’s not required yet).

Installing ml-agents

Both guides I’ve referenced suggest installing ml-agents from source (provided as part of ml-agents GitHub repo), but since I’m not going to mess with the code, and want to have a particular version to make this guide reproducible, I was going to simply install it via pip (pip install mlagents==0.6.0).

Turns out that later we’ll need some of the assets, that are included in this repo (mostly UnitySDK dir), so it makes more sense to install from source then. I guess there’s some benefit in this after all - version of ml-agents package will be synced with versions of gym-unity package (which we might or might not use later) and assets in UnitySDK folder.

It’s a good time to create a virtualenv for this project, I’m going to demonstrate that as well:

alexey@laptop:~$ mkvirtualenv unity-rl
Using base prefix '/usr'
New python executable in /home/alexey/.virtualenvs/unity-rl/bin/python3
Also creating executable in /home/alexey/.virtualenvs/unity-rl/bin/python
Installing setuptools, pip, wheel...
done.
virtualenvwrapper.user_scripts creating /home/alexey/.virtualenvs/unity-rl/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/alexey/.virtualenvs/unity-rl/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/alexey/.virtualenvs/unity-rl/bin/preactivate
virtualenvwrapper.user_scripts creating /home/alexey/.virtualenvs/unity-rl/bin/postactivate
virtualenvwrapper.user_scripts creating /home/alexey/.virtualenvs/unity-rl/bin/get_env_details

Some sanity checking - let’s make sure that default Python is the one we want to use (3.6 is the only one supported by ml-agents currently), and that none of the packages are installed yet:

(unity-rl) alexey@laptop:~$ python --version
Python 3.6.7
(unity-rl) alexey@laptop:~$ pip list
Package    Version
---------- -------
pip        19.0.1 
setuptools 40.6.3 
wheel      0.32.3

Now let’s clone the repo with python packages and assets that we need, and check out the version that we’ll use (0.6.0):

(unity-rl) alexey@laptop:~/Software$ git clone https://github.com/Unity-Technologies/ml-agents.git
...
(unity-rl) alexey@laptop:~/Software$ cd ml-agents
(unity-rl) alexey@laptop:~/Software/ml-agents$ git checkout 0.6.0
Note: checking out '0.6.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at cb0bfa0 Merge pull request #1494 from Unity-Technologies/release-v0.6

Time to install the ml-agents python package:

(unity-rl) alexey@laptop:~/Software/ml-agents$ cd ml-agents/
(unity-rl) alexey@laptop:~/Software/ml-agents/ml-agents$ pip install -e .
...

This will download and install a bunch of dependencies as well, hopefully successfully. Let’s see if we got lucky:

(unity-rl) alexey@laptop:~$ mlagents-learn --help

    
                        ▄▄▄▓▓▓▓
                   ╓▓▓▓▓▓▓█▓▓▓▓▓
              ,▄▄▄m▀▀▀'  ,▓▓▓▀▓▓▄                           ▓▓▓  ▓▓▌
            ▄▓▓▓▀'      ▄▓▓▀  ▓▓▓      ▄▄     ▄▄ ,▄▄ ▄▄▄▄   ,▄▄ ▄▓▓▌▄ ▄▄▄    ,▄▄
          ▄▓▓▓▀        ▄▓▓▀   ▐▓▓▌     ▓▓▌   ▐▓▓ ▐▓▓▓▀▀▀▓▓▌ ▓▓▓ ▀▓▓▌▀ ^▓▓▌  ╒▓▓▌
        ▄▓▓▓▓▓▄▄▄▄▄▄▄▄▓▓▓      ▓▀      ▓▓▌   ▐▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▌   ▐▓▓▄ ▓▓▌
        ▀▓▓▓▓▀▀▀▀▀▀▀▀▀▀▓▓▄     ▓▓      ▓▓▌   ▐▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▌    ▐▓▓▐▓▓
          ^█▓▓▓        ▀▓▓▄   ▐▓▓▌     ▓▓▓▓▄▓▓▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▓▄    ▓▓▓▓`
            '▀▓▓▓▄      ^▓▓▓  ▓▓▓       └▀▀▀▀ ▀▀ ^▀▀    `▀▀ `▀▀   '▀▀    ▐▓▓▌
               ▀▀▀▀▓▄▄▄   ▓▓▓▓▓▓,                                      ▓▓▓▓▀
                   `▀█▓▓▓▓▓▓▓▓▓▌
                        ¬`▀▀▀█▓

        
    Usage:
      mlagents-learn <trainer-config-path> [options]
      mlagents-learn --help

    Options:
      --env=<file>               Name of the Unity executable [default: None].
      --curriculum=<directory>   Curriculum json directory for environment [default: None].
      --keep-checkpoints=<n>     How many model checkpoints to keep [default: 5].
      --lesson=<n>               Start learning from this lesson [default: 0].
      --load                     Whether to load the model or randomly initialize [default: False].
      --run-id=<path>            The directory name for model and summary statistics [default: ppo].
      --num-runs=<n>             Number of concurrent training sessions [default: 1]. 
      --save-freq=<n>            Frequency at which to save model [default: 50000].
      --seed=<n>                 Random seed used for training [default: -1].
      --slow                     Whether to run the game at training speed [default: False].
      --train                    Whether to train model, or only run inference [default: False].
      --worker-id=<n>            Number to add to communication port (5005) [default: 0].
      --docker-target-name=<dt>  Docker volume to store training-specific files [default: None].
      --no-graphics              Whether to run the environment in no-graphics mode [default: False].

Nice!

Running 3D Ball environment

Ok, time to finally run Unity and try training/running an agent. For this purpose, we’ll refer to Basic-Guide.md from official docs. Here are my notes regarding following the steps in this guide:

Setting up the ML-Agents Toolkit within Unity

Setting up TensorFlowSharp
For purposes of future posts, downloading and installing TensorFlowSharp plugin is not really necessary (it is used to run trained agent without external control), but I still will do it, since it helps to check if everything is working properly.

Congrats! You’ve been able to run a pre-trained TensorFlow model in Unity. Make sure that you can follow section Training the environment as well - you should be able to enable external control of your agents to do anything useful. It worked out of the box for me so I won’t give any additional notes here.

Controlling the agent(s) externally

Speaking of external control - let’s try a bare-bones version of it. Basic-Guide.md has a Using the Basics Jupyter Notebook section, which almost works for me (more on that later).

Since we’ve installed all the packages into a virtualenv, we need to create a “kernel” to make their imports in the notebook work. You need to do this while your virtualenv is activated:

(unity-rl) alexey@laptop:~/Software/ml-agents$ python -m ipykernel install --user --name unity-rl --display-name "unity-rl"
Installed kernelspec unity-rl in /home/alexey/.local/share/jupyter/kernels/unity-rl

Now we can start jupyter:

(unity-rl) alexey@laptop:~/Software/ml-agents$ cd notebooks/
(unity-rl) alexey@laptop:~/Software/ml-agents/notebooks$ jupyter notebook
...

After you open getting-started.ipynb select unity-rl in Kernel -> Change kernel - you can think of is as a jupyter equivalent of workon unity-rl. Now you can execute the cells.

Don’t forget to change env_name from "../envs/3DBall" to None in first code cell - that’ll run the environment, which is currently open in Unity, instead of a compiled binary, so you have to make sure Unity is running and 3DBall scene is open there. After you run the cell which creates a UnityEnvironment you will be prompted to press “Run” in Unity - this sets up the connection between the controller script and environment. As you run other cells, you will get information about environment, run it for few episodes and close the connection.

This is the way I’m going to control the agents, during both training and inferencing. But having to click “Run” manually is not acceptable (harder to automate, requires Unity Editor running, can’t run in headless mode, etc), so we have to “build” the environments. Unfortunately, this is where I get stuck - the building process completes just fine, but when trying to run produced executables I get SIGABRT in native code, which seems to be related to TensorFlow.

After spending some time trying to find a workaround, I’ve realized that if I plan to always control the agents externally - I don’t really need TensorFlow, and can try to build the environment without it. And luckily it works! To do that simply remove ENABLE_TENSORFLOW from Project Settings -> Player -> Scripting Define Symbols, rebuild the environment and - voilà! You can now replace env_name back to "../envs/3DBall" in the notebook and run it without having to click the button and keep Unity running in background.

I guess that’s enough for this post, and we’ll try to do something interesting with this setup in one of the following ones.


Twitter Facebook Google+