Compiling Carla in Docker (#1841)

* Compiling Carla in a Docker

* Big improvement in Carla Docker build time

* Fixed Docker dependencies & added docker_importer

* Fixed Docker package exporting pipeline

* Fixed docs issues

* Fixed few issues from revision

* Merge branch 'master' into marcgpuig/docker

.travis.yml

@@ -70,9 +70,10 @@ matrix:
         - pip3 install -q --user -r PythonAPI/examples/requirements.txt
         - pip3 install -q --user -r PythonAPI/test/requirements.txt
         - pip3 install -q --user -r PythonAPI/util/requirements.txt
+        - pip3 install -q --user -r Util/Docker/requirements.txt
       script:
         - shopt -s globstar
-        - pylint --disable=R,C --rcfile=PythonAPI/.pylintrc PythonAPI/**/*.py
+        - pylint --disable=R,C --rcfile=PythonAPI/.pylintrc PythonAPI/**/*.py Util/Docker/*.py
 
     - env: TEST="MkDocs"
       install:

Docs/how_to_build_on_linux.md

@@ -6,11 +6,13 @@
 Install the build tools and dependencies
 
 ```
+sudo apt-get update
+sudo apt-get install wget software-properties-common
 sudo add-apt-repository ppa:ubuntu-toolchain-r/test
 wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
 sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-7 main"
 sudo apt-get update
-sudo apt-get install build-essential clang-7 lld-7 g++-7 cmake ninja-build libvulkan1 python python-pip python-dev python3-dev python3-pip libpng16-dev libtiff5-dev libjpeg-dev tzdata sed curl wget unzip autoconf libtool
+sudo apt-get install build-essential clang-7 lld-7 g++-7 cmake ninja-build libvulkan1 python python-pip python-dev python3-dev python3-pip libpng16-dev libtiff5-dev libjpeg-dev tzdata sed curl unzip autoconf libtool rsync
 pip2 install --user setuptools
 pip3 install --user setuptools
 ```
@@ -65,6 +67,10 @@ Now you need to download the assets package, to do so we provide a handy script
 that downloads and extracts the latest version (note that this package is >3GB,
 this step might take some time depending on your connection)
 
+!!! Tip
+    Optionally you can download aria2 (with `sudo apt-get install aria2`) so 
+    the following command will take advantage of it and will run quite faster.
+
 ```sh
 ./Update.sh
 ```

Util/Docker/Carla.Dockerfile

@@ -0,0 +1,16 @@
+FROM carla-prerequisites:latest
+
+ARG GIT_BRANCH
+
+USER ue4
+
+RUN cd /home/ue4 && \
+  if [ -z ${GIT_BRANCH+x} ]; then git clone --depth 1 https://github.com/carla-simulator/carla.git; \
+  else git clone --depth 1 --branch $GIT_BRANCH https://github.com/carla-simulator/carla.git; fi && \
+  cd /home/ue4/carla && \
+  ./Update.sh && \
+  make CarlaUE4Editor && \
+  make package && \
+  rm -r /home/ue4/carla/Dist
+
+WORKDIR /home/ue4/carla

Util/Docker/Prerequisites.Dockerfile

@@ -0,0 +1,49 @@
+ARG UE4_V=4.22.2
+FROM adamrehn/ue4-source:${UE4_V}-opengl
+
+USER root
+
+ENV UE4_ROOT /home/ue4/UnrealEngine
+
+RUN apt-get update ; \
+  apt-get install -y wget software-properties-common && \
+  add-apt-repository ppa:ubuntu-toolchain-r/test && \
+  wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|apt-key add - && \
+  apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-7 main" && \
+  apt-get update ; \
+  apt-get install -y build-essential \
+    clang-7 \
+    lld-7 \
+    g++-7 \
+    cmake \
+    ninja-build \
+    libvulkan1 \
+    python \
+    python-pip \
+    python-dev \
+    python3-dev \
+    python3-pip \
+    libpng-dev \
+    libtiff5-dev \
+    libjpeg-dev \
+    tzdata \
+    sed \
+    curl \
+    unzip \
+    autoconf \
+    libtool \
+    rsync \
+    aria2 && \
+  pip2 install setuptools && \
+  pip3 install setuptools && \
+  update-alternatives --install /usr/bin/clang++ clang++ /usr/lib/llvm-7/bin/clang++ 170 && \
+  update-alternatives --install /usr/bin/clang clang /usr/lib/llvm-7/bin/clang 170
+
+USER ue4
+
+RUN cd $UE4_ROOT && \
+  ./Setup.sh && \
+  ./GenerateProjectFiles.sh && \
+  make
+
+WORKDIR /home/ue4

Util/Docker/README.md

@@ -0,0 +1,104 @@
+# Building Carla in a Docker
+
+_These instructions have been tested in **Ubuntu 16.04**._
+
+This file is intended to explain how to build a Docker image that uses **Ubuntu 18.04** to compile Carla.
+
+Since this building process is based on [**ue4-docker**](https://github.com/adamrehn/ue4-docker) project, it is recommended to take a look at their [documentation](https://adamrehn.com/docs/ue4-docker/read-these-first/introduction-to-ue4-docker).
+
+## Important information
+
+**Building these Docker images can be high time and disk space consuming (~4 hours and up to 300GB on Linux and 500GB on Windows in the build process).**
+
+More information about large containers can be found [here](https://adamrehn.com/docs/ue4-docker/read-these-first/large-container-images-primer).
+
+**The Docker images produced by the ue4-docker Python package contain the UE4 Engine Tools in both source code and object code form. As per Section 1A of the [Unreal Engine EULA](https://www.unrealengine.com/en-US/eula), Engine Licensees are prohibited from public distribution of the Engine Tools unless such distribution takes place via the Unreal Marketplace or a fork of the Epic Games UE4 GitHub repository. Public distribution of the built images via an openly accessible Docker Registry (e.g. Docker Hub) is a direct violation of the license terms. It is your responsibility to ensure that any private distribution to other Engine Licensees (such as via an organization's internal Docker Registry) complies with the terms of the Unreal Engine EULA.**  
+
+For more details, see the [Unreal Engine EULA Restrictions](https://unrealcontainers.com/docs/obtaining-images/eula-restrictions) page on the [Unreal Containers community hub](https://unrealcontainers.com/).
+
+## Requirements
+
+```
+- 64-bit version of Docker in Ubuntu 16.04+.
+- Minimum 8GB of RAM
+- Minimum 300GB available disk space for building container images
+```
+
+---
+
+## Prerequisites
+
+You need Docker installed and configured so your Docker images can access to the Internet during the build process.
+
+Make sure you have installed **Python 3.6 or newer**, check that is in the path, and is callable using `python3` in your terminal. One possible way to achieve so is by using [virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/).
+
+## Dependencies
+
+As mentioned before, in order to use Unreal Engine inside Docker we use [ue4-docker](https://github.com/adamrehn/ue4-docker). You can install it using `pip3`.  
+Further information on installing ue4-docker on Linux can be found [here](https://adamrehn.com/docs/ue4-docker/configuration/configuring-linux).
+
+```
+pip3 install ue4-docker
+```
+
+You can use ue4-docker to automatically configure your Linux firewall:
+
+```
+ue4-docker setup
+```
+
+## Building the Docker images
+
+Navigate to `carla/Util/Docker` and use the following commands, each one will take a long time.  
+First, let's create a Docker image containing a compiled version of Unreal Engine 4 version `22.2`. Change the version if needed.
+
+```
+ue4-docker build 4.22.2 --no-engine --no-minimal
+```
+
+Next, this will build the image with all the necessary requisites to build Carla in a **Ubuntu 18.04**
+
+```
+docker build -t carla-prerequisites -f Prerequisites.Dockerfile .
+```
+
+Finally create the actual Carla image, it will search for `carla-prerequisites:latest`:
+
+```
+docker build -t carla -f Carla.Dockerfile .
+```
+
+---
+
+## Other useful information
+
+You can use a specific repository **branch** or **tag** from our repository, using:
+
+```
+docker build -t carla -f Carla.Dockerfile . --build-arg GIT_BRANCH=branch_name
+```
+
+Clean up the intermediate images from the build (keep the ue4-source image so you can use it for full rebuilds)
+
+```
+ue4-docker clean
+```
+
+## Using the Docker tools
+
+The `docker_tools.py` (in `/carla/Util/Docker`) is an example of how you can take advantages of these Docker images. It uses [docker-py](https://github.com/docker/docker-py) whose documentation can be found [here](https://docker-py.readthedocs.io/en/stable/).  
+The code is really simple and can be easily expanded to interact with docker in other ways.
+
+You can create a Carla package (distribution) from the Docker image using:
+
+```
+./docker_tools.py --output /output/path
+```
+
+Or you can use it to cook assets (like new maps and meshes), ready to be consumed by a Carla package (distribution):
+
+```
+./docker_tools.py --input /assets/to/import/path --output /output/path --packages PkgeName1,PkgeName2
+```
+
+The needed files and hierarchy to import assets is explained [here](https://carla.readthedocs.io/en/latest/export_import_dist/).

Util/Docker/docker_tools.py

@@ -0,0 +1,146 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2019 Computer Vision Center (CVC) at the Universitat Autonoma
+# de Barcelona (UAB).
+#
+# This work is licensed under the terms of the MIT license.
+# For a copy, see <https://opensource.org/licenses/MIT>.
+
+"""
+Helper script to generate consumables for Carla in Docker
+"""
+
+from __future__ import print_function
+
+import argparse
+import docker
+import docker_utils
+import os
+
+
+def print_formated_dict(dic):
+    for k, v in dic.items():
+        print(' - "' + str(k) + '"' + ": " + str(v))
+    print()
+
+
+def bold(text):
+    return ''.join([docker_utils.BOLD, text, docker_utils.ENDC])
+
+
+def bold_underline(text):
+    return ''.join([docker_utils.UNDERLINE, bold(text)])
+
+
+def parse_args():
+    argparser = argparse.ArgumentParser(
+        description=__doc__)
+    argparser.add_argument(
+        '-i', '--input',
+        type=str,
+        help='Path of all the assets to convert')
+    argparser.add_argument(
+        '-o', '--output',
+        type=str,
+        help='Path where all the assets will be after conversion. Default: current directory.')
+    argparser.add_argument(
+        '--packages',
+        type=str,
+        help='(Optional) Packages to generate. Usage: "--packages=PkgeName1,PkgeName2"')
+    argparser.add_argument(
+        '-v', '--verbose',
+        action='store_true',
+        default=False,
+        help='Prints extra information')
+    argparser.add_argument(
+        '--image',
+        type=str,
+        help='Use a specific Carla image. Default: "carla:latest"')
+    args = argparser.parse_args()
+
+    if not args.output:
+        args.output = os.getcwd()
+
+    if args.packages is not None and args.packages and not args.input:
+        print(
+            docker_utils.RED +
+            "[Error] The Input Path [-i|--input] must be specified "
+            "if you are processing individual packages." + docker_utils.ENDC)
+        exit(1)
+
+    print()
+
+    print(bold("- ") + bold_underline("Params:"))
+    print(" - Output path: " + str(args.output))
+    print(" - Packages:    " + str(args.packages))
+    print(" - Input path:  " + str(args.input))
+    print(" - Verbose:     " + str(args.verbose))
+    print()
+
+    return args
+
+
+def main():
+
+    args = parse_args()
+    carla_image_name = "carla:latest"
+    inbox_assets_path = '/home/ue4/carla/Import'
+    client = docker.from_env()
+
+    # All possible Docker arguments are here:
+    # https://docker-py.readthedocs.io/en/stable/containers.html
+    container_args = {
+        "image": carla_image_name,
+        "user": 'ue4',
+        "auto_remove": True,
+        "stdin_open": True,
+        "tty": True,
+        "detach": True}
+
+    if args.packages:
+        container_args["volumes"] = {
+            args.input: {'bind': inbox_assets_path, 'mode': 'ro'}}
+
+    print(bold("- ") + bold_underline("Docker arguments:"))
+    print_formated_dict(container_args)
+
+    try:
+
+        print("Runnig Docker...")
+        carla_container = client.containers.run(**container_args)
+
+        if args.packages:
+            # If there is packages, import them first and package them
+            docker_utils.exec_command(
+                carla_container,
+                'make import',
+                user='ue4', verbose=args.verbose, ignore_error=False)
+
+            docker_utils.exec_command(
+                carla_container,
+                'make package ARGS="--packages=' + str(args.packages) + '"',
+                user='ue4', verbose=args.verbose, ignore_error=False)
+        else:
+            # Just create a package of the whole project
+            docker_utils.exec_command(
+                carla_container,
+                'make package',
+                user='ue4', verbose=args.verbose, ignore_error=False)
+
+        # Get the files routes to export
+        files_to_copy = docker_utils.get_file_paths(
+            carla_container,
+            '/home/ue4/carla/Dist/*.tar.gz',
+            user='ue4', verbose=args.verbose)
+
+        # Copy these fles to the output folder
+        docker_utils.extract_files(carla_container, files_to_copy, args.output)
+
+    finally:
+
+        print("Closing container " + carla_image_name)
+        carla_container.stop()
+
+
+if __name__ == '__main__':
+    main()

Util/Docker/docker_utils.py

@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2019 Computer Vision Center (CVC) at the Universitat Autonoma
+# de Barcelona (UAB).
+#
+# This work is licensed under the terms of the MIT license.
+# For a copy, see <https://opensource.org/licenses/MIT>.
+
+import tarfile
+
+
+BLUE = '\033[94m'
+GREEN = '\033[92m'
+RED = '\033[91m'
+ENDC = '\033[0m'
+BOLD = '\033[1m'
+UNDERLINE = '\033[4m'
+
+
+class ReadableStream():
+
+    def __init__(self, generator):
+        self._generator = generator
+
+    def read(self):
+        return next(self._generator)
+
+
+def get_container_name(container):
+    return str(container.attrs['Config']['Image'])
+
+
+def exec_command(container, command, user="root",
+                 silent=False, verbose=False, ignore_error=True):
+    command_prefix = "bash -c '"
+    if not silent:
+        print(''.join([BOLD, BLUE,
+                       user, '@', get_container_name(container),
+                       ENDC, '$ ', str(command)]))
+
+    command_result = container.exec_run(
+        command_prefix + command + "'",
+        user=user)
+    if not silent and verbose and command_result.exit_code:
+        print(''.join([RED, 'Command: "', command,
+                       '" exited with error: ', str(command_result.exit_code),
+                       ENDC]))
+        print('Error:')
+    if not silent:
+        out = command_result.output.decode().strip()
+        if out:
+            print(out)
+    if not ignore_error and command_result.exit_code:
+        exit(1)
+    return command_result
+
+
+def get_file_paths(container, path, user="root",
+                   absolute_path=True, hidden_files=True, verbose=False):
+    # command = "bash -c 'ls -ad $PWD/* "
+    command = "ls "
+    if hidden_files:
+        command += "-a "
+    if absolute_path:
+        command += "-d "
+    result = exec_command(container, command + path, user=user, silent=True)
+    if result.exit_code:
+        if verbose:
+            print(RED + "No files found in " + path + ENDC)
+        return []
+    file_list = [x for x in result.output.decode('utf-8').split('\n') if x]
+    if verbose:
+        print("Found files: " + str(file_list))
+    return file_list
+
+
+def extract_files(container, file_list, out_path):
+    for file in file_list:
+        print('Copying "' + file + '" to ' + out_path)
+        strm, _ = container.get_archive(file)
+        with tarfile.open(fileobj=ReadableStream(strm), mode='r|*') as tar:
+            tar.extractall(out_path)

Util/Docker/requirements.txt

@@ -0,0 +1 @@
+docker