GIBSON ENVIRONMENT for Embodied Active Agents with Real-World Perception (V2)

Gibson V2 is an updated version of GibsonEnv, it achieves higher rendering performance and added the ability for robots to interact with objects.

Summary: Perception and being active (i.e. having a certain level of motion freedom) are closely tied. Learning active perception and sensorimotor control in the physical world is cumbersome as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given a fruitful rise to learning in the simulation which consequently casts a question on transferring to real-world. We developed Gibson environment with the following primary characteristics:

I. being from the real-world and reflecting its semantic complexity through virtualizing real spaces,
II. having a baked-in mechanism for transferring to real-world (Goggles function), and
III. embodiment of the agent and making it subject to constraints of space and physics via integrating a physics engine (Bulletphysics).

Naming: Gibson environment is named after James J. Gibson, the author of "Ecological Approach to Visual Perception", 1979. “We must perceive in order to move, but we must also move in order to perceive” – JJ Gibson

Please see the website (http://gibsonenv.stanford.edu/) for more technical details. This repository is intended for distribution of the environment and installation/running instructions.

Paper

"Gibson Env: Real-World Perception for Embodied Agents", in CVPR 2018 [Spotlight Oral].

Release

This is the gibson2 0.0.1 release. Bug reports, suggestions for improvement, as well as community developments are encouraged and appreciated. change log file.

Support for Gibson v1 will be moved to this repo.

Database

The full database includes 572 spaces and 1440 floors and can be downloaded here. A diverse set of visualizations of all spaces in Gibson can be seen here. To make the core assets download package lighter for the users, we include a small subset (39) of the spaces. Users can download the rest of the spaces and add them to the assets folder. We also integrated Stanford 2D3DS and Matterport 3D as separate datasets if one wishes to use Gibson's simulator with those datasets (access here).

Table of contents

• Installation
• Quick Start
  • Tests
  • Gibson FPS
  • Rendering Semantics
  • Robotic Agents
  • ROS Configuration
• Coding your RL agent
• Environment Configuration
• Goggles: transferring the agent to real-world
• Citation

Installation

Installation Method

Gibson v2 can be installed as a python package:

git clone https://github.com/fxia22/gibsonv2
cd gibsonv2

conda create -n py3-gibson python=3.6 anaconda
source activate py3-gibson
pip install -e .

System requirements

The minimum system requirements are the following:

• Ubuntu 16.04
• Nvidia GPU with VRAM > 6.0GB
• Nvidia driver >= 384
• CUDA >= 9.0, CuDNN >= v7

Download data

First, our environment core assets data are available here. You can store the data where you want and put the path in global_config.yaml. The assets folder stores necessary data (agent models, environments, etc) to run gibson environment.

Users can add more environments files into dataset folder and put the path in global_config.yaml to run gibson on more environments. Visit the database readme for downloading more spaces. Please sign the license agreement before using Gibson's database. The default path is:

assets_path: assets #put either absolute path or relative to current directory
dataset_path: assets/dataset

Uninstalling

Uninstall gibson is easy with pip uninstall gibson2

Quick Start

Tests

cd test
pytest # the tests should pass, it will take a few minutes

Gibson v2 Framerate

Gibson v2 framerate compared with gibson v1 is shown in the table below:

	Gibson V2	Gibson V1
RGBD, pre networkf	264.1	58.5
RGBD, post networkf	61.7	30.6
Surface Normal only	271.1	129.7
Semantic only	279.1	144.2
Non-Visual Sensory	1017.4	396.1

Rendering Semantics

TBA

Robotic Agents

Gibson provides a base set of agents. See videos of these agents and their corresponding perceptual observation here.

To enable (optionally) abstracting away low-level control and robot dynamics for high-level tasks, we also provide a set of practical and ideal controllers for each agent.

Agent Name	DOF	Information	Controller
Mujoco Ant	8	OpenAI Link	Torque
Mujoco Humanoid	17	OpenAI Link	Torque
Husky Robot	4	ROS, Manufacturer	Torque, Velocity, Position
Minitaur Robot	8	Robot Page, Manufacturer	Sine Controller
JackRabbot	2	Stanford Project Link	Torque, Velocity, Position
TurtleBot	2	ROS, Manufacturer	Torque, Velocity, Position
Quadrotor	6	Paper	Position

Starter Code

Demonstration examples can be found in examples/demo folder. demo.py shows the procedure of starting an environment with a random agent.

ROS Configuration

We provide examples of configuring Gibson with ROS here. We use turtlebot as an example, after a policy is trained in Gibson, it requires minimal changes to deploy onto a turtlebot. See README for more details.

Coding Your RL Agent

You can code your RL agent following our convention. The interface with our environment is very simple (see some examples in the end of this section).

First, you can create an environment by creating an instance of classes in gibson/core/envs folder.

env = AntNavigateEnv(is_discrete=False, config = config_file)

Then do one step of the simulation with env.step. And reset with env.reset()

obs, rew, env_done, info = env.step(action)

obs gives the observation of the robot. It is a dictionary with each component as a key value pair. Its keys are specified by user inside config file. E.g. obs['nonviz_sensor'] is proprioceptive sensor data, obs['rgb_filled'] is rgb camera data.

rew is the defined reward. env_done marks the end of one episode, for example, when the robot dies. info gives some additional information of this step; sometimes we use this to pass additional non-visual sensor values.

We mostly followed OpenAI gym convention when designing the interface of RL algorithms and the environment. In order to help users start with the environment quicker, we provide some examples at examples/train. The RL algorithms that we use are from openAI baselines with some adaptation to work with hybrid visual and non-visual sensory data. In particular, we used PPO and a speed optimized version of PPO.

Environment Configuration

Each environment is configured with a yaml file. Examples of yaml files can be found in examples/configs folder. Parameters for the file is explained below. For more informat specific to Bullet Physics engine, you can see the documentation here.

Argument name	Example value	Explanation
envname	AntClimbEnv	Environment name, make sure it is the same as the class name of the environment
model_id	space1-space8	Scene id, in beta release, choose from space1-space8
target_orn	[0, 0, 3.14]	Eulerian angle (in radian) target orientation for navigating, the reference frame is world frame. For non-navigation tasks, this parameter is ignored.
target_pos	[-7, 2.6, -1.5]	target position (in meter) for navigating, the reference frame is world frame. For non-navigation tasks, this parameter is ignored.
initial_orn	[0, 0, 3.14]	initial orientation (in radian) for navigating, the reference frame is world frame
initial_pos	[-7, 2.6, 0.5]	initial position (in meter) for navigating, the reference frame is world frame
fov	1.57	field of view for the camera, in radian
use_filler	true/false	use neural network filler or not. It is recommended to leave this argument true. See Gibson Environment website for more information.
display_ui	true/false	Gibson has two ways of showing visual output, either in multiple windows, or aggregate them into a single pygame window. This argument determines whether to show pygame ui or not, if in a production environment (training), you need to turn this off
show_diagnostics	true/false	show dignostics(including fps, robot position and orientation, accumulated rewards) overlaying on the RGB image
output	[nonviz_sensor, rgb_filled, depth]	output of the environment to the robot, choose from [nonviz_sensor, rgb_filled, depth]. These values are independent of ui_components, as ui_components determines what to show and output determines what the robot receives.
resolution	512	choose from [128, 256, 512] resolution of rgb/depth image
mode	gui/headless/web_ui	gui or headless, if in a production environment (training), you need to turn this to headless. In gui mode, there will be visual output; in headless mode, there will be no visual output. In addition to that, if you set mode to web_ui, it will behave like in headless mode but the visual will be rendered to a web UI server. (more information)
verbose	true/false	show diagnostics in terminal
fast_lq_render	true/false	if there is fast_lq_render in yaml file, Gibson will use a smaller filler network, this will render faster but generate slightly lower quality camera output. This option is useful for training RL agents fast.

Making Your Customized Environment

Gibson provides a set of methods for you to define your own environments. You can follow the existing environments inside gibson/core/envs.

Method name	Usage
robot.get_position()	Get current robot position.
robot.get_orientation()	Get current robot orientation.
robot.eyes.get_position()	Get current robot perceptive camera position.
robot.eyes.get_orientation()	Get current robot perceptive camera orientation.
robot.get_target_position()	Get robot target position.
robot.apply_action(action)	Apply action to robot.
robot.reset_new_pose(pos, orn)	Reset the robot to any pose.
robot.dist_to_target()	Get current distance from robot to target.

Goggles: transferring the agent to real-world

Gibson includes a baked-in domain adaptation mechanism, named Goggles, for when an agent trained in Gibson is going to be deployed in real-world (i.e. operate based on images coming from an onboard camera). The mechanisms is essentially a learned inverse function that alters the frames coming from a real camera to what they would look like if they were rendered via Gibson, and hence, disolve the domain gap.

More details: With all the imperfections in point cloud rendering, it has been proven difficult to get completely photo-realistic rendering with neural network fixes. The remaining issues make a domain gap between the synthesized and real images. Therefore, we formulate the rendering problem as forming a joint space ensuring a correspondence between rendered and real images, rather than trying to (unsuccessfully) render images that are identical to real ones. This provides a deterministic pathway for traversing across these domains and hence undoing the gap. We add another network "u" for target image (I_t) and define the rendering loss to minimize the distance between f(I_s) and u(I_t), where "f" and "I_s" represent the filler neural network and point cloud rendering output, respectively (see the loss in above figure). We use the same network structure for f and u. The function u(I) is trained to alter the observation in real-world, I_t, to look like the corresponding I_s and consequently dissolve the gap. We named the u network goggles, as it resembles corrective lenses for the agent for deployment in real-world. Detailed formulation and discussion of the mechanism can be found in the paper. You can download the function u and apply it when you deploy your trained agent in real-world.

In order to use goggle, you will need preferably a camera with depth sensor, we provide an example here for Kinect. The trained goggle functions are stored in assets/unfiller_{resolution}.pth, and each one is paired with one filler function. You need to use the correct one depending on which filler function is used. If you don't have a camera with depth sensor, we also provide an example for RGB only here.

Citation

If you use Gibson Environment's software or database, please cite:

@inproceedings{xiazamirhe2018gibsonenv,
  title={Gibson {Env}: real-world perception for embodied agents},
  author={Xia, Fei and R. Zamir, Amir and He, Zhi-Yang and Sax, Alexander and Malik, Jitendra and Savarese, Silvio},
  booktitle={Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on},
  year={2018},
  organization={IEEE}
}