p71924506
/
carla
mirror of https://github.com/carla-simulator/carla.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Added positions

This commit is contained in:
felipecode 2018-04-12 15:05:05 +02:00
parent a92480c431
commit 1c0f022a7e
8 changed files with 130 additions and 110 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

80
Docs/benchmark_creating.md
View File

@ -1,11 +1,20 @@
Benchmarking your Agent Benchmarking your Agent
--------------------------- ---------------------------
![Benchmark_structure](img/benchmark_diagram_small.png )
In this tutorial we show:
* [How to define a trivial agent with a forward going policy.](#defining-the-agent)
* [How to define a basic experiment suite.](#defining-the-experiment-suite)
#### Introduction
![Benchmark_structure](img/benchmark_diagram_small.png)
The driving benchmark is associated with other two modules. The driving benchmark is associated with other two modules.
The *agent* module, a controller which performs in The *agent* module, that is a controller which performs in
another module, the *experiment suite*. another module: the *experiment suite*.
Both modules are abstract classes that must be redefined by Both modules are abstract classes that must be redefined by
the user. the user.
@ -30,20 +39,18 @@ an example of how to apply a driving benchmark;
Following this excerpt, there are two classes to be defined. Following this excerpt, there are two classes to be defined.
The ForwardAgent() and the BasicExperiments(). The ForwardAgent() and the BasicExperimentSuite().
After that, we run the benchmark it. As result of the execution, the driving benchmark After that, the benchmark is run with the "run_driving_benchmark" function.
returns stores a summary of the calculated [performance metrics](benchmark_metrics.md). The summary of the execution, the [performance metrics](benchmark_metrics.md), are stored
in a json file and printed to the screen.
In this tutorial we are going to show how to define
a basic experiment suite and a trivial agent with a going
forward policy.
#### Defining the Agent #### Defining the Agent
The tested agent must inherit the base *Agent* class. The tested agent must inherit the base *Agent* class.
Lets start by deriving a simple Forward agent. Lets start by deriving a simple forward agent:
from carla.agent.agent import Agent from carla.agent.agent import Agent
from carla.client import VehicleControl from carla.client import VehicleControl
@ -75,7 +82,7 @@ This function receives the following parameters:
* Target Position: The position and orientation of the target. * Target Position: The position and orientation of the target.
With all this information, the *run_step* function is expected With all this information, the *run_step* function is expected
to return a [vehicle control message](measurements.md) containing, to return a [vehicle control message](measurements.md), containing:
steering value, throttle value, brake value, etc. steering value, throttle value, brake value, etc.
@ -88,7 +95,7 @@ the following steps:
* Create your custom class by inheriting the ExperimentSuite base class. * Create your custom class by inheriting the ExperimentSuite base class.
* Define the test and train weather conditions to be used. * Define the test and train weather conditions to be used.
* Build the *Experiment* objects * Build the *Experiment* objects .
@ -96,7 +103,7 @@ the following steps:
The defined set of experiments must derive the *ExperimentSuite* class The defined set of experiments must derive the *ExperimentSuite* class
as in the following code. as in the following code excerpt:
from carla.agent_benchmark.experiment import Experiment from carla.agent_benchmark.experiment import Experiment
from carla.sensor import Camera from carla.sensor import Camera
@ -105,9 +112,9 @@ as in the following code.
from .experiment_suite import ExperimentSuite from .experiment_suite import ExperimentSuite
class Basic(ExperimentSuite): class BasicExperimentSuite(ExperimentSuite):
##### Define the used weathers ##### Define test and train weather conditions
The user must select the weathers to be used. One should select the set The user must select the weathers to be used. One should select the set
of test weathers and the set of train weathers. This is defined as a of test weathers and the set of train weathers. This is defined as a
@ -124,27 +131,26 @@ class property as in the following example:
##### Building Experiments ##### Building Experiments
The [experiments are composed by a *task* that is defined by a set of *poses*](benchmark_structure.md). The [experiments are composed by a *task* that is defined by a set of *poses*](benchmark_structure.md).
Lets start by selecting poses for one of the cities, Town01 for instance. Lets start by selecting poses for one of the cities, lets take Town01, for instance.
First of all, we need to see all the possible positions, for that, with First of all, we need to see all the possible positions, for that, with
a CARLA simulator running in a terminal, run: a CARLA simulator running in a terminal, run:
python view_start_positions.py python view_start_positions.py
![town01_positions](img/town01_positions.png) ![town01_positions](img/town01_positions.png)
> Figure 2: All the possible start positions for CARLA Town01.
Now lets choose, for instance, 105 as start position and 29 Now lets choose, for instance, 105 as start position and 29
as end. This two positions can be visualized by running. as the end position. This two positions can be visualized by running:
python view_start_positions.py --pos 105,29 --no-labels python view_start_positions.py --pos 105,29 --no-labels
![town01_positions](img/town01_109_29.png)
> Figure 2: A start and an end position.
Lets choose two more poses, one for going straight, other one for one simple turn.
Lets define Also, lets also choose three poses for Town02:
two more poses, one for going straight, other one for one simple turn.
Also, lets also choose three poses for Town02.
Figure 3, shows these defined poses for both carla towns.
![town01_positions](img/initial_positions.png) ![town01_positions](img/initial_positions.png)
@ -156,13 +162,19 @@ Figure 3, shows these defined poses for both carla towns.
the goal is far away from the start position, usually more than one turn. the goal is far away from the start position, usually more than one turn.
We define each of this defined poses as tasks. Plus, we also set We define each of this defined poses as a task. Plus, we also set
the number of dynamic objects for each of these tasks and repeat the number of dynamic objects for each of these tasks and repeat
the arbitrary position task to have it also defined with dynamic the arbitrary position task to have it also defined with dynamic
objects. This is defined objects. In the following code excerpt we show the final
in the following code excerpt: defined positions and the number of dinamic objects for each task:
poses_tasks = [[[36, 40]], [[138, 17]], [[105, 29]], [[105, 29]]] # Define the start/end position above as tasks
poses_task0 = [[36, 40]]
poses_task1 = [[138, 17]]
poses_task2 = [[105, 29]]
poses_task3 = [[105, 29]]
# Concatenate all the tasks
poses_tasks = poses_task0, poses_task1 , poses_task1 , poses_task3]
vehicles_tasks = [0, 0, 0, 20] vehicles_tasks = [0, 0, 0, 20]
pedestrians_tasks = [0, 0, 0, 50] pedestrians_tasks = [0, 0, 0, 50]
@ -171,10 +183,10 @@ Keep in mind that a task is a set of episodes with start and end points.
Finally by using the defined tasks we can build the experiments Finally by using the defined tasks we can build the experiments
vector as we show in the following code excerpt: vector as we show in the following code excerpt:
```
experiments_vector = []
for weather in used_weathers:
experiments_vector = []
# The used weathers is the union between test and train weathers
for weather in used_weathers:
for iteration in range(len(poses_tasks)): for iteration in range(len(poses_tasks)):
poses = poses_tasks[iteration] poses = poses_tasks[iteration]
vehicles = vehicles_tasks[iteration] vehicles = vehicles_tasks[iteration]
@ -198,12 +210,12 @@ experiments_vector = []
Repetitions=1 Repetitions=1
) )
experiments_vector.append(experiment) experiments_vector.append(experiment)
```
The full code could be found at basic.py (LINK)
The full code could be found at [basic_experiment_suite.py](LINK)

40
Docs/benchmark_metrics.md
View File

@ -3,75 +3,71 @@ Driving Benchmark Performance Metrics
------------------------------ ------------------------------
This page explains the performance metrics module. This page explains the performance metrics module.
Use to compute a summary of results based on the agent This module is used to compute a summary of results based on the agent
actions when completing the experiments. actions when completing experiments.
### Provided performance metrics ### Provided performance metrics
The driving benchmark performance metrics module provides the following performance metrics: The driving benchmark performance metrics module provides the following performance metrics:
* Percentage of Success: The percentage of episodes (poses from tasks), * **Percentage of Success**: The percentage of episodes (poses from tasks),
that the agent successfully completed. that the agent successfully completed.
* Average Completion: The average distance towards the goal that the * **Average Completion**: The average distance towards the goal that the
agent was able to travel. agent was able to travel.
* Off Road Intersection: The number of times the agent goes out of the road. * **Off Road Intersection**: The number of times the agent goes out of the road.
The intersection is only counted if the area of the vehicle outside The intersection is only counted if the area of the vehicle outside
of the road is bigger than a *threshold*. of the road is bigger than a *threshold*.
* Other Lane Intersection: The number of times the agent goes to the other * **Other Lane Intersection**: The number of times the agent goes to the other
lane. The intersection is only counted if the area of the vehicle on the lane. The intersection is only counted if the area of the vehicle on the
other lane is bigger than a *threshold*. other lane is bigger than a *threshold*.
* Vehicle Collisions: The number of collisions with vehicles that have * **Vehicle Collisions**: The number of collisions with vehicles that had
an impact bigger than a *threshold*. an impact bigger than a *threshold*.
* Pedestrian Collisions: The number of collisions with pedestrians * **Pedestrian Collisions**: The number of collisions with pedestrians
that have an impact bigger than a *threshold*. that had an impact bigger than a *threshold*.
* General Collisions: The number of collisions with all other * **General Collisions**: The number of collisions with all other
objects with an impact bigger than a *threshold*. objects with an impact bigger than a *threshold*.
### Executing and Setting Parameters ### Executing and Setting Parameters
The metrics are computed as the final step of the benchmark The metrics are computed as the final step of the benchmark
and it is returned by the [benchmark_agent()](benchmark_creating.md). and stores a summary of the results a json file.
Internally it is executed as follows: Internally it is executed as follows:
metrics_object = Metrics(metrics_parameters) metrics_object = Metrics(metrics_parameters)
summary_dictionary = metrics_object.compute(path_to_execution_log) summary_dictionary = metrics_object.compute(path_to_execution_log)
The performance metric compute function The Metric's compute function
receives the full path to the execution log receives the full path to the execution log.
and dictionary with the performance metrics.
Also, the metric class is instanced with the metric parameters. Also, the metric class is instanced with the metric parameters.
The parameters are: The parameters are:
* Threshold: The threshold used by the metrics.
* Frames Recount: After making the infraction, set the number * **Threshold**: The threshold used by the metrics.
* **Frames Recount**: After making the infraction, set the number
of frames that the agent needs to keep doing the infraction for of frames that the agent needs to keep doing the infraction for
it to be counted as another infraction. it to be counted as another infraction.
* **Frames Skip**: It is related to the number of frames that are
* Frames Skip: It is related to the number of frames that are
skipped after a collision or a intersection starts. skipped after a collision or a intersection starts.
These parameters are defined as property of the *Experiment Suite* These parameters are defined as property of the *Experiment Suite*
base class and can be redefined at your base class and can be redefined at your
[custom *Experiment Suite*](benchmark_creating.md/#defining-the-experiment-suite). [custom *Experiment Suite*](benchmark_creating.md/#defining-the-experiment-suite).
The default defined parameters are: The default parameters are:
@property @property
def metrics_parameters(self): def metrics_parameters(self):
""" """
Property to return the parameters for the metric module Property to return the parameters for the metrics module
Could be redefined depending on the needs of the user. Could be redefined depending on the needs of the user.
""" """
return { return {

27
Docs/benchmark_start.md
View File

@ -7,17 +7,17 @@ metrics about its performance.
This module is mainly designed for: This module is mainly designed for:
* Users that work with autonomous driving agents and want * Users that work developing autonomous driving agents and want
to see how they perform in carla. to see how they perform in CARLA.
On this section you will learn. On this section you will learn.
* How to quickly get started and benchmark a trivial agent right away. * How to quickly get started and benchmark a trivial agent right away.
* Learn about the general implementation architecture of the driving * Learn about the general implementation [architecture of the driving
benchmark module. benchmark module](benchmark_structure.md).
* Learn how to set up your agent to be benchmarked and create your * Learn [how to set up your agent and create your
how set of experiments to challenge your agents. own set of experiments](benchmark_creating.md) for challenging your agents.
* Learn about the performance metrics used. * Learn about the [performance metrics used](benchmark_metrics.md).
@ -34,7 +34,7 @@ run:
Keep in mind that to run the command above, you need a CARLA simulator Keep in mind that to run the command above, you need a CARLA simulator
server running, at localhost and on port 2000. running at localhost and on port 2000.
We already provide the same benchmark used in the CoRL We already provide the same benchmark used in the CoRL
@ -49,10 +49,19 @@ Run the help command to see options available.
$ ./driving_benchmark_example.py --help $ ./driving_benchmark_example.py --help
One of the options available is to be able to continue
from a previous benchmark execution. For example,
to continue a experiment in CoRL2017 with a log name of "driving_benchmark_test", run:
$ ./driving_benchmark_example.py --continue-experiment -n driving_benchmark_test --corl-2017
! Note: if the log name already exist and you dont set it to continue it
will create another log witha number.
When running the driving benchmark for the basic configuration When running the driving benchmark for the basic configuration
you should [expect the following results](benchmark_creating.md/#expected-results) you should [expect the following results](benchmark_creating.md/#expected-results)
to be printed on the terminal. to be printed on the terminal in more or less 5 minutes,
depending on your machine configuration.

35
Docs/benchmark_structure.md
View File

@ -2,7 +2,7 @@
Driving Benchmark Structure Driving Benchmark Structure
------------------- -------------------
The Figure 1 one shows the general structure of the driving The Figure 1 shows the general structure of the driving
benchmark module. benchmark module.
@ -16,27 +16,29 @@ The *driving benchmark* is the module responsible for evaluating a certain
*agent* in a *experiment suite*. *agent* in a *experiment suite*.
The *experiment suite* is an abstract module. The *experiment suite* is an abstract module.
Thus, the user must define its own experiment suite to be tested Thus, the user must define its own derivation
on an agent. We already provide the CoRL2017 suite, and a simple of *experiment suite* to be tested
one for more basic testing. on an agent. We already provide the CoRL2017 suite and a simple
*experiment suite* for testing.
The *experiment suite* is composed by set of *experiments*. The *experiment suite* is composed by set of *experiments*.
Each *experiment* contains to *task* that consist of a set of navigation Each *experiment* contains a *task* that consists of a set of navigation
episodes, that are represented by a set of *poses*. episodes, that are represented by a set of *poses*.
These *poses* are tuples of a start and end point of the episode. These *poses* are tuples are the start and end points of an
episode.
The *experiments* are also associated with a *condition* which is The *experiments* are also associated with a *condition*. A
a [carla settings](carla_settings.md) object. The conditions specify condition is represented by a [carla settings](carla_settings.md) object.
simulation parameters such as: weather, sensor suite, number of The conditions specify simulation parameters such as: weather, sensor suite, number of
vehicles and pedestrians, etc. vehicles and pedestrians, etc.
The user also should define the *agent* class. The *agent* is the active The user also should derivate an *agent* class. The *agent* is the active
part which will be evaluated by the agent benchmark module. part which will be evaluated by the agent benchmark module.
The driving benchmark also contains two auxiliary modules. The driving benchmark also contains two auxiliary modules.
The *recording module* is used to keep track of all measurements and The *recording module* is used to keep track of all measurements and
can be used to pause and continue the evaluation. can be used to pause and continue a driving benchmark.
The [*metrics module*](benchmark_metrics.md) is used to compute the performance metrics The [*metrics module*](benchmark_metrics.md) is used to compute the performance metrics
by using the recorded measurements. by using the recorded measurements.
@ -47,7 +49,7 @@ Example: CORL 2017
---------------------- ----------------------
We already provide the CoRL 2017 experiment suite used to benchmark the We already provide the CoRL 2017 experiment suite used to benchmark the
agents for the [CoRL 2017 paper](LINK). agents for the [CoRL 2017 paper](http://proceedings.mlr.press/v78/dosovitskiy17a/dosovitskiy17a.pdf).
The CoRL 2017 experiment suite has the following composition: The CoRL 2017 experiment suite has the following composition:
@ -56,12 +58,13 @@ The CoRL 2017 experiment suite has the following composition:
* A task for making a single turn. * A task for making a single turn.
* A task for going to an arbitrary position. * A task for going to an arbitrary position.
* A task for going to an arbitrary position with dynamic objects. * A task for going to an arbitrary position with dynamic objects.
* Each task is composed of 25 poses and is repeated in 6 different weathers
(Clear Noon, Heavy Rain Noon, Clear Sunset, After Rain Noon, Cloudy After Rain
and Soft Rain Sunset).
* Each task is composed of 25 poses that is repeated in 6 different weathers (Clear Noon, Heavy Rain Noon, Clear Sunset, After Rain Noon, Cloudy After Rain and Soft Rain Sunset).
* The entire experiment set has 600 episodes. * The entire experiment set has 600 episodes.
* The CoRL 2017 can take up to 24 hours to execute for Town01 and up to 15 * The CoRL 2017 can take up to 24 hours to execute for Town01 and up to 15
hours for Town02 depending on the drivers performance. hours for Town02 depending on the agent performance.

BIN
Docs/img/benchmark_diagram_small.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 60 KiB

After

Width:  |  Height:  |  Size: 66 KiB

BIN
Docs/img/town01_109_29.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 92 KiB

BIN
Docs/img/town01_positions.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 330 KiB

After

Width:  |  Height:  |  Size: 320 KiB

2
Docs/index.md
View File

@ -15,7 +15,7 @@
* [How to build on Linux](how_to_build_on_linux.md) * [How to build on Linux](how_to_build_on_linux.md)
* [How to build on Windows](how_to_build_on_windows.md) * [How to build on Windows](how_to_build_on_windows.md)
<h3> Agent Benchmark </h3> <h3> Driving Benchmark </h3>
* [Quick Start](benchmark_start.md) * [Quick Start](benchmark_start.md)
* [General Structure](benchmark_structure.md) * [General Structure](benchmark_structure.md)