Docs: DVS camera documentation
This commit is contained in:
Javier Hidalgo-Carrió
Marc Garcia Puig
parent
26441431e4
commit
cfee8bdfaa
Binary file not shown.
|
After Width: | Height: | Size: 6.1 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
|
|
@ -2769,4 +2769,58 @@ Links another command to be executed right after. It allows to ease very common
|
|||
- **Parameters:**
|
||||
- `command` (_any carla Command_) – a Carla command.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## carla.DVSEvent<a name="carla.DVSEvent"></a>
|
||||
Class that defines a DVS event. An event is a quadruple, so a tuple of 4 elements, with x, y pixel coordinate location, timestamp t and polarity of the event. Learn more about them [here](ref_sensors.md).
|
||||
|
||||
<h3>Instance Variables</h3>
|
||||
- <a name="carla.DVSEvent.x"></a>**<font color="#f8805a">x</font>** (_int_)
|
||||
X pixel coordinate.
|
||||
- <a name="carla.DVSEvent.y"></a>**<font color="#f8805a">y</font>** (_int_)
|
||||
Y pixel coordinate.
|
||||
- <a name="carla.DVSEvent.t"></a>**<font color="#f8805a">t</font>** (_int_)
|
||||
Timestamp.
|
||||
- <a name="carla.DVSEvent.pol"></a>**<font color="#f8805a">pol</font>** (_bool_)
|
||||
Polarity of the event in boolean. True for positive and False for negative.
|
||||
|
||||
<h3>Dunder methods</h3>
|
||||
- <a name="carla.DVSEvent.__str__"></a>**<font color="#7fb800">\__str__</font>**(<font color="#00a6ed">**self**</font>)
|
||||
|
||||
---
|
||||
|
||||
## carla.DVSEventArray<a name="carla.DVSEventArray"></a>
|
||||
Class that defines a stream of events in [carla.DVSEvent](#carla.DVSEvent). Such stream is an array of arbitrary size depending on the number of events. This class also stores the field of view, the height and width of the image and the timestamp from convenience. Learn more about them [here](ref_sensors.md).
|
||||
|
||||
<h3>Instance Variables</h3>
|
||||
- <a name="carla.DVSEventArray.fov"></a>**<font color="#f8805a">fov</font>** (_float_)
|
||||
Horizontal field of view of the image in degrees.
|
||||
- <a name="carla.DVSEventArray.height"></a>**<font color="#f8805a">height</font>** (_int_)
|
||||
Image height in pixels.
|
||||
- <a name="carla.DVSEventArray.width"></a>**<font color="#f8805a">width</font>** (_int_)
|
||||
Image width in pixels.
|
||||
- <a name="carla.DVSEventArray.raw_data"></a>**<font color="#f8805a">raw_data</font>** (_bytes_)
|
||||
|
||||
<h3>Methods</h3>
|
||||
- <a name="carla.DVSEventArray.to_image"></a>**<font color="#7fb800">to_image</font>**(<font color="#00a6ed">**self**</font>)
|
||||
Converts the image following this pattern: blue indicates positive events, red indicates negative events.
|
||||
- <a name="carla.DVSEventArray.to_array"></a>**<font color="#7fb800">to_array</font>**(<font color="#00a6ed">**self**</font>)
|
||||
Convert the stream of events to an array of integer values in the following order [x, y, t, pol].
|
||||
- <a name="carla.DVSEventArray.to_array_x"></a>**<font color="#7fb800">to_array_x</font>**(<font color="#00a6ed">**self**</font>)
|
||||
Return an array with X pixel coordinate of all the events in the stream.
|
||||
- <a name="carla.DVSEventArray.to_array_y"></a>**<font color="#7fb800">to_array_y</font>**(<font color="#00a6ed">**self**</font>)
|
||||
Return an array with Y pixel coordinate of all the events in the stream.
|
||||
- <a name="carla.DVSEventArray.to_array_t"></a>**<font color="#7fb800">to_array_t</font>**(<font color="#00a6ed">**self**</font>)
|
||||
Return an array with the timestamp of all the events in the stream.
|
||||
- <a name="carla.DVSEventArray.to_array_pol"></a>**<font color="#7fb800">to_array_pol</font>**(<font color="#00a6ed">**self**</font>)
|
||||
Return an array with the polarity of all the events in the stream.
|
||||
|
||||
<h3>Dunder methods</h3>
|
||||
- <a name="carla.DVSEventArray.__getitem__"></a>**<font color="#7fb800">\__getitem__</font>**(<font color="#00a6ed">**self**</font>, <font color="#00a6ed">**pos**=int</font>)
|
||||
- <a name="carla.DVSEventArray.__iter__"></a>**<font color="#7fb800">\__iter__</font>**(<font color="#00a6ed">**self**</font>)
|
||||
- <a name="carla.DVSEventArray.__len__"></a>**<font color="#7fb800">\__len__</font>**(<font color="#00a6ed">**self**</font>)
|
||||
- <a name="carla.DVSEventArray.__setitem__"></a>**<font color="#7fb800">\__setitem__</font>**(<font color="#00a6ed">**self**</font>, <font color="#00a6ed">**pos**=int</font>, <font color="#00a6ed">**color**=[carla.Color](#carla.Color)</font>)
|
||||
- <a name="carla.DVSEventArray.__str__"></a>**<font color="#7fb800">\__str__</font>**(<font color="#00a6ed">**self**</font>)
|
||||
|
||||
---
|
||||
|
|
@ -11,6 +11,7 @@
|
|||
* [__RGB camera__](#rgb-camera)
|
||||
* [__RSS sensor__](#rss-sensor)
|
||||
* [__Semantic segmentation camera__](#semantic-segmentation-camera)
|
||||
* [__DVS camera__](#dvs-camera)
|
||||
|
||||
|
||||
---
|
||||
|
|
@ -1410,3 +1411,113 @@ The following tags are currently available:
|
|||
</table>
|
||||
|
||||
<br>
|
||||
|
||||
---
|
||||
## DVS camera
|
||||
|
||||
* __Blueprint:__ sensor.camera.dvs
|
||||
* __Output:__ [carla.DVSEventArray](python_api.md#carla.DVSEventArray) per step (unless `sensor_tick` says otherwise).
|
||||
|
||||
|
||||
A Dynamic Vision Sensor (DVS) or Event camera is a sensor that works
|
||||
radically differently from a conventional camera. Instead of capturing
|
||||
intensity images at a fixed rate, event cameras measure changes of intensity
|
||||
asynchronously, in the form of a stream of events, which encode per-pixel
|
||||
brightness changes. Event cameras possess outstanding properties when
|
||||
compared to standard cameras. They have a very high dynamic range (140 dB
|
||||
versus 60 dB), no motion blur, and high temporal resolution (in the order of
|
||||
microseconds). Event cameras are thus sensors that can provide high-quality
|
||||
visual information even in challenging high-speed scenarios and high dynamic
|
||||
range environments, enabling new application domains for vision-based
|
||||
algorithms.
|
||||
|
||||
The DVS camera outputs a stream of events. An event $e=(x,y,t,pol)$ is
|
||||
triggered at a pixel $x$, $y$ at a timestamp $t$ when the change in
|
||||
logarithmic intensity $L$ reaches a predefined constant threshold $C$
|
||||
(typically between 15% and 30%),
|
||||
|
||||
$$
|
||||
L(x,y,t) - L(x,y,t-\delta t) = pol C
|
||||
$$
|
||||
|
||||
where $t-\delta t$ is the time when the last event at that pixel was
|
||||
triggered and $pol$ is the polarity of the event according to the sign of the
|
||||
brightness change. The polarity is positive $+1$ when there is increment in
|
||||
brightness and negative $-1$ when a decrement in brightness occurs. The
|
||||
working principles depicted in the following figure. The standard
|
||||
camera outputs frames at a fixed rate, thus sending redundant information
|
||||
when no motion is present in the scene. In contrast, event cameras are
|
||||
data-driven sensors that respond to brightness changes with microsecond
|
||||
latency. At the plot, a positive (resp. negative) event (blue dot, resp. red
|
||||
dot) is generated whenever the (signed) brightness change exceeds the
|
||||
contrast threshold $C$ for one dimension $x$ over time $t$. Observe how the
|
||||
event rate grows when the signal changes rapidly.
|
||||
|
||||
|
||||
|
||||
The current implementation of the DVS camera works in a uniform sampling
|
||||
manner between two consecutive synchronous frames. Therefore, in order to
|
||||
emulate the high temporal resolution (order of microseconds) of a real event
|
||||
camera, the sensor requires to execute at a high frequency (much higher
|
||||
frequency than a conventional camera). Effectively, the number of events
|
||||
increases as faster a Carla car drives. Therefore, the sensor frequency
|
||||
should increase accordingly with the dynamic of the scene. The user should find
|
||||
their balance between time accuracy and computational cost.
|
||||
|
||||
The provided script `manual_control.py` uses the DVS camera in order to show
|
||||
how to configure the sensor, how to get the stream of events and how to depict such
|
||||
events in an image format, usually called event frame.
|
||||
|
||||
|
||||
|
||||
DVS is a camera and therefore has all the attributes available in the RGB camera. Nevertheless, there are few attributes
|
||||
exclusive to the working principle of an Event camera.
|
||||
|
||||
#### DVS camera attributes
|
||||
|
||||
<table class ="defTable">
|
||||
<thead>
|
||||
<th>Blueprint attribute</th>
|
||||
<th>Type</th>
|
||||
<th>Default</th>
|
||||
<th>Description</th>
|
||||
</thead>
|
||||
<tbody>
|
||||
<td>
|
||||
<code>positive_threshold</code> </td>
|
||||
<td>float</td>
|
||||
<td>0.3</td>
|
||||
<td>Positive threshold C associated to a increment in brightness change (0-1).</td>
|
||||
<tr>
|
||||
<td><code>negative_threshold</code></td>
|
||||
<td>float</td>
|
||||
<td>0.3</td>
|
||||
<td>Negative threshold C associated to a decrement in brightness change (0-1).</td>
|
||||
<tr>
|
||||
<td><code>sigma_positive_threshold</code></td>
|
||||
<td>float</td>
|
||||
<td>0</td>
|
||||
<td>White noise standard deviation for positive events (0-1)</td>
|
||||
<tr>
|
||||
<td><code>sigma_negative_threshold</code></td>
|
||||
<td>float</td>
|
||||
<td>0</td>
|
||||
<td>White noise standard deviation for negative events (0-1)</td>
|
||||
<tr>
|
||||
<td><code>refractory_period_ns</code></td>
|
||||
<td>int</td>
|
||||
<td>0.0</td>
|
||||
<td> Refractory period (time during which a pixel cannot fire events just after it fired one), in nanoseconds. It limits the highest frequency of triggering events.</td>
|
||||
<tr>
|
||||
<td><code>use_log</code></td>
|
||||
<td>bool</td>
|
||||
<td>true</td>
|
||||
<td>Whether to work in the logarithmic intensity scale.</td>
|
||||
<tr>
|
||||
<td><code>log_eps</code></td>
|
||||
<td>float</td>
|
||||
<td>0.001</td>
|
||||
<td>Epsilon value used to convert images to log: L = log(eps + I / 255.0). Where I is the grayscale value of the RGB image I = 0.2989*R + 0.5870*G + 0.1140*B.</td>
|
||||
</tbody>
|
||||
</table>
|
||||
<br>
|
||||
|
|
|
|||
Loading…
Reference in New Issue