Spatio-Temporal Memory for Navigation in a Mushroom Body Model
417
2 Methods
2.1 Input: Bio-inspired Event-Based Camera
Fig. 2. Comparing standard video to an event-based camera view of the test arena in
Fig. 4. Video (left) has static intensity frames at a fixed rate. ‘Events’ (centre) occur in
continuous time whenever a pixel changes intensity; integrated over a period of forward
motion, objects are visible in the movement ‘frame’ (right).
The input to our system is visual motion delivered in the form of spikes
from an event-based camera DAVIS 346 (346 × 260 pixels). Event-based cameras
(also called dynamic vision sensor (DVS)) output a spike when an individual
pixel detects brightness change. While conventional cameras capture from the
entire sensor at discrete time steps, event-based cameras only transmit pixel data
when the intensity of a given pixel shifts up or down by a predetermined amount.
This enables a time resolution of up to 1 ns (see Fig. 2). Event-based cameras
are inspired by that of vertebrate retina [17], but the principle of function is also
appropriate to represent motion perception in insect eyes. The asynchronous
pixels resemble transient photoreceptor responses encoding intensity change; and
neurons encoding either ON (dark to bright) or OFF (bright to dark) changes
are found in the insect lamina (neurons L1, L2 and L3) [18]. However, the spatial
acuity of insect eyes is usually between 2–5
◦ of their visual field, which is lower
than our camera resolution. Therefore, before feeding the camera output to the
neural network, the event flow is re-sampled by 16 pixel × 10 pixel × 1 ms spatiotemporal windows, with a threshold for activation that also reduces noise. After
re-sampling, the effective spatial acuity is 5 degrees and the temporal resolution
is 1 ms. Note that there are additional processing steps in the insect visual motion
pathway, but we have not included these steps in our modelling so far.
2.2 Mushroom Body Spiking Neural Network
Mushroom Body. The mushroom body (MB) is a multi-sensory processing
and learning centre in the insect brain. We have previously implemented an
417
2 Methods
2.1 Input: Bio-inspired Event-Based Camera
Fig. 2. Comparing standard video to an event-based camera view of the test arena in
Fig. 4. Video (left) has static intensity frames at a fixed rate. ‘Events’ (centre) occur in
continuous time whenever a pixel changes intensity; integrated over a period of forward
motion, objects are visible in the movement ‘frame’ (right).
The input to our system is visual motion delivered in the form of spikes
from an event-based camera DAVIS 346 (346 × 260 pixels). Event-based cameras
(also called dynamic vision sensor (DVS)) output a spike when an individual
pixel detects brightness change. While conventional cameras capture from the
entire sensor at discrete time steps, event-based cameras only transmit pixel data
when the intensity of a given pixel shifts up or down by a predetermined amount.
This enables a time resolution of up to 1 ns (see Fig. 2). Event-based cameras
are inspired by that of vertebrate retina [17], but the principle of function is also
appropriate to represent motion perception in insect eyes. The asynchronous
pixels resemble transient photoreceptor responses encoding intensity change; and
neurons encoding either ON (dark to bright) or OFF (bright to dark) changes
are found in the insect lamina (neurons L1, L2 and L3) [18]. However, the spatial
acuity of insect eyes is usually between 2–5
◦ of their visual field, which is lower
than our camera resolution. Therefore, before feeding the camera output to the
neural network, the event flow is re-sampled by 16 pixel × 10 pixel × 1 ms spatiotemporal windows, with a threshold for activation that also reduces noise. After
re-sampling, the effective spatial acuity is 5 degrees and the temporal resolution
is 1 ms. Note that there are additional processing steps in the insect visual motion
pathway, but we have not included these steps in our modelling so far.
2.2 Mushroom Body Spiking Neural Network
Mushroom Body. The mushroom body (MB) is a multi-sensory processing
and learning centre in the insect brain. We have previously implemented an