Fast Reverse Replays in a Robotic Hippocampal Model
399
to understand how replays emerge with those that utilise replays for improving
learning in artificial systems.
As noted, reverse replays are particularly thought to be involved in reinforcement learning, given their reward-modulated occurrences [1] and coordinated
activity with neurons in the striatum [14,32]. As such, an investigation into
whether reverse replays could improve learning in biophysical models of reinforcement learning, such as [40], should be conducted to further ground these
hypotheses in theory. In particular, the model of Vasilaki et al. [40] belongs to a
class of models termed three-factor learning rules [12], which predict that synaptic eligibility traces are necessary to support long-term synaptic modifications
across behavioural timescales. The third factor, often a reward signal, dictates
that synapses only undergo modifications in the presence of that third factor.
Reverse replays however offer an alternative (though not mutually exclusive)
mechanism to the synaptic eligibility trace hypothesis, since the replaying of cell
trajectories could support the conditions necessary for synaptic modification
under a third factor.
Finally, in order to generate cell activity that was place specific in this model,
the global x-y coordinates for the robot had to be used which is a biologically
unrealistic property of place cell emergence. Questions remain therefore around
how other models of the hippocampus, ones attempting to understand the emergence of place cells, grid cells, head direction cells, etc. [20,35], could be consolidated with hippocampal replay models. In addition, there is the question of how
hippocampal replay integrates with the rest of the brain. Interesting work at
building a more complete cognitive architecture has been done by Maffei et al.
[23], which includes a hippocampal model of place cell emergence. For path planning, shortest paths are found via a sweeping algorithm (Dijkstra’s algorithm),
but a more biologically plausible method might instead be through hippocampal
replays.
Acknowledgements. This work has been in part funded by the Human Brain
Project, under project number 785907 (SGA2).
References
1. Ambrose, R.E., Pfeiffer, B.E., Foster, D.J.: Reverse replay of hippocampal place
cells is uniquely modulated by changing reward. Neuron 91(5), 1124–1136 (2016)
2. Atherton, L.A., Dupret, D., Mellor, J.R.: Memory trace replay: the shaping of memory consolidation by neuromodulation. Trends Neurosci. 38(9), 560–570 (2015)
3. Aubin, L., Khamassi, M., Girard, B.: Prioritized sweeping neural DynaQ with multiple predecessors, and hippocampal replays. In: Vouloutsi, V., et al. (eds.) Living Machines 2018. LNCS (LNAI), vol. 10928, pp. 16–27. Springer, Cham (2018).
https://doi.org/10.1007/978-3-319-95972-6 4
4. Carr, M.F., Jadhav, S.P., Frank, L.M.: Hippocampal replay in the awake state: a
potential substrate for memory consolidation and retrieval. Nat. Neurosci. 14(2),
147 (2011)
5. Chenkov, N., Sprekeler, H., Kempter, R.: Memory replay in balanced recurrent
networks. PLoS Comput. Biol. 13(1), e1005359 (2017)
399
to understand how replays emerge with those that utilise replays for improving
learning in artificial systems.
As noted, reverse replays are particularly thought to be involved in reinforcement learning, given their reward-modulated occurrences [1] and coordinated
activity with neurons in the striatum [14,32]. As such, an investigation into
whether reverse replays could improve learning in biophysical models of reinforcement learning, such as [40], should be conducted to further ground these
hypotheses in theory. In particular, the model of Vasilaki et al. [40] belongs to a
class of models termed three-factor learning rules [12], which predict that synaptic eligibility traces are necessary to support long-term synaptic modifications
across behavioural timescales. The third factor, often a reward signal, dictates
that synapses only undergo modifications in the presence of that third factor.
Reverse replays however offer an alternative (though not mutually exclusive)
mechanism to the synaptic eligibility trace hypothesis, since the replaying of cell
trajectories could support the conditions necessary for synaptic modification
under a third factor.
Finally, in order to generate cell activity that was place specific in this model,
the global x-y coordinates for the robot had to be used which is a biologically
unrealistic property of place cell emergence. Questions remain therefore around
how other models of the hippocampus, ones attempting to understand the emergence of place cells, grid cells, head direction cells, etc. [20,35], could be consolidated with hippocampal replay models. In addition, there is the question of how
hippocampal replay integrates with the rest of the brain. Interesting work at
building a more complete cognitive architecture has been done by Maffei et al.
[23], which includes a hippocampal model of place cell emergence. For path planning, shortest paths are found via a sweeping algorithm (Dijkstra’s algorithm),
but a more biologically plausible method might instead be through hippocampal
replays.
Acknowledgements. This work has been in part funded by the Human Brain
Project, under project number 785907 (SGA2).
References
1. Ambrose, R.E., Pfeiffer, B.E., Foster, D.J.: Reverse replay of hippocampal place
cells is uniquely modulated by changing reward. Neuron 91(5), 1124–1136 (2016)
2. Atherton, L.A., Dupret, D., Mellor, J.R.: Memory trace replay: the shaping of memory consolidation by neuromodulation. Trends Neurosci. 38(9), 560–570 (2015)
3. Aubin, L., Khamassi, M., Girard, B.: Prioritized sweeping neural DynaQ with multiple predecessors, and hippocampal replays. In: Vouloutsi, V., et al. (eds.) Living Machines 2018. LNCS (LNAI), vol. 10928, pp. 16–27. Springer, Cham (2018).
https://doi.org/10.1007/978-3-319-95972-6 4
4. Carr, M.F., Jadhav, S.P., Frank, L.M.: Hippocampal replay in the awake state: a
potential substrate for memory consolidation and retrieval. Nat. Neurosci. 14(2),
147 (2011)
5. Chenkov, N., Sprekeler, H., Kempter, R.: Memory replay in balanced recurrent
networks. PLoS Comput. Biol. 13(1), e1005359 (2017)