408
P. A. Dananjaya et al.
5. As mentioned in [92], pooling layers can be avoided in deep neural network.
Hence, even though there are techniques to convert pooling layers in SDNN, we
can remove these layers from the DNN for simplicity sake.
6 On-Chip Learning on RRAM Based Neuromorphic
Hardware: Spike Based Algorithms
In the past decade, spike timing dependent plasticity (STDP) has been a popular unsupervised learning method due to its biological plausibility [93–95]. STDP mechanism
depends on the timing difference between the pre-synaptic and post-synaptic spikes
to adjust the synaptic weight. In the simple, doublet STDP [96], when a post-synaptic
spike happens after a pre-synaptic spike has arrived (pre-post event), then the weight
of the synapse increases i.e. synaptic potentiation takes place; whereas, if a postsynaptic spike happens before a pre-synaptic spike (post-pre event), then the weight
of the synapse decreases, i.e. depotentiation takes place. Like the doublet STDP, there
is another variant of STDP called the triplet STDP [97], whereby, three spike events
are considered (pre-post-pre, post-pre-post etc.). Apart from the time based STDP
there are other popular onchip learning rules based on spike rates like spike driven
synaptic plasticity (SDSP) [98], Bienenstock, Cooper, and Munro (BCM) rule [99].
Over the past decade, researchers have tried to implement the above-mentioned
plasticity rules onto integrated circuits [100–106]. There are CMOS devices such
as the floating gate MOSFET or nano-technology devices such as the memristors,
Resistive Random-Access Memories (RRAM), Phase Change Memories (PCM) and
Spin-Transfer Torque Magnetic Random-Access Memories (STT-MRAMs) used for
the implementation of artificial synapses. One of the challenges is to integrate these
nano-technology devices with CMOS. The characteristics of high synaptic density
on neuromorphic hardware has to be compromised. Interfacing circuitry or voltage
generators play a major role in mimicking the plasticity behaviours mentioned above
using a single two or three terminal devices. Understanding the device physics
becomes the key for the implementation of artificial synapses, especially while using
any of the technologies such as floating gate MOSFET, memristors or the more recent
spin devices to implement plasticity rules.
7 Conclusion and Outlook
The current state-of-the-art devices have demonstrated promising characteristics to
fulfil certain aspects of an ideal synaptic device. However, the presence of the tradeoffs among the device properties has raised a significant challenge in the array and
system level implementation, especially towards applications that require a relatively large scale NN. The importance of the gradual weight update with linear and
symmetrical weight modulations has been emphasized and thoroughly discussed for
P. A. Dananjaya et al.
5. As mentioned in [92], pooling layers can be avoided in deep neural network.
Hence, even though there are techniques to convert pooling layers in SDNN, we
can remove these layers from the DNN for simplicity sake.
6 On-Chip Learning on RRAM Based Neuromorphic
Hardware: Spike Based Algorithms
In the past decade, spike timing dependent plasticity (STDP) has been a popular unsupervised learning method due to its biological plausibility [93–95]. STDP mechanism
depends on the timing difference between the pre-synaptic and post-synaptic spikes
to adjust the synaptic weight. In the simple, doublet STDP [96], when a post-synaptic
spike happens after a pre-synaptic spike has arrived (pre-post event), then the weight
of the synapse increases i.e. synaptic potentiation takes place; whereas, if a postsynaptic spike happens before a pre-synaptic spike (post-pre event), then the weight
of the synapse decreases, i.e. depotentiation takes place. Like the doublet STDP, there
is another variant of STDP called the triplet STDP [97], whereby, three spike events
are considered (pre-post-pre, post-pre-post etc.). Apart from the time based STDP
there are other popular onchip learning rules based on spike rates like spike driven
synaptic plasticity (SDSP) [98], Bienenstock, Cooper, and Munro (BCM) rule [99].
Over the past decade, researchers have tried to implement the above-mentioned
plasticity rules onto integrated circuits [100–106]. There are CMOS devices such
as the floating gate MOSFET or nano-technology devices such as the memristors,
Resistive Random-Access Memories (RRAM), Phase Change Memories (PCM) and
Spin-Transfer Torque Magnetic Random-Access Memories (STT-MRAMs) used for
the implementation of artificial synapses. One of the challenges is to integrate these
nano-technology devices with CMOS. The characteristics of high synaptic density
on neuromorphic hardware has to be compromised. Interfacing circuitry or voltage
generators play a major role in mimicking the plasticity behaviours mentioned above
using a single two or three terminal devices. Understanding the device physics
becomes the key for the implementation of artificial synapses, especially while using
any of the technologies such as floating gate MOSFET, memristors or the more recent
spin devices to implement plasticity rules.
7 Conclusion and Outlook
The current state-of-the-art devices have demonstrated promising characteristics to
fulfil certain aspects of an ideal synaptic device. However, the presence of the tradeoffs among the device properties has raised a significant challenge in the array and
system level implementation, especially towards applications that require a relatively large scale NN. The importance of the gradual weight update with linear and
symmetrical weight modulations has been emphasized and thoroughly discussed for