RRAM-Based Neuromorphic Computing Systems
407
time averaged firing rate of a spiking neuron must be correlated with the activation
value of the corresponding neuron in the ANN. The generic steps involved for network
conversion is in the following:
1. Choose a CNN to train.
2. Use ReLU for activation functions in the CNN.
3. Fix the bias to zero throughout training using stochastic gradient descent.
4. Save all the weights after training.
5. Replace neurons in the CNN with integrate and fire neurons without refractory
period.
6. Map the saved weights to the SNN.
7. Convert the input image to Poisson spike trains with firing rates proportional to
each pixel intensity value.
Challenging Factors and Solutions in Achieving High Conversion Accuracy
The issues affecting conversion accuracy as mentioned in [66] are: in the CNN the
weights and biases can be negative. Since input integration is a weighted sum of
inputs and the bias, the output can be negative. If the sigmoid function is used for
activation it may also be negative. It is difficult to represent negative activations in
the CNN on a SNN. It is also difficult to represent biases in the SNN. Two-layer
neural network is needed to implement spatial maxpooling in the SNN.
CNN to SNN mapping requires the input image to be converted to Poisson spike
trains with firing rates proportional to the pixel intensity value. As a result, the loss
of accuracy during conversion can happen due to the factors [88]: Input spikes are
not enough to result in threshold crossing, hence no output spike is emitted when
activation values in the CNN are below threshold. If the spiking neuron receives
too many input spikes in a single timestep or if some of its synaptic weights are
higher than threshold, then the spiking neuron should emit more than one spike per
timestep, which it cannot, and hence introducing error in the process. Due to the
non-uniformity of the spike trains or the stochastic nature of the spiking input, a
specific feature set could be over- or under- activated by incoming spikes.
An analysis of conversion and its theory is proposed in [91]. One on one
mapping of the spiking neuron and the activation function of the CNN reveals
that during threshold crossing, the membrane potential reached maybe of any
value above threshold. This error would accumulate over time. The solution to the
above-mentioned issues are the following:
1. As mentioned in [66], are to remove biases from convolution layers, use ReLU
as activation function and use spatial linear subsampling instead of maxpooling.
2. As mentioned in [88] use weight normalization.
3. As mentioned in [91] use reset by subtraction instead of reset to zero for spiking
neurons. Instead of removing biases from convolutional layers, a constant input
current can be applied to emulate the biases. Also apply normalization techniques.
4. As mentioned in [90], to reduce the variability of input spikes, the multi-bit
values of the input maybe fed directly into the first hidden layer and spikes are
then output henceforth.
407
time averaged firing rate of a spiking neuron must be correlated with the activation
value of the corresponding neuron in the ANN. The generic steps involved for network
conversion is in the following:
1. Choose a CNN to train.
2. Use ReLU for activation functions in the CNN.
3. Fix the bias to zero throughout training using stochastic gradient descent.
4. Save all the weights after training.
5. Replace neurons in the CNN with integrate and fire neurons without refractory
period.
6. Map the saved weights to the SNN.
7. Convert the input image to Poisson spike trains with firing rates proportional to
each pixel intensity value.
Challenging Factors and Solutions in Achieving High Conversion Accuracy
The issues affecting conversion accuracy as mentioned in [66] are: in the CNN the
weights and biases can be negative. Since input integration is a weighted sum of
inputs and the bias, the output can be negative. If the sigmoid function is used for
activation it may also be negative. It is difficult to represent negative activations in
the CNN on a SNN. It is also difficult to represent biases in the SNN. Two-layer
neural network is needed to implement spatial maxpooling in the SNN.
CNN to SNN mapping requires the input image to be converted to Poisson spike
trains with firing rates proportional to the pixel intensity value. As a result, the loss
of accuracy during conversion can happen due to the factors [88]: Input spikes are
not enough to result in threshold crossing, hence no output spike is emitted when
activation values in the CNN are below threshold. If the spiking neuron receives
too many input spikes in a single timestep or if some of its synaptic weights are
higher than threshold, then the spiking neuron should emit more than one spike per
timestep, which it cannot, and hence introducing error in the process. Due to the
non-uniformity of the spike trains or the stochastic nature of the spiking input, a
specific feature set could be over- or under- activated by incoming spikes.
An analysis of conversion and its theory is proposed in [91]. One on one
mapping of the spiking neuron and the activation function of the CNN reveals
that during threshold crossing, the membrane potential reached maybe of any
value above threshold. This error would accumulate over time. The solution to the
above-mentioned issues are the following:
1. As mentioned in [66], are to remove biases from convolution layers, use ReLU
as activation function and use spatial linear subsampling instead of maxpooling.
2. As mentioned in [88] use weight normalization.
3. As mentioned in [91] use reset by subtraction instead of reset to zero for spiking
neurons. Instead of removing biases from convolutional layers, a constant input
current can be applied to emulate the biases. Also apply normalization techniques.
4. As mentioned in [90], to reduce the variability of input spikes, the multi-bit
values of the input maybe fed directly into the first hidden layer and spikes are
then output henceforth.