158
10 Spinglass and Neural Networks
10.1 Hopfield Model and Spinglass
Let us introduce the Hopfield model, a brain model of long-term memory and
association. The neural circuit is composed of the mutual connection of neural cells
(neurons). Neurons are connected at synapses, and neurons can have firing and nonfiring states. When one neuron is firing, signals are transmitted through the synapse
to the next neuron. The next neuron fires when the weighted sum of all incoming
electrical signals exceeds a certain threshold.
We model such a system as follows. In a system in which N neurons are
collected, the state of each neuron is {s i } (i = 1, 2, · · · , N), and the firing state
of the i–th neuron is defined as s i = 1 and the non-firing state as s i = 0. Then, write
the synaptic strength between the i–th neuron and the j –th neuron as J ij . The rule
that determines that the input changes the state s i to the state s
i is
s i → s
i ≡ θ
⎛
⎝
j
J ij s j − h i
⎞
⎠ .
(10.1)
Here, h i is the threshold, and θ(x) is the step function:
θ(x) =
1 (x > 0) ,
0 (x ≤ 0) .
(10.2)
Therefore, when the weighted sum of input signals
j J ij s j exceeds the threshold
h i , it fires: s
i = 1.
The rule of the Hopfield model (10.1) appears to be the same as those of the
neural networks with the inter-layer data propagation, as you look back at the neural
networks that have appeared so far, if you regard the step function as an activation
function. So how is the Hopfield model different from deep learning?
First, the deep neural network used for ordinary deep learning has a hierarchical
layer structure, and adopts a rule of successively propagating from the input layer to
the output layer. On the other hand, in the Hopfield model, one neuron is extracted
at random and only that neuron is updated with the rule (10.1). Then one repeats
the operation of taking out randomly again. This is based on the idea that updates in
the real brain are not perfectly synchronized in time. Therefore, the Hopfield model
does not have the framework of propagation through the layers. 2
Related to this, the Hopfield model assumes the following:
J ij = J ji .
(10.3)
2 The Hopfield model is similar to the Boltzmann machine, and a Boltzmann machine that does not
allow intra-layer coupling is called a restricted Boltzmann machine (RBM). See Chap. 6.
10 Spinglass and Neural Networks
10.1 Hopfield Model and Spinglass
Let us introduce the Hopfield model, a brain model of long-term memory and
association. The neural circuit is composed of the mutual connection of neural cells
(neurons). Neurons are connected at synapses, and neurons can have firing and nonfiring states. When one neuron is firing, signals are transmitted through the synapse
to the next neuron. The next neuron fires when the weighted sum of all incoming
electrical signals exceeds a certain threshold.
We model such a system as follows. In a system in which N neurons are
collected, the state of each neuron is {s i } (i = 1, 2, · · · , N), and the firing state
of the i–th neuron is defined as s i = 1 and the non-firing state as s i = 0. Then, write
the synaptic strength between the i–th neuron and the j –th neuron as J ij . The rule
that determines that the input changes the state s i to the state s
i is
s i → s
i ≡ θ
⎛
⎝
j
J ij s j − h i
⎞
⎠ .
(10.1)
Here, h i is the threshold, and θ(x) is the step function:
θ(x) =
1 (x > 0) ,
0 (x ≤ 0) .
(10.2)
Therefore, when the weighted sum of input signals
j J ij s j exceeds the threshold
h i , it fires: s
i = 1.
The rule of the Hopfield model (10.1) appears to be the same as those of the
neural networks with the inter-layer data propagation, as you look back at the neural
networks that have appeared so far, if you regard the step function as an activation
function. So how is the Hopfield model different from deep learning?
First, the deep neural network used for ordinary deep learning has a hierarchical
layer structure, and adopts a rule of successively propagating from the input layer to
the output layer. On the other hand, in the Hopfield model, one neuron is extracted
at random and only that neuron is updated with the rule (10.1). Then one repeats
the operation of taking out randomly again. This is based on the idea that updates in
the real brain are not perfectly synchronized in time. Therefore, the Hopfield model
does not have the framework of propagation through the layers. 2
Related to this, the Hopfield model assumes the following:
J ij = J ji .
(10.3)
2 The Hopfield model is similar to the Boltzmann machine, and a Boltzmann machine that does not
allow intra-layer coupling is called a restricted Boltzmann machine (RBM). See Chap. 6.