162
10 Spinglass and Neural Networks
Then, as before, you can see that each memory {S i = a
(m)
i } is a fixed point under
the update rule (10.1),
S
i = sgn
⎛
⎝
j
J ij a
(m)
j
⎞
⎠
= sgn
⎛
⎝
n
j
α
M
a
(n)
i a
(n)
j a
(m)
j
⎞
⎠
= sgn
α(N + 1)
M
a
(m)
i
= a
(m)
i
.
(10.16)
Here, the orthogonal condition of memory (10.15) was used. Also, we can evaluate
the Lyapunov function (10.9) about the memory {S i = a
(m)
i } as
E = −
α
2M
i,j
n
a
(n)
i a
(n)
j
a
(m)
i a
(m)
j
= −
α
2M
(N + 1)
2 .
(10.17)
After all, each memory is at the bottom of a deep valley of O(N 2 ).
If you start updating from any initial conditions, you are expected to eventually
reach a “near” deep valley. This can be interpreted as a mechanism whereby past
memories are awakened when a near phenomenon is perceived. Here, â ˘
AIJnearâ ˘
A ˙ I
specifically depends on the details of the structure of the Lyapunov function. Due to
the orthogonality condition of memory (10.15), it is expected that the “near” may
mean that the inner product with a pattern {S i = a
(m)
i } is large. This is because the
evaluation method of the Lyapunov function is the inner product, as in (10.17).
So far, we have seen that when the coupling J ij takes the value (10.11), a
deep valley is realized, which behaves as a stable fixed point. Then, how is the
value (10.11) itself realized? In supervised machine learning, neural networks
are trained by updating weights using the gradient descent method, while the
Hopfield model does not look at the correlation between an input and an output.
The realization of (10.11) is thought to be performed by a theory generally called
Hebbian learning theory. Hebb’s rule is a rule that synapses are strengthened when
both of the neurons they connect are firing, while they will attenuate when the
neurons did not fire. From this idea, for example, an equation like the following
is expected:
μ
d
dt
J ij (t) = −J ij (t) + αa i a j .
(10.18)
10 Spinglass and Neural Networks
Then, as before, you can see that each memory {S i = a
(m)
i } is a fixed point under
the update rule (10.1),
S
i = sgn
⎛
⎝
j
J ij a
(m)
j
⎞
⎠
= sgn
⎛
⎝
n
j
α
M
a
(n)
i a
(n)
j a
(m)
j
⎞
⎠
= sgn
α(N + 1)
M
a
(m)
i
= a
(m)
i
.
(10.16)
Here, the orthogonal condition of memory (10.15) was used. Also, we can evaluate
the Lyapunov function (10.9) about the memory {S i = a
(m)
i } as
E = −
α
2M
i,j
n
a
(n)
i a
(n)
j
a
(m)
i a
(m)
j
= −
α
2M
(N + 1)
2 .
(10.17)
After all, each memory is at the bottom of a deep valley of O(N 2 ).
If you start updating from any initial conditions, you are expected to eventually
reach a “near” deep valley. This can be interpreted as a mechanism whereby past
memories are awakened when a near phenomenon is perceived. Here, â ˘
AIJnearâ ˘
A ˙ I
specifically depends on the details of the structure of the Lyapunov function. Due to
the orthogonality condition of memory (10.15), it is expected that the “near” may
mean that the inner product with a pattern {S i = a
(m)
i } is large. This is because the
evaluation method of the Lyapunov function is the inner product, as in (10.17).
So far, we have seen that when the coupling J ij takes the value (10.11), a
deep valley is realized, which behaves as a stable fixed point. Then, how is the
value (10.11) itself realized? In supervised machine learning, neural networks
are trained by updating weights using the gradient descent method, while the
Hopfield model does not look at the correlation between an input and an output.
The realization of (10.11) is thought to be performed by a theory generally called
Hebbian learning theory. Hebb’s rule is a rule that synapses are strengthened when
both of the neurons they connect are firing, while they will attenuate when the
neurons did not fire. From this idea, for example, an equation like the following
is expected:
μ
d
dt
J ij (t) = −J ij (t) + αa i a j .
(10.18)