10.2 Memory and Attractor
161
10.2 Memory and Attractor
In the Hopfield model, “memory” refers to the set of spin values {S i = a i }. The
information of the memory is stored in a synapse J ij . Assume that the synapses
have the following values (α > 0), respectively, by a certain mechanism which we
explain later:
J ij = αa i a j .
(10.11)
Then the memory {S i = a i } is a fixed point under the update rule (10.1):
S
i = sgn
⎛
⎝
j
J ij a j
⎞
⎠ = sgn
⎛
⎝
j
αa i a j a j
⎞
⎠ = sgn (α(N + 1)a i ) = a i .
(10.12)
Therefore, once the spin state falls into {S i = a i }, it will stay there.
Is this state stable? When the state changes a little from the fixed point by some
external perturbation, will it return to the original fixed point? Let us evaluate the
Lyapunov function (10.9),
E = −
α
2
ij
a i a i a j a j = −
α
2
(N + 1)
2 .
(10.13)
This has a very large negative value. In fact, the value of O(N 2 ) is the maximum
magnitude that the Lyapunov function can have, and the memory {S i = a i } is at the
bottom of a very deep valley, so is a stable fixed point. In other words, when starting
from an arbitrary firing state, the state is updated according to the rule (10.1), and
finally reaches a stable fixed point {S i = a i } and stops moving. Such a fixed point
is called an attractor, from the viewpoint of dynamical systems. Reaching the
attractor is interpreted as a mechanism of “recalling” memory.
For an important memory the Lyapunov function must fall into deeper valleys.
And in addition, there are many shallow valleys because of the frustration. Even if it
gets stuck in such a shallow valley, it is expected that various external perturbations
will finally lead to the deep valley.
Of course, there are many patterns to remember, not just a single pattern {S i =
a i }. So, let us consider a situation where you want to store M kinds of patterns
{S i = a
(m)
i } (m = 1, 2, · · · , M). Then the appropriate coupling is
J ij =
α
M
m
a
(m)
i a
(m)
j .
(10.14)
In particular, it is assumed that the patterns of memory are orthogonal to each other:
i
a
(m)
i a
(n)
i = (N + 1)δ m,n .
(10.15)
161
10.2 Memory and Attractor
In the Hopfield model, “memory” refers to the set of spin values {S i = a i }. The
information of the memory is stored in a synapse J ij . Assume that the synapses
have the following values (α > 0), respectively, by a certain mechanism which we
explain later:
J ij = αa i a j .
(10.11)
Then the memory {S i = a i } is a fixed point under the update rule (10.1):
S
i = sgn
⎛
⎝
j
J ij a j
⎞
⎠ = sgn
⎛
⎝
j
αa i a j a j
⎞
⎠ = sgn (α(N + 1)a i ) = a i .
(10.12)
Therefore, once the spin state falls into {S i = a i }, it will stay there.
Is this state stable? When the state changes a little from the fixed point by some
external perturbation, will it return to the original fixed point? Let us evaluate the
Lyapunov function (10.9),
E = −
α
2
ij
a i a i a j a j = −
α
2
(N + 1)
2 .
(10.13)
This has a very large negative value. In fact, the value of O(N 2 ) is the maximum
magnitude that the Lyapunov function can have, and the memory {S i = a i } is at the
bottom of a very deep valley, so is a stable fixed point. In other words, when starting
from an arbitrary firing state, the state is updated according to the rule (10.1), and
finally reaches a stable fixed point {S i = a i } and stops moving. Such a fixed point
is called an attractor, from the viewpoint of dynamical systems. Reaching the
attractor is interpreted as a mechanism of “recalling” memory.
For an important memory the Lyapunov function must fall into deeper valleys.
And in addition, there are many shallow valleys because of the frustration. Even if it
gets stuck in such a shallow valley, it is expected that various external perturbations
will finally lead to the deep valley.
Of course, there are many patterns to remember, not just a single pattern {S i =
a i }. So, let us consider a situation where you want to store M kinds of patterns
{S i = a
(m)
i } (m = 1, 2, · · · , M). Then the appropriate coupling is
J ij =
α
M
m
a
(m)
i a
(m)
j .
(10.14)
In particular, it is assumed that the patterns of memory are orthogonal to each other:
i
a
(m)
i a
(n)
i = (N + 1)δ m,n .
(10.15)