150
9 Dynamical Systems and Neural Networks
always in that form. We will look at these subtleties in more detail later in the case
of Hamiltonian dynamical systems.
Now, as a further generalization of ResNet, there is a neural network called
RevNet (reversible residual network) [114]. The RevNet is again a residual learning,
but has a symmetric form:
x
(n+1)
i
= f (J ij y
(n)
j ) + x
(n)
i ,
(9.7)
y
(n+1)
i
= g(J ij x
(n+1)
j
) + y
(n)
i .
(9.8)
With the same analogy, it can be seen that this neural network is a discretized version
of the following dynamical system:
˙
x(t) = f (y(t)) , ˙
y(t) = g(x(t)) .
(9.9)
Reversible means that you can return to the input layer in order from the data output
value at the last layer. If you look closely at the right-hand side of (9.8), you find
x
(n+1)
j
instead of x
(n)
j . With this, if output data (y
(n+1)
i
, x
(n+1)
i
) is given, first y
(n)
i
is obtained through (9.8), then (9.7) returns x
(n)
i , and so on. Being able to return is
actually related to the amount of memory used for learning. In the normal neural
network backpropagation method, the values of the weight at each layer must be
stored in memory. However, in the case of a reversible neural network, the values of
the weight at each layer can be recalculated from the data of the last layer, so there
is no need to store the weight in memory, and efficient learning concerning memory
consumption is provided.
Furthermore, being reversible is deeply related to differential equations in
dynamical systems. In dynamical systems, a characteristic behavior called chaos is
one of the important research subjects. Chaos is the sensitivity to the initial values.
In terms of neural networks, the output value changes completely even if the initial
input data is slightly perturbed. A neural network obtained by a discretization of
a chaotic dynamical system is susceptible to perturbation of the input data, that
is, learning is considered to be difficult. A neural network based on a dynamical
system without chaos is called a “stable neural network” [115]. On the other hand,
in a reversible neural network, a similar problem can occur if there is chaotic
behavior when considering propagation in the reverse direction. In this case, rather,
it corresponds to the situation where the initial value difference completely collapses
before reaching the final layer (the output data does not change due to the difference
in the input data). It is related to the existence of an attractor. The degree of chaos
in a dynamical system is measured by a constant called the Lyapunov exponent.
The system is chaotic when it is positive, while the system is “collapsing” when it is
negative. Dynamical systems in which the Lyapunov exponent has no real part are
appropriate for reversible neural networks [116].
9 Dynamical Systems and Neural Networks
always in that form. We will look at these subtleties in more detail later in the case
of Hamiltonian dynamical systems.
Now, as a further generalization of ResNet, there is a neural network called
RevNet (reversible residual network) [114]. The RevNet is again a residual learning,
but has a symmetric form:
x
(n+1)
i
= f (J ij y
(n)
j ) + x
(n)
i ,
(9.7)
y
(n+1)
i
= g(J ij x
(n+1)
j
) + y
(n)
i .
(9.8)
With the same analogy, it can be seen that this neural network is a discretized version
of the following dynamical system:
˙
x(t) = f (y(t)) , ˙
y(t) = g(x(t)) .
(9.9)
Reversible means that you can return to the input layer in order from the data output
value at the last layer. If you look closely at the right-hand side of (9.8), you find
x
(n+1)
j
instead of x
(n)
j . With this, if output data (y
(n+1)
i
, x
(n+1)
i
) is given, first y
(n)
i
is obtained through (9.8), then (9.7) returns x
(n)
i , and so on. Being able to return is
actually related to the amount of memory used for learning. In the normal neural
network backpropagation method, the values of the weight at each layer must be
stored in memory. However, in the case of a reversible neural network, the values of
the weight at each layer can be recalculated from the data of the last layer, so there
is no need to store the weight in memory, and efficient learning concerning memory
consumption is provided.
Furthermore, being reversible is deeply related to differential equations in
dynamical systems. In dynamical systems, a characteristic behavior called chaos is
one of the important research subjects. Chaos is the sensitivity to the initial values.
In terms of neural networks, the output value changes completely even if the initial
input data is slightly perturbed. A neural network obtained by a discretization of
a chaotic dynamical system is susceptible to perturbation of the input data, that
is, learning is considered to be difficult. A neural network based on a dynamical
system without chaos is called a “stable neural network” [115]. On the other hand,
in a reversible neural network, a similar problem can occur if there is chaotic
behavior when considering propagation in the reverse direction. In this case, rather,
it corresponds to the situation where the initial value difference completely collapses
before reaching the final layer (the output data does not change due to the difference
in the input data). It is related to the existence of an attractor. The degree of chaos
in a dynamical system is measured by a constant called the Lyapunov exponent.
The system is chaotic when it is positive, while the system is “collapsing” when it is
negative. Dynamical systems in which the Lyapunov exponent has no real part are
appropriate for reversible neural networks [116].