Chapter 9
Dynamical Systems and Neural Networks
Abstract Neural networks are a way of expressing a variety of nonlinear functions,
but can also be thought of as waves of information propagating between layers. In
this chapter, we show that such multi-layer propagation can be interpreted as the
time evolution of dynamical systems, and hence of Hamiltonian systems, and look
at the close relationship between the fundamental concept of “time evolution” in
physics and deep neural networks.
If a variety of physical systems can be represented using neural networks, it will
greatly open the possibility of applying the systems to machine learning for analysis.
In physics, differential equations are the basic equations due to the concept of
locality and causality, so it is necessary to find out what differential equations
allow the representation of neural networks. In machine learning, there are so many
types of neural networks, and a new network structure is proposed mainly from the
viewpoint of improving learning efficiency, so the correspondence to differential
equations is not clear. In other words, the network obtained by discretizing the
differential equation has a different intention from the neural network for machine
learning and deep learning.
In this chapter, we review the relationship between differential equations and
neural networks, and discuss, in particular, Hamiltonian dynamical systems, and
which Hamiltonians allow the structure of a typical neural network.
9.1 Differential Equations and Neural Networks
First, let us define what we call “a typical neural network” in this chapter. Consider
a neural network given in Fig. 9.1. The layers are arranged from left to right, and
each layer has a vector x i of the same dimension (the subscript i labels the elements
of the vector). Between layers, the linear transformation x i → J ij x j and the local
nonlinear transformation x i → σ (x i ) by the activation function act in order. With
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021
A. Tanaka et al., Deep Learning and Physics, Mathematical Physics Studies,
https://doi.org/10.1007/978-981-33-6108-9_9
147
Dynamical Systems and Neural Networks
Abstract Neural networks are a way of expressing a variety of nonlinear functions,
but can also be thought of as waves of information propagating between layers. In
this chapter, we show that such multi-layer propagation can be interpreted as the
time evolution of dynamical systems, and hence of Hamiltonian systems, and look
at the close relationship between the fundamental concept of “time evolution” in
physics and deep neural networks.
If a variety of physical systems can be represented using neural networks, it will
greatly open the possibility of applying the systems to machine learning for analysis.
In physics, differential equations are the basic equations due to the concept of
locality and causality, so it is necessary to find out what differential equations
allow the representation of neural networks. In machine learning, there are so many
types of neural networks, and a new network structure is proposed mainly from the
viewpoint of improving learning efficiency, so the correspondence to differential
equations is not clear. In other words, the network obtained by discretizing the
differential equation has a different intention from the neural network for machine
learning and deep learning.
In this chapter, we review the relationship between differential equations and
neural networks, and discuss, in particular, Hamiltonian dynamical systems, and
which Hamiltonians allow the structure of a typical neural network.
9.1 Differential Equations and Neural Networks
First, let us define what we call “a typical neural network” in this chapter. Consider
a neural network given in Fig. 9.1. The layers are arranged from left to right, and
each layer has a vector x i of the same dimension (the subscript i labels the elements
of the vector). Between layers, the linear transformation x i → J ij x j and the local
nonlinear transformation x i → σ (x i ) by the activation function act in order. With
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021
A. Tanaka et al., Deep Learning and Physics, Mathematical Physics Studies,
https://doi.org/10.1007/978-981-33-6108-9_9
147