11.2 Tensor Networks and Neural Networks
171
What about a feedforward neural network instead of a Boltzmann machine? As
we saw in Chap. 3, feedforward neural networks have a universal approximation
theorem, so any nonlinear function can be expressed, and in this sense any tensor
network can be expressed. However, constructing a neural network to represent a
given tensor network has some difficulties, as we describe below.
First, in the case of Fig. 11.3 right, two A’s are connected to the two legs of
tensor B, and the state of the tensor network includes a multiplication of A. In
general, tensors combine by multiplication, which creates nonlinearities, whereas
feedforward neural networks do not, and create nonlinearities in the form of
activation functions acting on a unit-by-unit basis. To make up for this difference,
you need to use an activation function in the form of a multiplication. 5
A tensor network that can be expressed as a neural network using such a new
activation function is a tree graph. A tensor network of graphs with loops inside
(which also includes what is called MERA 6 ), instead of a tree, is difficult to express
with a feedforward type. This is because the loop graph has two or more outputs
in the middle layer, and tensors connected to them appear in parallel (the tensor
product structure), but in a feedforward neural network, when tensors arranged in
parallel are recombined, it allows only one of them to be expressed with priority,
and the parallelism of the tensor is hindered.
As described above, tensor networks are different from neural networks in many
ways. However, each has its advantages and can be moved back and forth through
common concepts. As explained in Chap. 10, neural networks originate from
physical many-body systems such as Hopfield models, so they are compatible with
physical systems. In the future, interconnection of quantum many-body systems and
machine learning will advance in various forms.
5 There is also a study that uses product pooling to see a correspondence [128].
6 MERA stands for Multiscale Entanglement Renormalization Ansatz.
171
What about a feedforward neural network instead of a Boltzmann machine? As
we saw in Chap. 3, feedforward neural networks have a universal approximation
theorem, so any nonlinear function can be expressed, and in this sense any tensor
network can be expressed. However, constructing a neural network to represent a
given tensor network has some difficulties, as we describe below.
First, in the case of Fig. 11.3 right, two A’s are connected to the two legs of
tensor B, and the state of the tensor network includes a multiplication of A. In
general, tensors combine by multiplication, which creates nonlinearities, whereas
feedforward neural networks do not, and create nonlinearities in the form of
activation functions acting on a unit-by-unit basis. To make up for this difference,
you need to use an activation function in the form of a multiplication. 5
A tensor network that can be expressed as a neural network using such a new
activation function is a tree graph. A tensor network of graphs with loops inside
(which also includes what is called MERA 6 ), instead of a tree, is difficult to express
with a feedforward type. This is because the loop graph has two or more outputs
in the middle layer, and tensors connected to them appear in parallel (the tensor
product structure), but in a feedforward neural network, when tensors arranged in
parallel are recombined, it allows only one of them to be expressed with priority,
and the parallelism of the tensor is hindered.
As described above, tensor networks are different from neural networks in many
ways. However, each has its advantages and can be moved back and forth through
common concepts. As explained in Chap. 10, neural networks originate from
physical many-body systems such as Hopfield models, so they are compatible with
physical systems. In the future, interconnection of quantum many-body systems and
machine learning will advance in various forms.
5 There is also a study that uses product pooling to see a correspondence [128].
6 MERA stands for Multiscale Entanglement Renormalization Ansatz.