Canonical Distribution in Statistical Mechanics
53
So, this neural network can express up to a cubic function, with seven parameters.
In the case of a general number of units, fitting is performed by a cubic function
using 3N unit + 1 parameters.
Conversely, considering the case where the number of units is 1 but with two
hidden layers, we find a ninth-order function,
f 2h-NN (x) = j L · σ act (J σ act (j 0 x + b 0 ) + b 1 ) + b L
(3.90)
= j L (j (j 0 x + b 0 )
3
+ b 1 )
3
+ b L .
(3.91)
The number of parameters is 6. In general, when the order of the activation function
is n and the number of hidden layers is h, the maximum order of the expressed
function is x (h+1)n . We can see that all orders appear when the activation function
is an odd function.
According to this model, the complexity of the functions that can be expressed
does not change even if the number of units in the middle layer is increased with
just three layers. On the other hand, if we increase the number of layers with an
activation function with n > 1, the complexity of the function causes a combinatorial explosion. In other words, more complex functions can be represented by neural
networks.
Finally, it should be emphasized that the discussion here is only with a model for
intuitive understanding. In particular, in the case of the activation function ReLU,
which is very often used, the argument here does not hold because of its linearity.
Nevertheless, since ReLU is also subject to the universal approximation [35] in fact,
it can be said that deep learning can be used with confidence by trusting the universal
approximation.
Column: Statistical Mechanics and Quantum Mechanics
Canonical Distribution in Statistical Mechanics
The statistical mechanics used in this book is called canonical distribution for
a given temperature. The physical situation is as follows. Suppose we have some
physical degrees of freedom, such as a spin or a particle position. Denote it as d.
Since this is a physical degree of freedom, it should have energy which depends on
its value. Let it be H (d). When this system is immersed in an environment with a
temperature of T (called a heat-bath), the probability of achieving d with the energy
H (d) is known to be
P (d) =
e
−
H (d)
k B T
Z
.
(3.92)
53
So, this neural network can express up to a cubic function, with seven parameters.
In the case of a general number of units, fitting is performed by a cubic function
using 3N unit + 1 parameters.
Conversely, considering the case where the number of units is 1 but with two
hidden layers, we find a ninth-order function,
f 2h-NN (x) = j L · σ act (J σ act (j 0 x + b 0 ) + b 1 ) + b L
(3.90)
= j L (j (j 0 x + b 0 )
3
+ b 1 )
3
+ b L .
(3.91)
The number of parameters is 6. In general, when the order of the activation function
is n and the number of hidden layers is h, the maximum order of the expressed
function is x (h+1)n . We can see that all orders appear when the activation function
is an odd function.
According to this model, the complexity of the functions that can be expressed
does not change even if the number of units in the middle layer is increased with
just three layers. On the other hand, if we increase the number of layers with an
activation function with n > 1, the complexity of the function causes a combinatorial explosion. In other words, more complex functions can be represented by neural
networks.
Finally, it should be emphasized that the discussion here is only with a model for
intuitive understanding. In particular, in the case of the activation function ReLU,
which is very often used, the argument here does not hold because of its linearity.
Nevertheless, since ReLU is also subject to the universal approximation [35] in fact,
it can be said that deep learning can be used with confidence by trusting the universal
approximation.
Column: Statistical Mechanics and Quantum Mechanics
Canonical Distribution in Statistical Mechanics
The statistical mechanics used in this book is called canonical distribution for
a given temperature. The physical situation is as follows. Suppose we have some
physical degrees of freedom, such as a spin or a particle position. Denote it as d.
Since this is a physical degree of freedom, it should have energy which depends on
its value. Let it be H (d). When this system is immersed in an environment with a
temperature of T (called a heat-bath), the probability of achieving d with the energy
H (d) is known to be
P (d) =
e
−
H (d)
k B T
Z
.
(3.92)