8
1 Forewords: Machine Learning and Physics
to 1 as possible corresponds to reducing the relative entropy. We mean,
Relative entropy = Amount to measure how close the prediction q i is to the truth p i .
The properties of the relative entropy will be explained later in this book. The
relative entropy is called Kullback-Leibler divergence in information theory, and
is important in machine learning as can be seen here, as well as having many
mathematically interesting properties. The fact that (1.11) is approximately the same
as the Kullback-Leibler divergence is one of the consequences from large deviation
theory called Sanov’s (I. N. Sanov) theorem [12, 13].
1.4 Machine Learning and Physics
So far, we have briefly described the relationship between physics and information
theory, and the relationship between machine learning and information theory. Then,
there may be a connection between machine learning and physics in some sense. The
aforementioned Fig. 1.1 shows the concept: physics and information are connected,
and information and machine learning are connected. Then, how can physics and
machine learning be connected?
A thought experiment
Suppose we have a fairy here. A button and a clock are placed in front of the fairy,
and every minute, the fairy chooses to press the button or not. If the fairy presses
the button, the machine outputs 1. If the fairy does not press the button, the output
is 0. The result of “a special fairy” is the following sequence:
{1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, . . . }.
(1.20)
The following figure shows the result of this monotonous job for about 4 hours,
displayed in a matrix of 15 × 15:
1 Forewords: Machine Learning and Physics
to 1 as possible corresponds to reducing the relative entropy. We mean,
Relative entropy = Amount to measure how close the prediction q i is to the truth p i .
The properties of the relative entropy will be explained later in this book. The
relative entropy is called Kullback-Leibler divergence in information theory, and
is important in machine learning as can be seen here, as well as having many
mathematically interesting properties. The fact that (1.11) is approximately the same
as the Kullback-Leibler divergence is one of the consequences from large deviation
theory called Sanov’s (I. N. Sanov) theorem [12, 13].
1.4 Machine Learning and Physics
So far, we have briefly described the relationship between physics and information
theory, and the relationship between machine learning and information theory. Then,
there may be a connection between machine learning and physics in some sense. The
aforementioned Fig. 1.1 shows the concept: physics and information are connected,
and information and machine learning are connected. Then, how can physics and
machine learning be connected?
A thought experiment
Suppose we have a fairy here. A button and a clock are placed in front of the fairy,
and every minute, the fairy chooses to press the button or not. If the fairy presses
the button, the machine outputs 1. If the fairy does not press the button, the output
is 0. The result of “a special fairy” is the following sequence:
{1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, . . . }.
(1.20)
The following figure shows the result of this monotonous job for about 4 hours,
displayed in a matrix of 15 × 15: