1.3 Machine Learning and Information Theory
5
Maxwell’s demon
Maxwell’s demon is a virtual devil that appears in a thought experiment and breaks
the second law of thermodynamics, at least superficially; it was introduced by James
Clerk Maxwell. Assume a box contains a gas at temperature T ; a partition plate with
a small hole is inserted in the middle of this box. The hole is small enough to allow
one gas molecule to pass through. There is also a switch next to the hole, which can
be pressed to close or open the hole. According to statistical mechanics, in a gas
with a gas molecular mass m and temperature T , molecules with speed v exist with
probability proportional to e
−
mv 2
2k B T . This means that gas molecules of various speeds
are flying: there are some fast molecules, and some slow molecules. Assuming that
a small devil is sitting near the hole in the partition plate, the devil lets “only the
fast molecules coming from the right go through the hole to the left, only the slow
molecules from the left go through the hole to the right.” As a result, relatively slow
molecules remain on the right, and fast-moving molecules gather on the left. That is,
if the right temperature is T R and the left temperature is T L , it means that T R < T L .
Using the ideal gas equation of state, p R < p L , so the force F = p L −p R acts in the
direction of the room on the right. If we allow the partition to slide and attach some
string to it, then this F can do some work. There should be something strange in
this story, because it can convert heat to work endlessly, which means that we have
created a perpetual motion machine of the second kind. In recent years, information
theory has been shown to be an effective way to fill this gap [8].
In this way, relations with information theory have been more and more
prominent in various aspects of theoretical physics. Wheeler, famous for his work on
gravity theory, even says “it from bit” (physical existence come from information)
[9]. In recent years, not only information theory based on ordinary probability
theory, but also research in a field called quantum information theory based on
quantum mechanics with interfering probabilities has been actively performed, and
various developments are reported daily. This is not covered in this book, but
interested readers may want to read [10] and other books.
1.3 Machine Learning and Information Theory
One way to mathematically formulate machine learning methods is based on
probability theory. In fact, this book follows that format. One of the nice things about
this method is that we can still use various concepts of information theory, including
entropy. The purpose of machine learning is “to predict the future unknown from
some experience,” and when formulating this mathematically, as in (1.8), it is
necessary to deal with a quantity measuring the degree of difficulty in predicting
things. However, the “predictability” described in (1.8) is based on the assumption
that we know the probability p i of the occurrence of the event. Even in machine
learning, it is assumed that there exists p i behind the phenomenon, while its value
5
Maxwell’s demon
Maxwell’s demon is a virtual devil that appears in a thought experiment and breaks
the second law of thermodynamics, at least superficially; it was introduced by James
Clerk Maxwell. Assume a box contains a gas at temperature T ; a partition plate with
a small hole is inserted in the middle of this box. The hole is small enough to allow
one gas molecule to pass through. There is also a switch next to the hole, which can
be pressed to close or open the hole. According to statistical mechanics, in a gas
with a gas molecular mass m and temperature T , molecules with speed v exist with
probability proportional to e
−
mv 2
2k B T . This means that gas molecules of various speeds
are flying: there are some fast molecules, and some slow molecules. Assuming that
a small devil is sitting near the hole in the partition plate, the devil lets “only the
fast molecules coming from the right go through the hole to the left, only the slow
molecules from the left go through the hole to the right.” As a result, relatively slow
molecules remain on the right, and fast-moving molecules gather on the left. That is,
if the right temperature is T R and the left temperature is T L , it means that T R < T L .
Using the ideal gas equation of state, p R < p L , so the force F = p L −p R acts in the
direction of the room on the right. If we allow the partition to slide and attach some
string to it, then this F can do some work. There should be something strange in
this story, because it can convert heat to work endlessly, which means that we have
created a perpetual motion machine of the second kind. In recent years, information
theory has been shown to be an effective way to fill this gap [8].
In this way, relations with information theory have been more and more
prominent in various aspects of theoretical physics. Wheeler, famous for his work on
gravity theory, even says “it from bit” (physical existence come from information)
[9]. In recent years, not only information theory based on ordinary probability
theory, but also research in a field called quantum information theory based on
quantum mechanics with interfering probabilities has been actively performed, and
various developments are reported daily. This is not covered in this book, but
interested readers may want to read [10] and other books.
1.3 Machine Learning and Information Theory
One way to mathematically formulate machine learning methods is based on
probability theory. In fact, this book follows that format. One of the nice things about
this method is that we can still use various concepts of information theory, including
entropy. The purpose of machine learning is “to predict the future unknown from
some experience,” and when formulating this mathematically, as in (1.8), it is
necessary to deal with a quantity measuring the degree of difficulty in predicting
things. However, the “predictability” described in (1.8) is based on the assumption
that we know the probability p i of the occurrence of the event. Even in machine
learning, it is assumed that there exists p i behind the phenomenon, while its value