2
1 Forewords: Machine Learning and Physics
Fig. 1.1 Physics, machine learning, and information. Do they form a triangle?
1.1 Introduction to Information Theory
Quantifying information
This chapter explores the relationship between physics and machine learning
using information theory as a bridge. For that, we need to define exactly what
“information” is. What is “information” in the first place? First, read the following
two sentences [3]:
• Humans teach gorillas how to add numbers.
(1.1)
• Gorillas teach humans how to add numbers.
(1.2)
Which of these two sentences can be said to have more information? (1.1) is
not surprising, if true, because it is possible, regardless of whether the gorilla
understands it or not. You may have actually heard such news. On the other hand,
if (1.2) is true, it is quite surprising. Then one will change one’s mind and remember
that there could be a gorilla which can teach humans. This may be said to be
increasing the amount of information. In other words [4],
Amount of information = Extent of surprise.
(1.3)
Let us go ahead with this policy anyway. The greater the surprise, the less likely it
is to happen. In addition, if we try to make “information increase by addition”, with