1.1 Introduction to Information Theory
3
P (event) the probability that an event will occur, we have 1
Amount of information of event A = − log P (event A).
(1.4)
When the probability is low, the amount of information is large.
Average amount of information
Let us further assume that various events A 1 , A 2 , . . . , A W occur with probabilities
p 1 , p 2 , . . . , p W , respectively. At this time, the amount of information of each event
is − log p i , and its expectation value
S information = −
W
i=1
p i log p i .
(1.5)
is called information entropy. 2 Let us take a concrete example of what information
entropy represents. Suppose there are W boxes, and let p i be the probability that the
ith box contains a treasure. Of course, we want to predict which box contains the
treasure as accurately as possible, but the predictability depends on the value of p i .
For example, if we know that the treasure is always in the first box, it is easy to
predict. The value of the information entropy for this case is zero:
p i =
1 (i = 1)
0 (other than that)
S information = 0.
(1.6)
On the other hand, if the probability is completely random,
p i =
1
W
S information = log W.
(1.7)
For this case, even if we know the probability, it is difficult to predict because we
do not know which box it is in. This time, the information entropy has a large value,
log W . In other words, the more difficult it is to predict, the greater the information
entropy. Therefore, the relation to the commonly referred to “information” is as
follows:
•
Little “information” ⇔ difficult to predict ⇔ large information entropy
A lot of “information” ⇔ easy to predict ⇔ small information entropy
(1.8)
1 In this book, as in many physics textbooks, the base of the logarithm is taken to be that of the
natural log, e = 2.718 · · · .
2 This quantity was first introduced in the context of information in the monumental paper by
C. Shannon on mathematics in communication [5]. He calls this quantity entropy, so physicists can
easily understand this concept.
3
P (event) the probability that an event will occur, we have 1
Amount of information of event A = − log P (event A).
(1.4)
When the probability is low, the amount of information is large.
Average amount of information
Let us further assume that various events A 1 , A 2 , . . . , A W occur with probabilities
p 1 , p 2 , . . . , p W , respectively. At this time, the amount of information of each event
is − log p i , and its expectation value
S information = −
W
i=1
p i log p i .
(1.5)
is called information entropy. 2 Let us take a concrete example of what information
entropy represents. Suppose there are W boxes, and let p i be the probability that the
ith box contains a treasure. Of course, we want to predict which box contains the
treasure as accurately as possible, but the predictability depends on the value of p i .
For example, if we know that the treasure is always in the first box, it is easy to
predict. The value of the information entropy for this case is zero:
p i =
1 (i = 1)
0 (other than that)
S information = 0.
(1.6)
On the other hand, if the probability is completely random,
p i =
1
W
S information = log W.
(1.7)
For this case, even if we know the probability, it is difficult to predict because we
do not know which box it is in. This time, the information entropy has a large value,
log W . In other words, the more difficult it is to predict, the greater the information
entropy. Therefore, the relation to the commonly referred to “information” is as
follows:
•
Little “information” ⇔ difficult to predict ⇔ large information entropy
A lot of “information” ⇔ easy to predict ⇔ small information entropy
(1.8)
1 In this book, as in many physics textbooks, the base of the logarithm is taken to be that of the
natural log, e = 2.718 · · · .
2 This quantity was first introduced in the context of information in the monumental paper by
C. Shannon on mathematics in communication [5]. He calls this quantity entropy, so physicists can
easily understand this concept.