12.1 Defining Information
545
question which has only two possible answers, you get less information from the
answer than if there were a thousand possible answers. We will use the count W for
the number of ways a message could be clearly received, and I for the information
gained when that message is clearly received.
The second property means that I is logarithmically dependent on W : I =
k ln W . The argument is this: Two independent messages have W = W 1 W 2 ways of
being clearly received, where W i is the number of ways of clearly receiving message
i. We require I = I 1 + I 2 , i.e. I (W 1 W 2 ) = I (W 1 ) + I (W 2 ). Only logarithms satisfy
this condition for any W 1 and W 2 .
The third property makes k in I = k ln W take the value 1/ ln 2, so that
I = log 2 W . This choice of k is quite convenient in quantifying the information
sent and received by electronic digital devices, because only two symbols are used,
represented in wires by on or off electric pressure. In the wire, a message is then
a series of timed pulses, on or off. Within a given time interval, no pulse is a ‘0’
symbol, and a positive pulse is a ‘1’ symbol. These messages can be used to code
letters of an alphabet, sounds, or positions, intensities, colors within images, images
themselves, and so forth.
If a message is scrambled by noise, we lose information. Let W a represent the
number of ways a message, once received, can be misinterpreted. With ambiguity,
we have a reduced number of different messages we can read. If W c is the number of
ways the message could have been composed, then the number of different messages
which could be received and clearly interpreted is W = W c /W a , making the
information received: I = ln 2 W c −ln 2 W a , as should be expected, in that ambiguity
has reduced the information in a message.
Now suppose a message can use L symbols from the variety V =
{a 1 , a 2 , . . . , a L } in a sequence. Let the probability of receiving the symbol a l
be p l , and assume there are no correlations between the received symbols. The
average information gathered when the symbol a l is detected will be log 2 (1/p l ).
After a large number N of symbols within the variety V are received, we can detect
about Np l occurrences of the symbol a l , so that the information received will be
I =
Np l log 2 (1/p l ). The average information per symbol is then
I/N = −
p l log 2 p l .
(12.1)
A simple application of this result is to the game of 21 questions, in which you
try to guess what object in a room a person has imagined by asking no more than
21 “yes-no” questions. The best strategy, the experienced player knows, is to ask
only questions which have about a 50–50 chance of being answered yes. You will
then get the most information from the answer to the question. This observation
follows from Shannon’s information Eq. (12.1), which is maximum, for varying p i
under the constraint
p l = 1, when all the p l are equal. 3 With only two possible
answers, p l = 1/2.
3 To prove this statement, use the Lagrange-multiplier method (described in Appendix I.1) to
maximize −p i log 2 p i under the constraint
p i = 1, i.e., ∂/∂p i
−p i log 2 p i + α
p i − 1
=
0, which become p l is a constant, which must then be 1/N .
545
question which has only two possible answers, you get less information from the
answer than if there were a thousand possible answers. We will use the count W for
the number of ways a message could be clearly received, and I for the information
gained when that message is clearly received.
The second property means that I is logarithmically dependent on W : I =
k ln W . The argument is this: Two independent messages have W = W 1 W 2 ways of
being clearly received, where W i is the number of ways of clearly receiving message
i. We require I = I 1 + I 2 , i.e. I (W 1 W 2 ) = I (W 1 ) + I (W 2 ). Only logarithms satisfy
this condition for any W 1 and W 2 .
The third property makes k in I = k ln W take the value 1/ ln 2, so that
I = log 2 W . This choice of k is quite convenient in quantifying the information
sent and received by electronic digital devices, because only two symbols are used,
represented in wires by on or off electric pressure. In the wire, a message is then
a series of timed pulses, on or off. Within a given time interval, no pulse is a ‘0’
symbol, and a positive pulse is a ‘1’ symbol. These messages can be used to code
letters of an alphabet, sounds, or positions, intensities, colors within images, images
themselves, and so forth.
If a message is scrambled by noise, we lose information. Let W a represent the
number of ways a message, once received, can be misinterpreted. With ambiguity,
we have a reduced number of different messages we can read. If W c is the number of
ways the message could have been composed, then the number of different messages
which could be received and clearly interpreted is W = W c /W a , making the
information received: I = ln 2 W c −ln 2 W a , as should be expected, in that ambiguity
has reduced the information in a message.
Now suppose a message can use L symbols from the variety V =
{a 1 , a 2 , . . . , a L } in a sequence. Let the probability of receiving the symbol a l
be p l , and assume there are no correlations between the received symbols. The
average information gathered when the symbol a l is detected will be log 2 (1/p l ).
After a large number N of symbols within the variety V are received, we can detect
about Np l occurrences of the symbol a l , so that the information received will be
I =
Np l log 2 (1/p l ). The average information per symbol is then
I/N = −
p l log 2 p l .
(12.1)
A simple application of this result is to the game of 21 questions, in which you
try to guess what object in a room a person has imagined by asking no more than
21 “yes-no” questions. The best strategy, the experienced player knows, is to ask
only questions which have about a 50–50 chance of being answered yes. You will
then get the most information from the answer to the question. This observation
follows from Shannon’s information Eq. (12.1), which is maximum, for varying p i
under the constraint
p l = 1, when all the p l are equal. 3 With only two possible
answers, p l = 1/2.
3 To prove this statement, use the Lagrange-multiplier method (described in Appendix I.1) to
maximize −p i log 2 p i under the constraint
p i = 1, i.e., ∂/∂p i
−p i log 2 p i + α
p i − 1
=
0, which become p l is a constant, which must then be 1/N .