12.2 Entropy and Information
547
in a given energy state is equivalent to the probability of a letter appearing in a
message.
With Szilárd’s work on Maxwell’s demon, 4 we know that if a computer, while
operating in a cycle, produces order in a system, such as putting a number of words
in alphabetical order, the computer must expel heat. By analyzing a storage device
which stores a single bit of information, Landauer proved 5 that each time a bit of
information is erased to make that memory available for the next bit, there is an
entropy increase of k B ln 2. Thus, the minimal amount of entropy produced by any
cyclic message-generating device which creates I bits of information is
S = (k B ln e 2)I.
The cyclic processes which create information necessarily generate entropy. This
occurs in our cells as amino acids are assembled into proteins, as DNA is replicated,
and in our brains as we learn.
Receiving written messages supplies information. For the English language,
messages are coded in letters, space, and punctuation. Depending on the author,
the frequency of various letters differ, but over a library of literary works, the
probabilities for some ‘popular’ letters are close to the table below:
Letter Probability ASCII (binary)
e
0.127
01100101
t
0.091
01110100
a
0.082
01100001
o
0.075
01101111
i
0.070
01101001
n
0.067
01101110
s
0.063
01110011
h
0.061
01101000
r
0.060
01110010
In a computer, letters, symbols, and other typewriter controls are coded in 8bits, according to a convention called the American Standard Code for Information
Interchange, or ‘ASCII’.
Now, because the probabilities of the letters are not all equal, less information
is transmitted sending messages by such letters than if the letters were of equal
probability. We can save message length (and money, if we are paying for our
4 L. Szilárd, On the decrease of entropy in a thermodynamic system by the intervention of intelligent
beings, Z Phys 53, 840 (1929).
5 R. Landauer, Irreversibility and heat generation in the computing process IBM J Res. Dev 5 (3),
183 (1961); see also C.H. Bennett, Demons, engines and the Second Law, Sci Am 257 (5), 108–116
(1987).
547
in a given energy state is equivalent to the probability of a letter appearing in a
message.
With Szilárd’s work on Maxwell’s demon, 4 we know that if a computer, while
operating in a cycle, produces order in a system, such as putting a number of words
in alphabetical order, the computer must expel heat. By analyzing a storage device
which stores a single bit of information, Landauer proved 5 that each time a bit of
information is erased to make that memory available for the next bit, there is an
entropy increase of k B ln 2. Thus, the minimal amount of entropy produced by any
cyclic message-generating device which creates I bits of information is
S = (k B ln e 2)I.
The cyclic processes which create information necessarily generate entropy. This
occurs in our cells as amino acids are assembled into proteins, as DNA is replicated,
and in our brains as we learn.
Receiving written messages supplies information. For the English language,
messages are coded in letters, space, and punctuation. Depending on the author,
the frequency of various letters differ, but over a library of literary works, the
probabilities for some ‘popular’ letters are close to the table below:
Letter Probability ASCII (binary)
e
0.127
01100101
t
0.091
01110100
a
0.082
01100001
o
0.075
01101111
i
0.070
01101001
n
0.067
01101110
s
0.063
01110011
h
0.061
01101000
r
0.060
01110010
In a computer, letters, symbols, and other typewriter controls are coded in 8bits, according to a convention called the American Standard Code for Information
Interchange, or ‘ASCII’.
Now, because the probabilities of the letters are not all equal, less information
is transmitted sending messages by such letters than if the letters were of equal
probability. We can save message length (and money, if we are paying for our
4 L. Szilárd, On the decrease of entropy in a thermodynamic system by the intervention of intelligent
beings, Z Phys 53, 840 (1929).
5 R. Landauer, Irreversibility and heat generation in the computing process IBM J Res. Dev 5 (3),
183 (1961); see also C.H. Bennett, Demons, engines and the Second Law, Sci Am 257 (5), 108–116
(1987).