546
12 Entropy and Information in Biology
12.2 Entropy and Information
As Szilárd showed, there is a direct connection between entropy and information.
Shannon formally introduced the mathematical concept of information received in
a coded message as proportional to the number of ways that message could have
been received divided by the number of independent ways the message could be
interpreted after being received, the result being called W .
As we saw above, for long messages, the information per symbol received is
i = −
p i log 2 p i .
(12.2)
Compare this Shannon information with the Boltzmann entropy for a system which
has W ways of being disordered:
S = k B ln W k B
N ln N −
n i ln n i
= −k B N
p i ln p i
(12.3)
where p i = n i /N approximates the probability of finding a particle in the state with
label i for large N . (This expression was presented in the Chapter on The Statistical
Basis of Bioenergetics, Eq. (10.81).)
Thus, the information I gained by reading the complete state of a system is
related to the entropy S of that system by
I =
1
k B ln e 2
S .
(12.4)
We recall that the statistical entropy of a macrosystem is Boltzmann’s constant
times the logarithm of the number of ways that the microsystem can be distributed
for the given macrosystem. We showed that this definition is the same as thermodynamic entropy when the system was at or near thermal equilibrium. But the measure
of disorder, S D = k B ln W , applies to any system, including a system of physical
objects which can be arranged in a variety of restricted ways. If we know the
possible ways your desk can be arranged, we can calculate its disorder. When you
put your desk into a certain order, the work you do in creating that arrangement must
produce entropy elsewhere, such as in your environment through the heat produced.
The total change, a negative quantity for your desk materials plus a positive quantity
for the environment, cannot be negative. A perfectly ordered system contains no
information, since you already know what you will get by observing the system. In
contrast, if the system has disorder, there are a variety of possible ‘readings’ of that
system, so it contains more information than one with lesser entropy.
We now see that Boltzmann’s entropy and Shannon information are connected by
the following: If we determine how the particles are distributed in the microstates of
a system, we gain information about the system as if we received a message which
told us how the particles are distributed. The probability of a particle being found
12 Entropy and Information in Biology
12.2 Entropy and Information
As Szilárd showed, there is a direct connection between entropy and information.
Shannon formally introduced the mathematical concept of information received in
a coded message as proportional to the number of ways that message could have
been received divided by the number of independent ways the message could be
interpreted after being received, the result being called W .
As we saw above, for long messages, the information per symbol received is
i = −
p i log 2 p i .
(12.2)
Compare this Shannon information with the Boltzmann entropy for a system which
has W ways of being disordered:
S = k B ln W k B
N ln N −
n i ln n i
= −k B N
p i ln p i
(12.3)
where p i = n i /N approximates the probability of finding a particle in the state with
label i for large N . (This expression was presented in the Chapter on The Statistical
Basis of Bioenergetics, Eq. (10.81).)
Thus, the information I gained by reading the complete state of a system is
related to the entropy S of that system by
I =
1
k B ln e 2
S .
(12.4)
We recall that the statistical entropy of a macrosystem is Boltzmann’s constant
times the logarithm of the number of ways that the microsystem can be distributed
for the given macrosystem. We showed that this definition is the same as thermodynamic entropy when the system was at or near thermal equilibrium. But the measure
of disorder, S D = k B ln W , applies to any system, including a system of physical
objects which can be arranged in a variety of restricted ways. If we know the
possible ways your desk can be arranged, we can calculate its disorder. When you
put your desk into a certain order, the work you do in creating that arrangement must
produce entropy elsewhere, such as in your environment through the heat produced.
The total change, a negative quantity for your desk materials plus a positive quantity
for the environment, cannot be negative. A perfectly ordered system contains no
information, since you already know what you will get by observing the system. In
contrast, if the system has disorder, there are a variety of possible ‘readings’ of that
system, so it contains more information than one with lesser entropy.
We now see that Boltzmann’s entropy and Shannon information are connected by
the following: If we determine how the particles are distributed in the microstates of
a system, we gain information about the system as if we received a message which
told us how the particles are distributed. The probability of a particle being found