12.2 Relation to the Thermodynamic Entropy
199
apart from a term linear in τ , which can be shown to be zero [8]. For the entropy σ
we then obtain
σ = −
∂ F
∂τ
= log Z +
1
τ
i
p i E i = −
F
τ
+
U
τ
.
(12.6)
Note that this equation is equivalent to the definition of the the Helmholtz free energy
F = U − τ σ . Instead of calculating the entropy by differentiating the free energy
with respect to the temperature τ , we can also use the probabilities p i = e
−E i /τ
/Z
to directly calculate H from (12.2)
H = −
i
p i log p i = −
i
e
−E i /τ
Z
−
E i
τ
− log Z
=
U
τ
−
F
τ
(12.7)
where we used U =
i p i E i and F = −τ log Z . Note that we used the natural
logarithm in (12.7) instead of the logarithm with base 2, which simply means that we
use a different unit to measure the information; here we measure the information in
nats instead of bits. We find, however, that the thermodynamic entropy, as specified by
S = k B σ agrees with the concept of information H , as derived from the probabilities
p i in (12.2) or (12.7).
With this understanding of the concept “entropy” we are ready to move it from
one place to the next, which allows banks to transfer salaries from their digital vaults
to the recipients.
12.3 Moving Information Through Discrete Channels
The process of moving information from a source to a receiver is schematically
displayed in Fig. 12.1. The information, a message, flows from the source to an
encoder, shown in red. This could be a Huffman encoder, mentioned in Sect. 12.1,
where the message is converted into a sequence of symbols x i . For simplicity, we
assume that the data are represented as binary data with two symbols “1” and “0”.
The probabilities of the two symbols does not have to be equal; encoding 2, discussed
in Sect. 12.1, uses six “1” and only three “0”. After the encoder, the data are passed
through a transmission channel, shown in blue in Fig. 12.1. There the data may be
subject to noise that corrupts the data stream, for example, by randomly flipping a
bit from “0” to “1” or vice versa. Figure 12.2 illustrates such a binary symmetric
channel in which ε is the probability that a bit is flipped and α parameterizes the
probabilities that “0” or “1” appear, where α = 1/2 describes equal probabilities for
“0” and “1”. On the receiving end, a decoder, shown in magenta, undoes the coding,
recovers the message using the symbols used by the source, and finally passes it on
to the receiver.
The key question we need to answer is how much we can learn about the transmitted symbols x i by observing a received symbol y j . Or, since we a priori already
199
apart from a term linear in τ , which can be shown to be zero [8]. For the entropy σ
we then obtain
σ = −
∂ F
∂τ
= log Z +
1
τ
i
p i E i = −
F
τ
+
U
τ
.
(12.6)
Note that this equation is equivalent to the definition of the the Helmholtz free energy
F = U − τ σ . Instead of calculating the entropy by differentiating the free energy
with respect to the temperature τ , we can also use the probabilities p i = e
−E i /τ
/Z
to directly calculate H from (12.2)
H = −
i
p i log p i = −
i
e
−E i /τ
Z
−
E i
τ
− log Z
=
U
τ
−
F
τ
(12.7)
where we used U =
i p i E i and F = −τ log Z . Note that we used the natural
logarithm in (12.7) instead of the logarithm with base 2, which simply means that we
use a different unit to measure the information; here we measure the information in
nats instead of bits. We find, however, that the thermodynamic entropy, as specified by
S = k B σ agrees with the concept of information H , as derived from the probabilities
p i in (12.2) or (12.7).
With this understanding of the concept “entropy” we are ready to move it from
one place to the next, which allows banks to transfer salaries from their digital vaults
to the recipients.
12.3 Moving Information Through Discrete Channels
The process of moving information from a source to a receiver is schematically
displayed in Fig. 12.1. The information, a message, flows from the source to an
encoder, shown in red. This could be a Huffman encoder, mentioned in Sect. 12.1,
where the message is converted into a sequence of symbols x i . For simplicity, we
assume that the data are represented as binary data with two symbols “1” and “0”.
The probabilities of the two symbols does not have to be equal; encoding 2, discussed
in Sect. 12.1, uses six “1” and only three “0”. After the encoder, the data are passed
through a transmission channel, shown in blue in Fig. 12.1. There the data may be
subject to noise that corrupts the data stream, for example, by randomly flipping a
bit from “0” to “1” or vice versa. Figure 12.2 illustrates such a binary symmetric
channel in which ε is the probability that a bit is flipped and α parameterizes the
probabilities that “0” or “1” appear, where α = 1/2 describes equal probabilities for
“0” and “1”. On the receiving end, a decoder, shown in magenta, undoes the coding,
recovers the message using the symbols used by the source, and finally passes it on
to the receiver.
The key question we need to answer is how much we can learn about the transmitted symbols x i by observing a received symbol y j . Or, since we a priori already