12.4 Continuous Information Channels
205
where the integral extends over the domain in which x is defined. Note that we use
the natural logarithm with base e, as we did in Sect. 12.2, so the entropy has units
of nats. The other concepts, introduced in Sect. 12.3, such as joint and conditional
probabilities, joint and conditional entropies, as well as mutual information can be
generalized likewise; we only have to replace the sums by integrals. In the following,
we will liberally make use of this analogy and refer to the equations in the previous
section that were proven for discrete variables. The entropies of the distributions,
shown in Fig. 12.4 are H = −0.460 for the Beethoven symphony and a significantly
smaller value of H = −0.868 for the sine. The Gaussian, shown as the red dashed
curves, has the largest entropy H = −0.259.
Since we want to transfer the maximum information across a channel, we might
ask ourselves: which probability distribution function g(x) has the largest entropy?
For physical reasons, we consider only voltages that dissipate a finite average power
u
2
/R in some resistance R. This implies that the distribution g(x) must have a finite
second moment, defined by σ
2
=
x
2 g(x)dx. Moreover, since it is a probability
distribution function, g(x) must be non-negative and it must be normalized with
g(x)dx = 1. Thus g(x) can be found from minimizing the functional J [g] with
respect to g(x)
J [g] = λ 1 + λ 2 σ
2
+
−g(x) log(g(x)) − λ 1 g(x) − λ 2 x
2 g(x)
dx (12.21)
with two Lagrange multipliers λ 1 and λ 2 . The variation with respect to δg(x) yields
0 = − log(g(x)) − 1 − λ 1 − λ 2 x
2
or
g(x) = e
−λ 2 −1 e
−λ 2 x
2 .
(12.22)
Differentiating with respect to the Lagrange multipliers, we recover the constraints,
which we use to determine λ 1 and λ 2 . We thus find a Gaussian
g(x) =
1
√
2πσ
e
−x
2 /2σ
2
(12.23)
as the normalized probability distribution with σ
2 as second moment that maximizes
the entropy given by (12.20). It is now straightforward to calculate the entropy of the
Gaussian as
H g =
1
2
log
2π eσ
2
.
(12.24)
The value calculated from (12.24) agrees with the value we determined numerically
for the Gaussian in Fig. 12.4 and now we understand why it is the largest of the three
entropies that we determined from the amplitude-probability distributions. On the
coming pages we will use these Gaussians not only as models for the sources that
produce the information-carrying signals, but also for the useless information, which
is added by random voltages while the signals traverse a communication channel.
205
where the integral extends over the domain in which x is defined. Note that we use
the natural logarithm with base e, as we did in Sect. 12.2, so the entropy has units
of nats. The other concepts, introduced in Sect. 12.3, such as joint and conditional
probabilities, joint and conditional entropies, as well as mutual information can be
generalized likewise; we only have to replace the sums by integrals. In the following,
we will liberally make use of this analogy and refer to the equations in the previous
section that were proven for discrete variables. The entropies of the distributions,
shown in Fig. 12.4 are H = −0.460 for the Beethoven symphony and a significantly
smaller value of H = −0.868 for the sine. The Gaussian, shown as the red dashed
curves, has the largest entropy H = −0.259.
Since we want to transfer the maximum information across a channel, we might
ask ourselves: which probability distribution function g(x) has the largest entropy?
For physical reasons, we consider only voltages that dissipate a finite average power
u
2
/R in some resistance R. This implies that the distribution g(x) must have a finite
second moment, defined by σ
2
=
x
2 g(x)dx. Moreover, since it is a probability
distribution function, g(x) must be non-negative and it must be normalized with
g(x)dx = 1. Thus g(x) can be found from minimizing the functional J [g] with
respect to g(x)
J [g] = λ 1 + λ 2 σ
2
+
−g(x) log(g(x)) − λ 1 g(x) − λ 2 x
2 g(x)
dx (12.21)
with two Lagrange multipliers λ 1 and λ 2 . The variation with respect to δg(x) yields
0 = − log(g(x)) − 1 − λ 1 − λ 2 x
2
or
g(x) = e
−λ 2 −1 e
−λ 2 x
2 .
(12.22)
Differentiating with respect to the Lagrange multipliers, we recover the constraints,
which we use to determine λ 1 and λ 2 . We thus find a Gaussian
g(x) =
1
√
2πσ
e
−x
2 /2σ
2
(12.23)
as the normalized probability distribution with σ
2 as second moment that maximizes
the entropy given by (12.20). It is now straightforward to calculate the entropy of the
Gaussian as
H g =
1
2
log
2π eσ
2
.
(12.24)
The value calculated from (12.24) agrees with the value we determined numerically
for the Gaussian in Fig. 12.4 and now we understand why it is the largest of the three
entropies that we determined from the amplitude-probability distributions. On the
coming pages we will use these Gaussians not only as models for the sources that
produce the information-carrying signals, but also for the useless information, which
is added by random voltages while the signals traverse a communication channel.