13 Solutions for Selected Exercises
257
H [x] = H b (α)
and
H [y] = H b (γ ) with γ = ε + α − 2εα . (13.76)
The joint entropy H [x, y], defined in (12.14), depends on the joint probabilities
p xy (x i , y j ) that we calculated in the previous exercise and can therefore be written
as
H [x, y] = −
(1 − ε)(1 − α) log 2 ((1 − ε)(1 − α)) + εα log 2 (εα)
+ε(1 − α) log 2 ((ε(1 − α)) + (1 − ε)α log 2 ((1 − ε)α)
= −
(1 − ε) log 2 (1 − ε) + (1 − α) log 2 (1 − α)
(13.77)
+ε log 2 (ε) + α log 2 (α)
= H b (α) + H b (ε)
where the calculations that take us from the first to the second line of the equation are
rather lengthy. The conditional entropy H [y|x] follows from (12.15) as H [y|x] =
H [x, y] − H [x] = H b (ε) and inserting H [x] and H [x, y] from this exercise. We
note that it can also be calculated by inserting the joint and conditional probabilities
in the definition of H [y|x] given on the right-hand side in (12.15), though the ensuing
calculations are lengthy.
Exercise 12.4
We use the definition of the joint entropy from (12.14) and insert the factorized
probabilities. This leads to
H [x, y] = −
i
j
p x (x i ) p y (y j ) log 2 ( p x (x i ) p y (y j ))
= −
i
j
p x (x i ) p y (y j )
log 2 ( p x (x i )) + log 2 ( p y (y j ))
= −
⎡
⎣
j
p y (y j )
i
p x (x i ) log 2 ( p x (x i ))
(13.78)
+
i
p x (x j )
j
p y (y j ) log 2 ( p y (y j ))
⎤
⎦
= H [x]
j
p y (y j ) + H [y]
i
p x (x j ) .
Since the probabilities
i p x (x i ) = 1 add up to unity and likewise
j p y (y j ) = 1,
it follows that H [x, y] = H [x] + H [y].
Exercise 12.5
Equation 12.15 tells us that H [y|x] = H [x, y] − H [x] and from the previous exercise we know that H [x, y] = H [x] + H [y] for statistically independent variables x
and y. Combining these two equations leads to H [y|x] = H [y].