2.6 Shannon and von Neumann Entropy
23
Clearly, in the quantum case the entropy of a subsystem can be larger than the
entropy of the composite system and the conditional entropy becomes negative:
H (A|B) = H (AB) − H (B) = −1 < 0.
Other important properties of the von Neumann entropy are the strong subadditivity:
H (A|BC) ≤ H (A|B),
(2.51)
and the conditional entropy of a c.q. state. Let ρ AB =
a Pr(a) |aa | ⊗ ρ
a
B be
a c.q. state, where the state on B depends on the value a. Then the entropy of B
conditioned on A can be expressed as:
H (B|A) ρ =
a
Pr(a) H (ρ
a
B ).
(2.52)
2.6.1 Operational Meaning
We conclude this Section by briefly providing an operational meaning of the Shannon
and von Neumann entropies, which helps us motivate the introduction of smooth
entropies.
Consider a source emitting a sequence of random symbols represented by random
variables X 1 , X 2 ,…,X n , each of them distributed according to P X and independent
from each other. They are said to be independent and identically distributed (i.i.d.)
random variables. The goal is to store the data by encoding it in a bitstring without
losing information, so that it can be later retrieved. Then Shannon’s noiseless coding
theorem affirms that asymptotically—i.e. for diverging n—the amount of bits needed
per source symbol is given by H (X ).
More formally, if
ε
compr (X ) is the minimum amount of bits needed to compress
X without losing information, except for probability ε, then the compression rate of
the example above is given by:
r compr (X ) := lim
ε→0
lim
n→∞
ε
compr (X 1 X 2 · · · X n )
n
= H (X ).
(2.53)
Similarly, we consider the quantum i.i.d. source defined by the state ρ, with spectral decomposition ρ =
i λ i |ψ i ψ i |. In other words, the source emits a sequence
of quantum states drawn from {|ψ i }, according to the distribution { p i }. For Schumacher’s noiseless coding theorem, the fraction of qubits needed to reliably encode
and decode each state in the sequence is given by the von Neumann entropy H (ρ).
However, if one removes the asymptotic or the i.i.d. assumption, the Shannon
and von Neumann entropies no longer describe operational quantities. Consider for
instance a single realization (non-asymptotic regime) of a random variable X representing an n-bit string. With probability 1 /2 the string is composed of all zeroes and
with probability 1 /2 the string is random and uniformly distributed. The probability
23
Clearly, in the quantum case the entropy of a subsystem can be larger than the
entropy of the composite system and the conditional entropy becomes negative:
H (A|B) = H (AB) − H (B) = −1 < 0.
Other important properties of the von Neumann entropy are the strong subadditivity:
H (A|BC) ≤ H (A|B),
(2.51)
and the conditional entropy of a c.q. state. Let ρ AB =
a Pr(a) |aa | ⊗ ρ
a
B be
a c.q. state, where the state on B depends on the value a. Then the entropy of B
conditioned on A can be expressed as:
H (B|A) ρ =
a
Pr(a) H (ρ
a
B ).
(2.52)
2.6.1 Operational Meaning
We conclude this Section by briefly providing an operational meaning of the Shannon
and von Neumann entropies, which helps us motivate the introduction of smooth
entropies.
Consider a source emitting a sequence of random symbols represented by random
variables X 1 , X 2 ,…,X n , each of them distributed according to P X and independent
from each other. They are said to be independent and identically distributed (i.i.d.)
random variables. The goal is to store the data by encoding it in a bitstring without
losing information, so that it can be later retrieved. Then Shannon’s noiseless coding
theorem affirms that asymptotically—i.e. for diverging n—the amount of bits needed
per source symbol is given by H (X ).
More formally, if
ε
compr (X ) is the minimum amount of bits needed to compress
X without losing information, except for probability ε, then the compression rate of
the example above is given by:
r compr (X ) := lim
ε→0
lim
n→∞
ε
compr (X 1 X 2 · · · X n )
n
= H (X ).
(2.53)
Similarly, we consider the quantum i.i.d. source defined by the state ρ, with spectral decomposition ρ =
i λ i |ψ i ψ i |. In other words, the source emits a sequence
of quantum states drawn from {|ψ i }, according to the distribution { p i }. For Schumacher’s noiseless coding theorem, the fraction of qubits needed to reliably encode
and decode each state in the sequence is given by the von Neumann entropy H (ρ).
However, if one removes the asymptotic or the i.i.d. assumption, the Shannon
and von Neumann entropies no longer describe operational quantities. Consider for
instance a single realization (non-asymptotic regime) of a random variable X representing an n-bit string. With probability 1 /2 the string is composed of all zeroes and
with probability 1 /2 the string is random and uniformly distributed. The probability