208
F. Nielsen and K. Sun
As D(·, ·) is a metric, the density r
(y, z) is concentrated on the region p(x|y) =
q(x|z) so that
r
(y, z) p(x|y)dydz =
r
(y, z)q(x|z)dydz.
We therefore have
p(x) =
p(y) p(x|y)dy =
r
(y, z)dzp(x|y)dy =
r
(y, z) p(x|y)dydz
=
r
(y, z)q(x|z)dydz =
r
(y, z)dyq(x|z)dz =
q(z)q(x|z)dz
= q(x).
Symmetry.
H D ( p, q) =
inf
r ∈( p(y),q(z))
r (y, z)D
p(x|y), q(x|z)
dydz
=
inf
r ∈( p(y),q(z))
r (y, z)D
q(x|z), p(x|y)
dydz
=
inf
R∈(q(z), p(y))
R(z, y)D
q(x|z), p(x|y)
dzdy
= H D (q, p),
where R(z, y) = r (y, z) s.t.
R(z, y)dy = q(z) and
R(z, y)dz = p(y).
Triangle inequality. Denote
r 12 = arg min r ∈( p 1 (y 1 ), p 2 (y 2 )) E r (y 1 ,y 2 ) D( p 1 (x|y 1 ), p 2 (x|y 2 )),
r 23 = arg min r ∈( p 2 (y 2 ), p 3 (y 3 )) E r (y 2 ,y 3 ) D( p 2 (x|y 2 ), p 3 (x|y 3 )).
H D ( p 1 , p 2 ) + H D ( p 2 , p 3 )
=E r 12 (y 1 ,y 2 ) D( p 1 (x|y 1 ), p 2 (x|y 2 )) + E r 23 (y 2 ,y 3 ) D( p 2 (x|y 2 ), p 3 (x|y 3 ))
≥ inf
s
E s(y 1 ,y 2 ,y 3 ) [D( p 1 (x|y 1 ), p 2 (x|y 2 )) + D( p 2 (x|y 2 ), p 3 (x|y 3 ))]
≥ inf
s
E s(y 1 ,y 2 ,y 3 ) D( p 1 (x|y 1 ), p 3 (x|y 3 ))
= inf
r
E r (y,z) D( p 1 (x|y), p 3 (x|z))
=H D ( p 1 , p 3 ),
where s(y 1 , y 2 , y 3 ) denotes the set of all probability measures on Y
3 with marginals
p 1 , p 2 and p 3 . Clearly,
r 12 (y 1 ,y 2 )r 23 (y 2 ,y 3 )
p 2 (y 2 )
∈ s(y 1 , y 2 , y 3 ).