210
F. Nielsen and K. Sun
Multiplying both sides of the inequality by − p(x) < 0 (and reversing the inequality),
we end up with
p(x) log
p(x)
q(x)
≤
p
2
(x)
q(x)
− p(x).
Then taking the integral over the support X of the distributions yields:
KL( p : q) ≤
X
p(x)
2
q(x)
dμ(x) − 1,
with equality when p(x) = q(x) almost everywhere. Notice that the right-hand side
integral
X
p(x)
2
q(x)
dμ(x) may diverge (e.g., when KL is infinite).
Now, let us consider two mixtures m(x) =
k
i=1 w i p i (x) and m
(x) =
k
i=1 w
i p
i (x). Apply Lemma 8.9 to get
KL(m : m
) ≤
i, j
w i w j
p i (x) p j (x)
m (x)
dμ(x) − 1.
Let us upper bound A i j =
p i (x) p j (x)
m (x)
dμ(x) to upper bound
KL(m : m
) ≤
i, j
w i w j A i j − 1.
For bounding the terms A i j , we interpret the mixture density as an arithmetic
weighted mean that is greater or equal than a geometric mean (AGM inequality).
Therefore we get:
p i (x) p j (x)
m (x)
dμ(x) ≤
p i (x) p j (x)
k
l=1 w
l p
l (x)
dμ(x).
When the mixture components belong to a same exponential family [39], we
get a closed-form upper bound since θ i + θ j −
k
l=1 w
l θ
l ∈ : Let ¯
θ
=
k
l=1 w
l θ
l
denote the barycenter of the natural parameters of the mixture components of m
.
We have:
p(x; θ i ) p(x; θ j )
k
l=1 w
l p(x; θ
l )
= exp
⎛
⎝
θ i + θ j − ¯
θ
t (x) − F(θ i ) − F(θ j ) +
k
l=1
w
l F(θ
l ) + k(x)
⎞
⎠ .