212
F. Nielsen and K. Sun
D. Square Root of the Symmetric α-Jensen–Shannon
Divergence
TV is bounded in [0, 1] which makes it difficult to appreciate the quality of the CROT
upper bounds in general. We shall consider a different parametric distance D α that
is upper bounded by an arbitrary bound: D α ( p, q) ≤ C α .
It is well known that the square root of the Jensen–Shannon divergence is a
metric [19] satisfying the triangle inequality. In [35], a generalization of the Jensen–
Shannon divergence was proposed, given by
JS α ( p : q):=
1
2
KL( p : ( pq) α ) +
1
2
KL(q : ( pq) α ),
(8.18)
where ( pq) α :=(1 − α) p + αq. JS α unifies (twice) the Jensen–Shannon divergence
(obtained when α =
1
2
) with the Jeffreys divergence (α = 1; [35]). A nice property
is that the skew K -divergence is upper bounded as follows:
KL( p : ( pq) α ) ≤
p log
p
(1 − α) p
≤ − log(1 − α)
for α ∈ (0, 1), so that JS α [ p : q] ≤ −
1
2
log(1 − α) −
1
2
log α for α ∈ (0, 1).
Thus, we have the square root of the symmetrized α-divergence that is upper
bounded by
JS α ( p : q) ≤ C α =
−
1
2
log(1 − α) −
1
2
log α.
However,
√
JS α [ p : q] is not a metric in general [47]. Indeed, in the extreme case
of α = 1, it is known that any positive power of the Jeffreys divergence does not
yield a metric.
Observe that JS α is a f -divergence since K α ( p : q):=KL( p : ( pq) α ) is a f -
divergence for the generator f (u) = − log((1 − α) + αu), and we have KL(q :
( pq) α ) = K 1−α (q : p). Since I f (q : p) = I f ( p : q) for g(u) = u f (1/u), it follows
that the f -generator f JS α for the JS α divergence is:
f JS α (u) = − log ((1 − α) + αu) − log
α +
1 − α
u
.
(8.19)
Figure 8.4 and Table 8.4 display the experimental results obtained for the α-JS
divergences. One can have similar observations with the TV results.