7 On Geodesic Triangles with Right Angles in a Dually Flat Space
171
B F (θ 1 : θ) + B F (θ 2 : θ) = 2JB F (θ 1 , θ 2 ) + 2B F
θ 1 + θ 2
2
: θ
,
(7.58)
B F (θ 1 : θ) + B F (θ 2 : θ) = 2J F (θ 1 , θ 2 ) + 2B F
θ 1 + θ 2
2
: θ
,
(7.59)
where JB F and J F denote the Jensen-Bregman divergence [21] and the Jensen divergence [16], respectively:
JB F (θ 1 , θ 2 ) :=
1
2
B F
θ 1 :
θ 1 + θ 2
2
+ B F
θ 2 :
θ 1 + θ 2
2
,
(7.60)
=
F(θ 1 ) + F(θ 2 )
2
− F
θ 1 + θ 2
2
=: J F (θ 1 , θ 2 ).
(7.61)
The family of categorical distributions forms a mixture family [22] in information
geometry for the Shannon negentropy generator F Shannon , and we have B F Shannon (θ 1 :
θ 2 ) = KL( p θ 1 : p θ 2 ). It follows that for any three categorical distributions p, q, and
r , we have
KL( p : r ) + KL(q : r ) = 2JS( p, q) + 2KL
p + q
2
: r
,
(7.62)
since
p+q
2
is a categorical distribution. In general, the identity does not hold for
members of an exponential family (i.e., Gaussian family) since the mixture density
p+q
2
does not belong to the exponential family. The categorical family is a very
special case of family that is both an exponential family and a mixture family [3].
Since there exists a bijection between regular exponential families and regular
Bregman divergences [4], we can state the parallelogram-type identity of Eq. 7.57
for parametric densities { p θ } belonging to the same exponential family [18] (with
cumulant function F) with respect to both the Kullback–Leibler divergence and the
Bhattacharyya divergence [16]
Bhat( p, q) := − log
p(x)q(x)dμ(x),
(7.63)
as:
KL( p θ : p θ 1 ) + KL( p θ : p θ 2 ) = 2Bhat( p θ 1 , p θ 2 ) + 2KL
p θ 1 +θ 2
2
: p θ
, (7.64)
since KL( p θ : p θ ) = B F (θ
: θ) and Bhat( p θ : p θ ) = J F (θ, θ
) for densities p θ and
p θ belonging to the same exponential family. For example, the identity of Eq. 7.64
applies to any two densities of the Gaussian family.
The 3-parameter property of Bregman divergences is a particular instance of the
following 4-parameter property [26] (a quadrilateral relation):
Property 2 (Bregman 4-parameter property) For any four points p 1 , p 2 , q 1 , q 2 , we
have the following identity:
171
B F (θ 1 : θ) + B F (θ 2 : θ) = 2JB F (θ 1 , θ 2 ) + 2B F
θ 1 + θ 2
2
: θ
,
(7.58)
B F (θ 1 : θ) + B F (θ 2 : θ) = 2J F (θ 1 , θ 2 ) + 2B F
θ 1 + θ 2
2
: θ
,
(7.59)
where JB F and J F denote the Jensen-Bregman divergence [21] and the Jensen divergence [16], respectively:
JB F (θ 1 , θ 2 ) :=
1
2
B F
θ 1 :
θ 1 + θ 2
2
+ B F
θ 2 :
θ 1 + θ 2
2
,
(7.60)
=
F(θ 1 ) + F(θ 2 )
2
− F
θ 1 + θ 2
2
=: J F (θ 1 , θ 2 ).
(7.61)
The family of categorical distributions forms a mixture family [22] in information
geometry for the Shannon negentropy generator F Shannon , and we have B F Shannon (θ 1 :
θ 2 ) = KL( p θ 1 : p θ 2 ). It follows that for any three categorical distributions p, q, and
r , we have
KL( p : r ) + KL(q : r ) = 2JS( p, q) + 2KL
p + q
2
: r
,
(7.62)
since
p+q
2
is a categorical distribution. In general, the identity does not hold for
members of an exponential family (i.e., Gaussian family) since the mixture density
p+q
2
does not belong to the exponential family. The categorical family is a very
special case of family that is both an exponential family and a mixture family [3].
Since there exists a bijection between regular exponential families and regular
Bregman divergences [4], we can state the parallelogram-type identity of Eq. 7.57
for parametric densities { p θ } belonging to the same exponential family [18] (with
cumulant function F) with respect to both the Kullback–Leibler divergence and the
Bhattacharyya divergence [16]
Bhat( p, q) := − log
p(x)q(x)dμ(x),
(7.63)
as:
KL( p θ : p θ 1 ) + KL( p θ : p θ 2 ) = 2Bhat( p θ 1 , p θ 2 ) + 2KL
p θ 1 +θ 2
2
: p θ
, (7.64)
since KL( p θ : p θ ) = B F (θ
: θ) and Bhat( p θ : p θ ) = J F (θ, θ
) for densities p θ and
p θ belonging to the same exponential family. For example, the identity of Eq. 7.64
applies to any two densities of the Gaussian family.
The 3-parameter property of Bregman divergences is a particular instance of the
following 4-parameter property [26] (a quadrilateral relation):
Property 2 (Bregman 4-parameter property) For any four points p 1 , p 2 , q 1 , q 2 , we
have the following identity: