172
F. Nielsen
θ
θ 2
θ 1
B F (θ 1 : θ) + B F (θ 2 : θ)=B F θ 1 :
θ1+θ2
2
+ B F θ 2 :
θ1+θ2
2
+ 2B F
θ1+θ2
2
: θ
θ1+θ2
2
x2
J F (θ1, θ 2 ) =
1
2 B F (θ 1 :
θ1+θ2
2 ) +
1
2 B F (θ 2 :
θ1+θ2
2 )
Fig. 7.10 A parallelogram-type identity for Bregman divergences
B F (θ( p 1 ) : θ(q 1 )) + B F (θ( p 2 ) : θ(q 2 )) −B F (θ( p 1 ) : θ(q 2 )) − B F (θ( p 2 ) : θ(q 1 ))
−(θ( p 2 ) − θ( p 1 ))
(η(q 1 ) − η(q 2 )) = 0. (7.65)
Indeed, let q 1 = p 1 . Then we recove:r
B F (θ( p 2 ) : θ(q 2 )) = B F (θ( p 1 ) : θ(q 2 )) + B F (θ( p 2 ) : θ(q 1 )) − (θ( p 2 ) − θ( p 1 )) (η(q 2 ) − η( p 1 )),(7.66)
D F ( p 2 : q 2 ) = D F ( p 2 : p 1 ) + D F ( p 1 : q 2 ) − g p 1 ( ˙
γ p 1 p 2 (0), ˙
γ ∗
p 1 q 2
(0)).
(7.67)
Similarly, let q 2 = p 2 . Then we get:
B F (θ( p 1 ) : θ(q 1 )) = B F (θ( p 1 ) : θ( p 2 )) + B F (θ( p 2 ) : θ(q 1 )) − (θ( p 1 ) − θ( p 2 )) (η(q 1 ) − η( p 2 )),(7.68)
D F ( p 1 : q 1 ) = D F ( p 1 : p 2 ) + D F ( p 2 : q 1 ) − g p 2 ( ˙
γ p 2 p 1 (0), ˙
γ ∗
p 2 q 1
(0)).
(7.69)
To derive the 4-parameter identity, we use twice the 3-parameter identity as follows:
B F (θ 1 : θ 3 ) = B F (θ 1 : θ 2 ) + B F (θ 2 : θ 3 ) − (θ 3 − θ 2 ) (∇ F(θ 1 ) − ∇ F(θ 2 )),
(7.70)
B F (θ 4 : θ 3 ) = B F (θ 4 : θ 2 ) + B F (θ 2 : θ 3 ) − (θ 3 − θ 2 ) (∇ F(θ 4 ) − ∇ F(θ 2 )).
(7.71)
Indeed, subtracting Eq. 7.71 from Eq. 7.70, we get
B F (θ 1 : θ 3 ) − B F (θ 4 : θ 3 ) = B F (θ 1 : θ 2 ) − B F (θ 4 : θ 2 ) − (θ 3 − θ 2 )
(∇ F(θ 1 ) − ∇ F(θ 4 )).
(7.72)
We can interpret geometrically the 4-parameter identity as follows:
D F ( p 1 : p 3 ) − D F ( p 4 : p 3 ) = D F ( p 1 : p 2 ) − D F ( p 4 : p 2 ) − g p2 ( ˙
γ p2 p3 (0), ˙
γ
∗
p2 p1 (0))
+ g p2 ( ˙
γ p2 p3 (0), , ˙
γ
∗
p2 p4 (0)).
(7.73)
Note that the Bregman 4-parameter identity is said parallelogram-type, because if
we choose F Euc (θ ) =
1
2
θ
θ =
1
2
2
2 , then we have B F Euc (θ : θ
) =
1
2
− θ
2 , and
the 4-parameter identity becomes for θ = 0: