Relative Entropy
33
So here we consider
p(x) =
1
√
2πσ p
e
−
1
2σ 2
p
(x−μ p ) 2
,
(2.50)
q(x) =
1
√
2πσ q
e
−
1
2σ 2
q
(x−μ q ) 2
.
(2.51)
Then by definition
D KL (p||q) =
∞
−∞
dx p(x)
log
σ q
σ p
−
1
2σ 2
p
(x − μ p )
2 +
1
2σ 2
q
(x − μ q )
2
= log
σ q
σ p
−
1
2σ 2
p
(σ p )
2 +
1
2σ 2
q
∞
−∞
dx p(x)
(x − μ q )
2
(μ p −μ q ) 2 +2(μ p −μ q )(x−μ p )+(x−μ p ) 2
= log
σ q
σ p
−
1
2σ 2
p
(σ p )
2 +
1
2σ 2
q
(μ p − μ q )
2 + 0 + σ
2
p
=
1
2
− log
σ 2
p
σ 2
q
+
σ 2
p
σ 2
q
− 1
+
1
σ 2
q
(μ p − μ q )
2
(2.52)
At the second equality we made a Taylor expansion. You can again show that this
value is positive, using (2.46).
Gaussian distribution and AdS spacetime
The definition of relative entropy seems to represent the “distance” between probability distributions. If there is a distance, each probability distribution corresponds
to a point, and when it is collected, it becomes a “space.” 18 In fact, in the Gaussian
distribution example above, the mean μ takes a value between (−∞, +∞) and σ
takes a value between (0, ∞) so the whole Gaussian distributions can form a “space”
(−∞, +∞)×(0, +∞). Let us find the “infinitesimal distance” in this case from the
relative entropy. We consider a point
σ p = σ , μ p = μ ,
(2.53)
and a point close to it
σ q = σ + dσ , μ q = μ + dμ .
(2.54)
18 In the research field called information geometry, this viewpoint is used.