32
2 Introduction to Machine Learning
Fig. 2.3 Green: (X − 1), Blue: graph of log X
the integrand of (2.45) is always equal to or larger than zero. Since the integrand is
always equal to or larger than zero, we can show
D KL (p||q) ≥ 0.
(2.47)
Next, the equality is satisfied when the equality of (2.46) is established (X = 1).
This means that for any x we need to have
q(x)
p(x)
= 1,
(2.48)
which shows (2.43).
Example: Gaussian distribution
To get a sense of the relative entropy, let us calculate the relative entropy for the
Gaussian distribution:
1
√
2πσ
e
−
1
2σ 2 (x−μ) 2
.
(2.49)
2 Introduction to Machine Learning
Fig. 2.3 Green: (X − 1), Blue: graph of log X
the integrand of (2.45) is always equal to or larger than zero. Since the integrand is
always equal to or larger than zero, we can show
D KL (p||q) ≥ 0.
(2.47)
Next, the equality is satisfied when the equality of (2.46) is established (X = 1).
This means that for any x we need to have
q(x)
p(x)
= 1,
(2.48)
which shows (2.43).
Example: Gaussian distribution
To get a sense of the relative entropy, let us calculate the relative entropy for the
Gaussian distribution:
1
√
2πσ
e
−
1
2σ 2 (x−μ) 2
.
(2.49)