5.1 Central Limit Theorem and Its Role in Machine Learning
79
we only know how many times each event occurred,
•
A 1 : # 1 times, A 2 : # 2 times, . . . , A W : # W times,
# =
W
i=1 # i times in total.
(5.2)
In Chap. 1, we used, without any proof, the fact that the maximum likelihood
estimation converges to the desired probability p i with in the limit of the number of
data # → ∞,
# i
#
→ p i ,
(5.3)
which seems intuitively correct. However, when asked why it is fine to use (5.3),
how can we answer it properly?
Law of large numbers
Here, consider the following:
X
(i)
n =
1 if event A i occurs in the nth trial,
0 if event A i does not occur in the nth trial.
(5.4)
Then, we can write
# i
#
=
1
#
#
n=1
X
(i)
n .
(5.5)
Let us consider the “expectation value” of this ratio, which is
# i
#
p
=
1
#
#
n=1
X
(i)
n
p
=
1
#
#
n=1
X
(i)
n
p
=
1
#
#
n=1
p i · 1
X
(i)
n
+(1 − p i ) · 0
X
(i)
n
=
1
#
#
n=1
p i = p i ,
(5.6)
and is equal to the probability of occurrence of event A i . This is quite reminiscent
of (5.3), but not exactly equal to (5.3) itself. So, let us investigate the “difference”