2.2 Machine Learning and Occam’s Razor
23
If we use just 10 data instead, we obtain
ˆ
P (x, d) =
d = 0 d = 1
x = 1 0.0 0.0
x = 2 0.0 0.0
x = 3 0.1 0.0
x = 4 0.1 0.0
x = 5 0.1 0.0
x = 6 0.2 0.5
(2.10)
One finds that values that should be non-zero are zero, and that it is far from the true
probability (2.5). In this case, the model Q J (x, d) can be modeled only on (2.10),
so the accuracy for an approximation of the true probability (2.2) is worse.
2.2.1 Generalization
We have to keep in mind that the ultimate goal was to make (2.8) as close to zero
as possible, even though what we really can calculate is only (2.9). From that point
of view, solving the problem of making (2.9) vanish may not always be helpful.
To make matters worse, reducing the empirical error (2.9) to an extremely small
value often causes a phenomenon called over-training, in which the value of the
generalization error (2.8) increases even though the value of the empirical error is
small. This is something similar to the following daily experience: even if one gets
a perfect score in a regular test by memorizing every word in one’s textbook, one
has not reached real understanding, so one gets a low score in an achievement test. 9
Rather, it is better to consider reducing (2.8) only from the information of (2.9).
If this is achieved, Q J will be sufficient as an approximation of the data generation
probability P , and sampling from Q J will have the generalization ability, the ability
to “predict about unknown data.”
Difficulty of generalization
In fact, by using the above definitions, one can show the inequality [21]: 10
(Generalization error) ≤ (Experience error) + O
log(#/d V C )
#/d V C
.
(2.11)
9 If the reader knows experimental physics, recall overfitting. See also [20].
10 As described in the footnote below, this holds when the story is limited to binary classification.
In addition, it is an inequality that can be used when both the generalization error and the empirical
error have been normalized to take a value of [0, 1] by some method [21].
23
If we use just 10 data instead, we obtain
ˆ
P (x, d) =
d = 0 d = 1
x = 1 0.0 0.0
x = 2 0.0 0.0
x = 3 0.1 0.0
x = 4 0.1 0.0
x = 5 0.1 0.0
x = 6 0.2 0.5
(2.10)
One finds that values that should be non-zero are zero, and that it is far from the true
probability (2.5). In this case, the model Q J (x, d) can be modeled only on (2.10),
so the accuracy for an approximation of the true probability (2.2) is worse.
2.2.1 Generalization
We have to keep in mind that the ultimate goal was to make (2.8) as close to zero
as possible, even though what we really can calculate is only (2.9). From that point
of view, solving the problem of making (2.9) vanish may not always be helpful.
To make matters worse, reducing the empirical error (2.9) to an extremely small
value often causes a phenomenon called over-training, in which the value of the
generalization error (2.8) increases even though the value of the empirical error is
small. This is something similar to the following daily experience: even if one gets
a perfect score in a regular test by memorizing every word in one’s textbook, one
has not reached real understanding, so one gets a low score in an achievement test. 9
Rather, it is better to consider reducing (2.8) only from the information of (2.9).
If this is achieved, Q J will be sufficient as an approximation of the data generation
probability P , and sampling from Q J will have the generalization ability, the ability
to “predict about unknown data.”
Difficulty of generalization
In fact, by using the above definitions, one can show the inequality [21]: 10
(Generalization error) ≤ (Experience error) + O
log(#/d V C )
#/d V C
.
(2.11)
9 If the reader knows experimental physics, recall overfitting. See also [20].
10 As described in the footnote below, this holds when the story is limited to binary classification.
In addition, it is an inequality that can be used when both the generalization error and the empirical
error have been normalized to take a value of [0, 1] by some method [21].