320
C.H. Reick' A. Grünewald . B. Page
suitable for automatization. In the following we will show how this ean be done.
For further details see Grünewald (2000).
Supervised training proeedures, like baekpropagation, use the output error to
optimize iteratively the network parameters. Hereby the network is adapted to the
data. But, as already diseussed, adaptation and generalization are mutually
excluding aims. Therefore, the network has to be trained for sufficiently many
training eycles to prevent underadaptation, but the training has to be stopped
before the network begins to overadapt. Overadaption ean be deteeted only by
data that were not used for training. So one splits the in-sample data onee more
into two parts: the (genuine) training set and a validation set. The idea is to use, as
usual, the training set to adapt the network, but to use the validation set to
terminate training. This is done by monitoring during training the validation error
that is obtained when applying the Neural Network after eaeh iteration step to the
validation set. One often observes that during training this validation error first
deereases and then passes through aminimum, where the error starts inereasing
(see e.g. the diagrams in Weigend (1994». If in parallel the error on the training
set is monitored, one observes that even beyond the minimum of the validation
error the training set error deereases. This can be interpreted as the onset of
overadaptation: the network starts learning details that are not only irrelevant for
predieting new data, but also impair the prediction quality. Therefore the optimal
stopping point is at the minimum of the validation error eurve. Following Dodier
(1994), this teehnique of stopping, originally introdueed by Weigend et al. (1991),
will be ealled early stopping in the following.
To eonvert eady stopping into a useful algorithm, two problems have to be
surmounted: (i) The validation error does not always undergo a minimum. In that
ease another stopping eriterion has to be used. (ii) The validation error eurve need
not be smooth, but may show noisy behavior atop of an overall deerease or
inerease. In that ease the eurve eontains many loeal minima that have to be
eseaped in order to seareh for the (hopefully) global minimum. Our solution to
these problems is the algorithm shown in Fig. 16.1.1t eontains three parameters:
MAXCYC Maximum number of training fYfles after whieh every training is
stopped.
DELAY Number of training eycles by whieh stopping is delayed after a
possible stopping point is deteeted.
TOL Improvement in validation error that is eonsidered insignifieant
("toleranee").
To explain the algorithm, the parts related to the delay counter "D" (inside the
dashed frame in Fig. 1) are first ignored. Without them the algorithm works as
follows: As usual the network is trained iteratively. In eaeh step the validation
error ("err") is eomputed and if this error is smaller than the best error found so
far ("bestErr"), the parameters of that - so far - best network and best error are
saved ("bestNet = net", "bestErr = err"). The whole proeess is stopped, when the
number of iteration eycles ("C') reaehes the maximum number of iteration eycles
C.H. Reick' A. Grünewald . B. Page
suitable for automatization. In the following we will show how this ean be done.
For further details see Grünewald (2000).
Supervised training proeedures, like baekpropagation, use the output error to
optimize iteratively the network parameters. Hereby the network is adapted to the
data. But, as already diseussed, adaptation and generalization are mutually
excluding aims. Therefore, the network has to be trained for sufficiently many
training eycles to prevent underadaptation, but the training has to be stopped
before the network begins to overadapt. Overadaption ean be deteeted only by
data that were not used for training. So one splits the in-sample data onee more
into two parts: the (genuine) training set and a validation set. The idea is to use, as
usual, the training set to adapt the network, but to use the validation set to
terminate training. This is done by monitoring during training the validation error
that is obtained when applying the Neural Network after eaeh iteration step to the
validation set. One often observes that during training this validation error first
deereases and then passes through aminimum, where the error starts inereasing
(see e.g. the diagrams in Weigend (1994». If in parallel the error on the training
set is monitored, one observes that even beyond the minimum of the validation
error the training set error deereases. This can be interpreted as the onset of
overadaptation: the network starts learning details that are not only irrelevant for
predieting new data, but also impair the prediction quality. Therefore the optimal
stopping point is at the minimum of the validation error eurve. Following Dodier
(1994), this teehnique of stopping, originally introdueed by Weigend et al. (1991),
will be ealled early stopping in the following.
To eonvert eady stopping into a useful algorithm, two problems have to be
surmounted: (i) The validation error does not always undergo a minimum. In that
ease another stopping eriterion has to be used. (ii) The validation error eurve need
not be smooth, but may show noisy behavior atop of an overall deerease or
inerease. In that ease the eurve eontains many loeal minima that have to be
eseaped in order to seareh for the (hopefully) global minimum. Our solution to
these problems is the algorithm shown in Fig. 16.1.1t eontains three parameters:
MAXCYC Maximum number of training fYfles after whieh every training is
stopped.
DELAY Number of training eycles by whieh stopping is delayed after a
possible stopping point is deteeted.
TOL Improvement in validation error that is eonsidered insignifieant
("toleranee").
To explain the algorithm, the parts related to the delay counter "D" (inside the
dashed frame in Fig. 1) are first ignored. Without them the algorithm works as
follows: As usual the network is trained iteratively. In eaeh step the validation
error ("err") is eomputed and if this error is smaller than the best error found so
far ("bestErr"), the parameters of that - so far - best network and best error are
saved ("bestNet = net", "bestErr = err"). The whole proeess is stopped, when the
number of iteration eycles ("C') reaehes the maximum number of iteration eycles