318
C.H. Reick . A. Grünewald . B. Page
identical to the usual check for model correctness by correlations (see e.g. Theil
(1966».
The second type of generalization failure, the inadequacy of network structure,
leads to a different generalization check. The structure of a Neural Network is
independent of the particular values of its various parameters. So, in contrast to
the previous case, here not a particular (trained) Neural Network is considered,
but a whole family of networks, all with the same structure. With respect to the
network structure reliability means here, that the predictive performance of the
network is independent of the choice of the in-sample data. Clearly, for testing
this, it has to be assumed that the training is optimal, i.e. that the particular Neural
Network is neither under- nor overadapted. Assuming this, it is obvious how this
aspect of generalization can be checked: Not only a single splitting of the data into
in-sample and out-of-sample data has to be considered, but various different
splittings. For each of these data splittings one trains the Neural Network with the
in-sample data and determines the prediction errors for the associated out-ofsampie data. For a good generalization with respect to the network structure the
prediction errors should be independent of the particular data splitting. This can be
checked, e.g., by considering the fluctuations of the mean prediction errors of the
various sets of out-of-sample data. Only if these fluctuations are small, the
prediction quality is independent from the particular splitting. An even simpler
check would be to plot the prediction errors for overlapping out-of-sample data
sets. Once more a small variation of the errors indicates a good ability to
generalize with respect to network structure. This technique is well known in the
time series literature under the names "cross-validation" and "v-leave-out" (see
e.g.Weiss and Kulikowski (1990».
These considerations show, that particular aspects of generalization can be
checked, even if the quality of the predictions is low.
16.3
Automatie Termination of Training
As discussed above, to investigate whether a network structure is suited for a
particular prediction problem, one has to perform crossvalidation studies.
Unfortunately, using standard software with visual supervision of the training
process, this is very laborous, because the training has to be repeated for many
different training sets. In this situation it would be advantageous to automatize the
training. Actually, the problem is not the training itself, but how to stop the
training optimally to prevent under- and overadaptation. This problem is tackled
by a stopping technique introduced by Weigend et al. (1991). But although this
technique has been used in a number of studies (Weigend et al. 1991; Dodier
1994; Wan 1994), it seems, that nobody has tried to cast it into an algorithm
C.H. Reick . A. Grünewald . B. Page
identical to the usual check for model correctness by correlations (see e.g. Theil
(1966».
The second type of generalization failure, the inadequacy of network structure,
leads to a different generalization check. The structure of a Neural Network is
independent of the particular values of its various parameters. So, in contrast to
the previous case, here not a particular (trained) Neural Network is considered,
but a whole family of networks, all with the same structure. With respect to the
network structure reliability means here, that the predictive performance of the
network is independent of the choice of the in-sample data. Clearly, for testing
this, it has to be assumed that the training is optimal, i.e. that the particular Neural
Network is neither under- nor overadapted. Assuming this, it is obvious how this
aspect of generalization can be checked: Not only a single splitting of the data into
in-sample and out-of-sample data has to be considered, but various different
splittings. For each of these data splittings one trains the Neural Network with the
in-sample data and determines the prediction errors for the associated out-ofsampie data. For a good generalization with respect to the network structure the
prediction errors should be independent of the particular data splitting. This can be
checked, e.g., by considering the fluctuations of the mean prediction errors of the
various sets of out-of-sample data. Only if these fluctuations are small, the
prediction quality is independent from the particular splitting. An even simpler
check would be to plot the prediction errors for overlapping out-of-sample data
sets. Once more a small variation of the errors indicates a good ability to
generalize with respect to network structure. This technique is well known in the
time series literature under the names "cross-validation" and "v-leave-out" (see
e.g.Weiss and Kulikowski (1990».
These considerations show, that particular aspects of generalization can be
checked, even if the quality of the predictions is low.
16.3
Automatie Termination of Training
As discussed above, to investigate whether a network structure is suited for a
particular prediction problem, one has to perform crossvalidation studies.
Unfortunately, using standard software with visual supervision of the training
process, this is very laborous, because the training has to be repeated for many
different training sets. In this situation it would be advantageous to automatize the
training. Actually, the problem is not the training itself, but how to stop the
training optimally to prevent under- and overadaptation. This problem is tackled
by a stopping technique introduced by Weigend et al. (1991). But although this
technique has been used in a number of studies (Weigend et al. 1991; Dodier
1994; Wan 1994), it seems, that nobody has tried to cast it into an algorithm