Chapter 16 . Time-Series Prediction of Marine Zooplankton
317
type of network structure is appropriate for a particular problem so that one can
never exclude that a performance failure is the result of an unsuited network type.
A partial solution to this problem is the use of training procedures that change not
only the network parameters, but also its topology.
From these five causes for performance failures the first two
("unpredictability" and "poor data") are independent of Neural Networks and
from the remaining three only "overadaptation" and "unsuited network structure"
are related to generalization failures, while the cause "underadaptation" is the
result of an insufficient adaptation. Nevertheless, in practice, one is usually not
able to identify the particular cause for a performance failure so that in practice a
general performance failure cannot be distinguished from a generalization failure.
But besides prediction quality, there are also a second and third aspect of
generalization and these turn out to be measurable almost independently of the
prediction quality. The second aspect, as already mentioned, concerns the
reliability of the predictions. One characteristic of generalization failures is their
independence from the data: Let us assurne, we had trained a Neural Network with
high quality data. We now take other data, documenting the same phenomenon,
but of low quality (e.g. by adding noise to the high quality data). If the Neural
Network generalizes for the high quality data quite weil, it will clearly produce
poorer predictions with the low quality data, but the predictions are still reliable
(although the prediction error will be larger), because the Neural Network had
learned the essential features of the underlying system. This characteristic of
generalization can even be detected from low quality data, because reliability
means that the estimated prediction error can be trusted, even if it is high.
The third aspect of generalization concerns model correctness. Each trained
Neural Network can be considered as a (formal) model representing the dynamics
underlying the data. The generalization would obviously fail, if this formal model
would be incorrect. In that case the predictions errors would show systematic
deviations from the correct values. Accordingly this aspect of generalization is
related to the correlation between errors and data.
To see how reliability and model correctness can be detected we consider the
two types of generalization failures separately. First overadaptation is considered.
This type of generalization failure is related to a particular trained network: As
usual one splits the available data into in-sample and out-oJ-sample data, trains the
Neural Netwark with the in-sample data and uses the out-of-sample data to
measure the predictive performance. For the reliability of the predictions one has
to look whether the prediction error can be trusted, or, expressed otherwise,
whether the prediction error is stationary. This is usually only possible, if the data
set is sufficiently large and this is often not the case. It is much simpler to test for
model correctness. Here one has to check whether the prediction errors are
uncorrelated to the data. It is easy to show that (linear) uncorrelatedness between
errors and data is identical to an ideal correlation between predictions and data
(besides an error in bias). Therefore the check for this aspect of generalization is
317
type of network structure is appropriate for a particular problem so that one can
never exclude that a performance failure is the result of an unsuited network type.
A partial solution to this problem is the use of training procedures that change not
only the network parameters, but also its topology.
From these five causes for performance failures the first two
("unpredictability" and "poor data") are independent of Neural Networks and
from the remaining three only "overadaptation" and "unsuited network structure"
are related to generalization failures, while the cause "underadaptation" is the
result of an insufficient adaptation. Nevertheless, in practice, one is usually not
able to identify the particular cause for a performance failure so that in practice a
general performance failure cannot be distinguished from a generalization failure.
But besides prediction quality, there are also a second and third aspect of
generalization and these turn out to be measurable almost independently of the
prediction quality. The second aspect, as already mentioned, concerns the
reliability of the predictions. One characteristic of generalization failures is their
independence from the data: Let us assurne, we had trained a Neural Network with
high quality data. We now take other data, documenting the same phenomenon,
but of low quality (e.g. by adding noise to the high quality data). If the Neural
Network generalizes for the high quality data quite weil, it will clearly produce
poorer predictions with the low quality data, but the predictions are still reliable
(although the prediction error will be larger), because the Neural Network had
learned the essential features of the underlying system. This characteristic of
generalization can even be detected from low quality data, because reliability
means that the estimated prediction error can be trusted, even if it is high.
The third aspect of generalization concerns model correctness. Each trained
Neural Network can be considered as a (formal) model representing the dynamics
underlying the data. The generalization would obviously fail, if this formal model
would be incorrect. In that case the predictions errors would show systematic
deviations from the correct values. Accordingly this aspect of generalization is
related to the correlation between errors and data.
To see how reliability and model correctness can be detected we consider the
two types of generalization failures separately. First overadaptation is considered.
This type of generalization failure is related to a particular trained network: As
usual one splits the available data into in-sample and out-oJ-sample data, trains the
Neural Netwark with the in-sample data and uses the out-of-sample data to
measure the predictive performance. For the reliability of the predictions one has
to look whether the prediction error can be trusted, or, expressed otherwise,
whether the prediction error is stationary. This is usually only possible, if the data
set is sufficiently large and this is often not the case. It is much simpler to test for
model correctness. Here one has to check whether the prediction errors are
uncorrelated to the data. It is easy to show that (linear) uncorrelatedness between
errors and data is identical to an ideal correlation between predictions and data
(besides an error in bias). Therefore the check for this aspect of generalization is