286
H. Wilson . F. Recknagel
For the remaining model structures, the fact that hidden layer neurons make
little difference to predictive performance means that it is not dear that the ANN
is finding a consistent non-linearity. In these cases, it can be expected that a linear
model will perform nearly as weIl as an ANN. However, despite the lack of
evidence for improved RMS error, examination of time series plots revealed a
subtle performance improvement with hidden layer nodes. One explanation for
this observation is that the RMS error penalises small timing errors of prediction
much more heavily than a visual analysis of the traces. For example, a model that
correctly predicts the magnitude of a bloom, but predicts it one time step too late
or too early will be evaluated more favourably by a visual analysis than a model
that makes no prediction at all. However, RMS error would rate such a model
worse than a non-prediction, since there is a false positive as weIl as a false
negative prediction.
The bootstrap aggregation procedure was shown to be highly effective at
accounting for variance in ANN models - particularly in cases where there was
strong overfitting exhibited. However, it is important to note that such a benefit
comes at the cost of several drawbacks. Firstly, the bootstrap aggregate model is
constrained to being conservative in the context of small datasets, because over a
third of the data is left out of training on average. Secondly, the use of such a
procedure in the context of a time series raises questions about its effect on the
independence of test data, since training data may be sampled from random
positions in the time series. In the real world, it is impossible for the model to
make use of information regarding water quality that is in the future. However,
random resampling of training data provides the model with that information since
it has freedom to pick training records from a short time period in the future in
relation to a given test set record. In this application, there can be two arguments
against such objections relating to test set independence:
Any positive bias of the model performance estimation caused by
representation of "future" information in the training set counterbalances the
negative bias caused by holding out one third of the data from training.
Since most time series have sampling frequencies of between 2 weeks and one
month, it is likely that the serial correlation in the data that causes bias due to
"future" information is insignificant.
One way of overcoming this problem may be to break the data up into
randomly sampled chunks of several months or years.
14.6
Conclusions
This study has established the following findings:
1. The generic ANN structure considering phosphorous, nitrogen, secchi disc
depth and temperature as inputs, and chlorophyll a concentration as output,
achieved a reasonable level of predictive success for every model. The fact that
no significant modelling failures were identified can be considered a validation
of this structure across a range of fresh water bodies.
H. Wilson . F. Recknagel
For the remaining model structures, the fact that hidden layer neurons make
little difference to predictive performance means that it is not dear that the ANN
is finding a consistent non-linearity. In these cases, it can be expected that a linear
model will perform nearly as weIl as an ANN. However, despite the lack of
evidence for improved RMS error, examination of time series plots revealed a
subtle performance improvement with hidden layer nodes. One explanation for
this observation is that the RMS error penalises small timing errors of prediction
much more heavily than a visual analysis of the traces. For example, a model that
correctly predicts the magnitude of a bloom, but predicts it one time step too late
or too early will be evaluated more favourably by a visual analysis than a model
that makes no prediction at all. However, RMS error would rate such a model
worse than a non-prediction, since there is a false positive as weIl as a false
negative prediction.
The bootstrap aggregation procedure was shown to be highly effective at
accounting for variance in ANN models - particularly in cases where there was
strong overfitting exhibited. However, it is important to note that such a benefit
comes at the cost of several drawbacks. Firstly, the bootstrap aggregate model is
constrained to being conservative in the context of small datasets, because over a
third of the data is left out of training on average. Secondly, the use of such a
procedure in the context of a time series raises questions about its effect on the
independence of test data, since training data may be sampled from random
positions in the time series. In the real world, it is impossible for the model to
make use of information regarding water quality that is in the future. However,
random resampling of training data provides the model with that information since
it has freedom to pick training records from a short time period in the future in
relation to a given test set record. In this application, there can be two arguments
against such objections relating to test set independence:
Any positive bias of the model performance estimation caused by
representation of "future" information in the training set counterbalances the
negative bias caused by holding out one third of the data from training.
Since most time series have sampling frequencies of between 2 weeks and one
month, it is likely that the serial correlation in the data that causes bias due to
"future" information is insignificant.
One way of overcoming this problem may be to break the data up into
randomly sampled chunks of several months or years.
14.6
Conclusions
This study has established the following findings:
1. The generic ANN structure considering phosphorous, nitrogen, secchi disc
depth and temperature as inputs, and chlorophyll a concentration as output,
achieved a reasonable level of predictive success for every model. The fact that
no significant modelling failures were identified can be considered a validation
of this structure across a range of fresh water bodies.