268
H. Wilson . F. Recknagel
• Heuristics such as calculation of the number of hidden nodes from the number
of training records and/or inputs and outputs typically produce very large
ranges of meta-parameter values.
• Where empirical meta-parameter optimisation is performed, the upper limit to
modelling effort is potentially boundless, since there is no clear answer to the
question of when to stop the train-adjust-train cycle.
• There may be doubts about the independence of test data used to calibrate
meta-parameters. While the use of a tuning set in addition to the test data
increases the preceived validation independence (Maier and Dandy 2000), it is
a data inefficient strategy in the context of applications that are highly
constrained in terms of data availability.
A simple approach for improving ANN model performance called "bagging"
(short for "bootstrap aggregation") was introduced by Breiman (1996). Bagging is
a so-called "perturb and combine" approach, where, as the name suggests, a
number of peturbed models are approximated and then combined by averaging.
A veraging reduces the variance component of prediction error thus lowering the
risk of overfitting of ANN models. The perturbation step in the bagging
procedure is achieved by varying the training data on wh ich the ANN is trained
through bootstrap resampling (Efron and Tibshirani 1993) from the available
database. The use of the boots trap in this way allows simulation of the effect of
database sampling variability on the model. Bootstrapping mayaiso be used for
calculation of confidence intervals (e.g. Baxt and White 1995).
underfitting
optimum
overfitting
total error
(bias + variance)
1
1
1
1
I
I
I
I
I
I
I
I
11 variance
bias
-'.-.
few hidden nodes
many hidden nodes
short training
long training
Penalisation of ANN approximation
Fig.14.1. Total prediction error, bias, and variance versus ANN penalisation by
limiting training time, the number of hidden nodes, or other means.
H. Wilson . F. Recknagel
• Heuristics such as calculation of the number of hidden nodes from the number
of training records and/or inputs and outputs typically produce very large
ranges of meta-parameter values.
• Where empirical meta-parameter optimisation is performed, the upper limit to
modelling effort is potentially boundless, since there is no clear answer to the
question of when to stop the train-adjust-train cycle.
• There may be doubts about the independence of test data used to calibrate
meta-parameters. While the use of a tuning set in addition to the test data
increases the preceived validation independence (Maier and Dandy 2000), it is
a data inefficient strategy in the context of applications that are highly
constrained in terms of data availability.
A simple approach for improving ANN model performance called "bagging"
(short for "bootstrap aggregation") was introduced by Breiman (1996). Bagging is
a so-called "perturb and combine" approach, where, as the name suggests, a
number of peturbed models are approximated and then combined by averaging.
A veraging reduces the variance component of prediction error thus lowering the
risk of overfitting of ANN models. The perturbation step in the bagging
procedure is achieved by varying the training data on wh ich the ANN is trained
through bootstrap resampling (Efron and Tibshirani 1993) from the available
database. The use of the boots trap in this way allows simulation of the effect of
database sampling variability on the model. Bootstrapping mayaiso be used for
calculation of confidence intervals (e.g. Baxt and White 1995).
underfitting
optimum
overfitting
total error
(bias + variance)
1
1
1
1
I
I
I
I
I
I
I
I
11 variance
bias
-'.-.
few hidden nodes
many hidden nodes
short training
long training
Penalisation of ANN approximation
Fig.14.1. Total prediction error, bias, and variance versus ANN penalisation by
limiting training time, the number of hidden nodes, or other means.