178
Chapter 10: The Evaluation of Forecasts
The remainder of the chapter will highlight three related topics. First,
basic ingredients and considerations for objective fore cast verification will
be discussed in Section 10.2, which borrows heavily from Livezey (1987).
This is followed in Sections 10.3 and 10.4 by overviews of skill measures in
modern use and some of their important properties and interrelationships
for both categorical and continuous forecasts respectively. Virtually all of
the material in these sections is synthesized from papers that have appeared
in the last six years. Finally, the difficult problem of evaluating statistical
prediction schemes in developmental situations where it is not feasible to
reserve a large portion of the total sampie for independent testing is examined
in Section 10.5. Cross-validation (Michaelson, 1987) will be the emphasized
solution because of its versatility, but references to Monte Carlo and other
alternatives for evaluation ofmultiple (including screening) regression models
will be provided. A complementary sour ce rich in examples for much of the
material in this chapter is the forthcoming book by H. von Storch and Zwiers
(1995).
10.2 Considerations for Objective Verification
The skill of a fore cast er or forecast system is easy to overstate and this has
frequently been done both in and out of the refereed literat ure. Such overstatement inevitably leads to overstatement ab out the usefulness of forecasts,
though as pointed out above skill alone is not a sufficient condition for forecast
utility. Although the analysis of utility is beyond the scope of this chapter, it
is possible to describe four elements whose consideration should enhance the
possibility of objective assessment of the level of skill, and thereby of utility.
Some of these components may strike some as obvious or trivial. Nevertheless, it has been the experience of the author that they need to be continually
restated.
10.2.1 Quantification
Forecasts and verifying observations must be precisely defined, so that there
is no ambiguity whatsoever whether a particular forecast is right or wrong,
and, for the latter, what its error iso Also, it is important that it be clear
what is not being predicted. For example, predictions of the amplitude of a
particular atmospheric structure could be highly accurate but of little value
for most users if the variance of that structure represents only a small fraction
of total variance of the field.
Chapter 10: The Evaluation of Forecasts
The remainder of the chapter will highlight three related topics. First,
basic ingredients and considerations for objective fore cast verification will
be discussed in Section 10.2, which borrows heavily from Livezey (1987).
This is followed in Sections 10.3 and 10.4 by overviews of skill measures in
modern use and some of their important properties and interrelationships
for both categorical and continuous forecasts respectively. Virtually all of
the material in these sections is synthesized from papers that have appeared
in the last six years. Finally, the difficult problem of evaluating statistical
prediction schemes in developmental situations where it is not feasible to
reserve a large portion of the total sampie for independent testing is examined
in Section 10.5. Cross-validation (Michaelson, 1987) will be the emphasized
solution because of its versatility, but references to Monte Carlo and other
alternatives for evaluation ofmultiple (including screening) regression models
will be provided. A complementary sour ce rich in examples for much of the
material in this chapter is the forthcoming book by H. von Storch and Zwiers
(1995).
10.2 Considerations for Objective Verification
The skill of a fore cast er or forecast system is easy to overstate and this has
frequently been done both in and out of the refereed literat ure. Such overstatement inevitably leads to overstatement ab out the usefulness of forecasts,
though as pointed out above skill alone is not a sufficient condition for forecast
utility. Although the analysis of utility is beyond the scope of this chapter, it
is possible to describe four elements whose consideration should enhance the
possibility of objective assessment of the level of skill, and thereby of utility.
Some of these components may strike some as obvious or trivial. Nevertheless, it has been the experience of the author that they need to be continually
restated.
10.2.1 Quantification
Forecasts and verifying observations must be precisely defined, so that there
is no ambiguity whatsoever whether a particular forecast is right or wrong,
and, for the latter, what its error iso Also, it is important that it be clear
what is not being predicted. For example, predictions of the amplitude of a
particular atmospheric structure could be highly accurate but of little value
for most users if the variance of that structure represents only a small fraction
of total variance of the field.