Chapter 10
The Evaluation of
Forecasts
by Robert E. Livezey
10.1 Introduction
The ultimate goal of climate dynamics is the prediction of climate variability
on interannual to interdecadal and longer timescales. Two necessary, but not
sufficient, conditions for the usefulness of such forecasts is that they have real
skill in discriminating one forecast situation from another and that this skill
is known in advance by the user. Two "no-skill" forecast schemes would be
random forecasts or constant (e.g. fixed climatology) forecasts. An additional
factor relevant to a scheme's usefulness is its efficiency - in other words its
skill with respect to more simple forecast schemes (like persistence of existing
conditions).
The evaluation of these two crucial factors to the utility of forecasts, absolute and relative skill, are the concerns ofthis chapter. The information contained here, combined with some of the considerations in preceding chapters
(especially Chapter 10) regarding tests of null hypotheses (here hypotheses
of no skill or no difference in skill), should be sufficient to design and conduct a wide range of forecast assessments. One exception is the evaluation of
probability forecasts, for which the reader is referred to Daan (1985). Also,
ultimately forecast usefulness depends on the costjbenefit environment of
individual users. Existing analyses of user-dependent utility are few in number (see Mjelde et al., 1993, for arecent example), often quite complex, and
beyond the scope of this chapter.
The Evaluation of
Forecasts
by Robert E. Livezey
10.1 Introduction
The ultimate goal of climate dynamics is the prediction of climate variability
on interannual to interdecadal and longer timescales. Two necessary, but not
sufficient, conditions for the usefulness of such forecasts is that they have real
skill in discriminating one forecast situation from another and that this skill
is known in advance by the user. Two "no-skill" forecast schemes would be
random forecasts or constant (e.g. fixed climatology) forecasts. An additional
factor relevant to a scheme's usefulness is its efficiency - in other words its
skill with respect to more simple forecast schemes (like persistence of existing
conditions).
The evaluation of these two crucial factors to the utility of forecasts, absolute and relative skill, are the concerns ofthis chapter. The information contained here, combined with some of the considerations in preceding chapters
(especially Chapter 10) regarding tests of null hypotheses (here hypotheses
of no skill or no difference in skill), should be sufficient to design and conduct a wide range of forecast assessments. One exception is the evaluation of
probability forecasts, for which the reader is referred to Daan (1985). Also,
ultimately forecast usefulness depends on the costjbenefit environment of
individual users. Existing analyses of user-dependent utility are few in number (see Mjelde et al., 1993, for arecent example), often quite complex, and
beyond the scope of this chapter.