Section 10.2: Considerations for Objective Verification
179
10.2.2 Authentication
Verifications of forecast methodologies are conducted on one or another of
two types of prediction sets, often referred to as forecasts and hindcasts. In
the former prognostic models or experience are based only on information
preceding independent events that are predicted, while in the latter they are
based on information either following or bot4 preceding and following the
target forecast period.
In operational situations only forecasts are possible and their issuance by
firm deadlines preceding the forecast period is sufficient guarantee that they
are true forecasts. In the laboratory, either type is possible and special attention must be paid to ensure that no skill inflation takes place. For forecasts,
this means that absolutely no use is made offuture information in formulating
prognostic procedures or in making actual forecasts. A frequently encountered and subtle example of a violation of this condition would be a set of
zero-lead forecasts made using predictor series subjected to a symmetric filter; in this instance the forecasts would not be possible in practice because
future information would be necessary to determine current values of the filtered predictors. Skill estimates for such laboratory forecasts will invariably
be inflated.
In the case of laboratory hindcasts, uninflated estimates of forecast skill
are impossible unless it can be assumed that the climate is reasonably stationary. If the assumption is appropriate then hindcasts can be considered
true forecasts if the target forecast period is statistically independent of the
model developmental sampie. Much more will be said about these matters
in Section 10.5.
10.2.3 Description of Probability Distributions
Because there is generally a non-trivial probability that at least some number of random categorical forecasts can be made correctly or that the mean
error of some number of random continuous forecasts will be less than some
amount, the notion of forecast skill in the absence of information ab out the
joint probability distribution offorecasts and observations is meaningless (e.g.
a statement like "the forecasts are 80% correct" is a statement ab out the accuracy not skill). Thus it is not possible to make objective assessments of
forecast skill without some information about this joint distribution.
A very instructive example occurs for the prediction of monthly mean precipitation. The distribution of this quantity for a wide range of climates is
negatively skewed, i.e. the median is less than the mean, as schematically illustrated in Figure 10.1. For most U. S.locations in all seasons this skewness
is such that about 60% of cases have below normal preciptitation and 40%
above. Thus the prob ability of randomly making a correct fore cast of above
or below normal precipitation is not 0.5 (i.e. 2.0.5 2 ), but is 0.52 (0.6 2 + 0.4 2 ).
An equally unintelligent (and useless) forecast procedure would be to fore-
179
10.2.2 Authentication
Verifications of forecast methodologies are conducted on one or another of
two types of prediction sets, often referred to as forecasts and hindcasts. In
the former prognostic models or experience are based only on information
preceding independent events that are predicted, while in the latter they are
based on information either following or bot4 preceding and following the
target forecast period.
In operational situations only forecasts are possible and their issuance by
firm deadlines preceding the forecast period is sufficient guarantee that they
are true forecasts. In the laboratory, either type is possible and special attention must be paid to ensure that no skill inflation takes place. For forecasts,
this means that absolutely no use is made offuture information in formulating
prognostic procedures or in making actual forecasts. A frequently encountered and subtle example of a violation of this condition would be a set of
zero-lead forecasts made using predictor series subjected to a symmetric filter; in this instance the forecasts would not be possible in practice because
future information would be necessary to determine current values of the filtered predictors. Skill estimates for such laboratory forecasts will invariably
be inflated.
In the case of laboratory hindcasts, uninflated estimates of forecast skill
are impossible unless it can be assumed that the climate is reasonably stationary. If the assumption is appropriate then hindcasts can be considered
true forecasts if the target forecast period is statistically independent of the
model developmental sampie. Much more will be said about these matters
in Section 10.5.
10.2.3 Description of Probability Distributions
Because there is generally a non-trivial probability that at least some number of random categorical forecasts can be made correctly or that the mean
error of some number of random continuous forecasts will be less than some
amount, the notion of forecast skill in the absence of information ab out the
joint probability distribution offorecasts and observations is meaningless (e.g.
a statement like "the forecasts are 80% correct" is a statement ab out the accuracy not skill). Thus it is not possible to make objective assessments of
forecast skill without some information about this joint distribution.
A very instructive example occurs for the prediction of monthly mean precipitation. The distribution of this quantity for a wide range of climates is
negatively skewed, i.e. the median is less than the mean, as schematically illustrated in Figure 10.1. For most U. S.locations in all seasons this skewness
is such that about 60% of cases have below normal preciptitation and 40%
above. Thus the prob ability of randomly making a correct fore cast of above
or below normal precipitation is not 0.5 (i.e. 2.0.5 2 ), but is 0.52 (0.6 2 + 0.4 2 ).
An equally unintelligent (and useless) forecast procedure would be to fore-