Section 10.4: Measures and Relationships
1 n
MSE/ te = - L(fi - Xi)2
n i=l
191
(10.3)
For convenience suppose that the forecasts and observations can and have
been temporally standardized (to means of zero and unit standard deviations). This standardization is not possible for climatology forecasts. The
correlation is then
1 n
p/te
- L: fixi and
n i=l
(10.4)
MSE/ te = 2(1- P/te).
(10.5)
If the forecasts or observations are imperfectly standardized so that the sampIes do not both have zero means and unit standard deviations (i.e. standardization is based on a different sample's statistics, e.g. a different period's
climatology), the comparable expression to (10.5) contains additional terms
to ac count for bias and amplitude errors in the forecasts. These terms are
analogous to those developed in the next subsection for verification of fields.
However, Pfte remains insensitive to bias and amplitude errors.
The relationship (10.5) is plotted in Figure 10.6 from Barnston (1992).
Note from the figure or (10.5) that for random forecasts (Pfte = 0) MSEfte =
2, but from (10.3) for climatology forecasts (fi == 0) MSEfte = 1, so that
random forecasts have double the squared error of climatology forecasts. Note
further that for MSE/ te < 1, the level for climatology, Pfte > 0.5. The latter
is a commonly used criterion in medium to long range prediction for usability
of circulation forecasts.
MSEfte can be reduced by damping the Xi by an estimate of p/te. The
result of this is represented by the lower curve in Figure 10.6.
An additional useful skill measure called the Brier-based score is the percent
reduction of M S E from some reference or control fore cast system. If the
control is climatology, c, then from (10.3) the Brier-based score is
ß = MSEcte - MSE/ te = 1- MSEfte = 1- O"~
MSEcte
MSEcte
O"te
(10.6)
where the last term is the ratio of the error variance to the observed variance. Thus the Brier-based score with climatology as the reference is also
the proportion of explained variance.
If the forecasts and observations are standardized, the denominator in
(10.6) becomes one and, from (10.5)
ß = 2pfte -1.
(10.7)
Thus, for ß> 0, Pfte > 0.5, and for random forecasts ß = -1.
To close this subsection, it is worth recalling from Section 10.2.4 that it
is not appropriate to compare a forecast scheme to persistence if M SEfte
is used as the measure of accuracy, because M SE/te is relatively large for
1 n
MSE/ te = - L(fi - Xi)2
n i=l
191
(10.3)
For convenience suppose that the forecasts and observations can and have
been temporally standardized (to means of zero and unit standard deviations). This standardization is not possible for climatology forecasts. The
correlation is then
1 n
p/te
- L: fixi and
n i=l
(10.4)
MSE/ te = 2(1- P/te).
(10.5)
If the forecasts or observations are imperfectly standardized so that the sampIes do not both have zero means and unit standard deviations (i.e. standardization is based on a different sample's statistics, e.g. a different period's
climatology), the comparable expression to (10.5) contains additional terms
to ac count for bias and amplitude errors in the forecasts. These terms are
analogous to those developed in the next subsection for verification of fields.
However, Pfte remains insensitive to bias and amplitude errors.
The relationship (10.5) is plotted in Figure 10.6 from Barnston (1992).
Note from the figure or (10.5) that for random forecasts (Pfte = 0) MSEfte =
2, but from (10.3) for climatology forecasts (fi == 0) MSEfte = 1, so that
random forecasts have double the squared error of climatology forecasts. Note
further that for MSE/ te < 1, the level for climatology, Pfte > 0.5. The latter
is a commonly used criterion in medium to long range prediction for usability
of circulation forecasts.
MSEfte can be reduced by damping the Xi by an estimate of p/te. The
result of this is represented by the lower curve in Figure 10.6.
An additional useful skill measure called the Brier-based score is the percent
reduction of M S E from some reference or control fore cast system. If the
control is climatology, c, then from (10.3) the Brier-based score is
ß = MSEcte - MSE/ te = 1- MSEfte = 1- O"~
MSEcte
MSEcte
O"te
(10.6)
where the last term is the ratio of the error variance to the observed variance. Thus the Brier-based score with climatology as the reference is also
the proportion of explained variance.
If the forecasts and observations are standardized, the denominator in
(10.6) becomes one and, from (10.5)
ß = 2pfte -1.
(10.7)
Thus, for ß> 0, Pfte > 0.5, and for random forecasts ß = -1.
To close this subsection, it is worth recalling from Section 10.2.4 that it
is not appropriate to compare a forecast scheme to persistence if M SEfte
is used as the measure of accuracy, because M SE/te is relatively large for