Section 10.3: Measures and Relationships
187
(Radok,1988). Table 10.2a, which is reproduced from Barnston (1992) is a
template for the assignment of weights for each forecast outcome to determine
H.
In this specific example the Heidke score is equitable (Gandin and Murphy,
1992) because no advantage can be gained by fore casting one class or another
all of the time. Thus an equitable skill score is one in which the biased
forecasting described in the examples of Section 10.2.3 would not result in
higher skills. Suppose forecast and observed classes are not equiprobable, say
with (A,N,B) probabilitiesof(0.3, 0.4, 0.3) and E = (0.3 2 +0.4 2 +O.3 2 )T=
0.34T. In this instance, the Heidke is inequitable, because all forecasts ofnear
normal lead to H = O.4T and a positive score, but all above or below normal
lead to H = 0.3T and a negative score.
b) Refinements
The Heidke score can be thought of as a "poor man's" correlation score (the
correlation score is defined in Section 10.4) so it is ofsome interest to examine
the relationship between these respective categorical and continuous forecast
scores. The results of Monte Carlo simulations by Barnston (1992) in Figures
10.5a and b show that the Heidke is generally sm aller than the correlation and
that this difference increases with increasing number of equi-probable classes
for the categorical forecasts. In other words, for a given level offorecastability
(as measured by correlation between forecasts and observations) the largest
Heidke scores will be for categorical forecasts with the fewest classes. This
undesirable property is related to the sour ce of a more serious deficiency of
the Heidke score, namely its inability to distinguish between two forecasts
with the same number of hits but different numbers of one-, two-, etc. -class
misses. Both problems are caused by the failure to ac count for the severity
of misses.
An adjusted scheme that linearly discriminates between zero-, one- and
two-class misses is shown in Table 10.2b, in which a two-class miss is now
regarded as a loss of a hit. To use (10.1) with this scheme requires the
redefinition of E as the expected number of hits with possible loss of hits
from two-class misses taken into account, and corresponding redefinition of
H. To accommodate this and more general redefinitions with more complex
reward/penalty matrices, let the entries for boxes (i, j) in any of the panels
in Table 10.2 be denoted by Sij and nij the number of cases for the box.
Now,
T 3
3
E = - L: Sij and H = L: Sijnij
(10.2)
9 .. 1
.. 1
1,)=
',)=
Note from Figures 10.5c and d (again from Monte Carlo simulations conducted by Barnston, 1992) that this scheme is an improvement on the Heidke
score in two respects: The score is generally closer to the correlation score
and scores for different numbers of classes are quite similar.
187
(Radok,1988). Table 10.2a, which is reproduced from Barnston (1992) is a
template for the assignment of weights for each forecast outcome to determine
H.
In this specific example the Heidke score is equitable (Gandin and Murphy,
1992) because no advantage can be gained by fore casting one class or another
all of the time. Thus an equitable skill score is one in which the biased
forecasting described in the examples of Section 10.2.3 would not result in
higher skills. Suppose forecast and observed classes are not equiprobable, say
with (A,N,B) probabilitiesof(0.3, 0.4, 0.3) and E = (0.3 2 +0.4 2 +O.3 2 )T=
0.34T. In this instance, the Heidke is inequitable, because all forecasts ofnear
normal lead to H = O.4T and a positive score, but all above or below normal
lead to H = 0.3T and a negative score.
b) Refinements
The Heidke score can be thought of as a "poor man's" correlation score (the
correlation score is defined in Section 10.4) so it is ofsome interest to examine
the relationship between these respective categorical and continuous forecast
scores. The results of Monte Carlo simulations by Barnston (1992) in Figures
10.5a and b show that the Heidke is generally sm aller than the correlation and
that this difference increases with increasing number of equi-probable classes
for the categorical forecasts. In other words, for a given level offorecastability
(as measured by correlation between forecasts and observations) the largest
Heidke scores will be for categorical forecasts with the fewest classes. This
undesirable property is related to the sour ce of a more serious deficiency of
the Heidke score, namely its inability to distinguish between two forecasts
with the same number of hits but different numbers of one-, two-, etc. -class
misses. Both problems are caused by the failure to ac count for the severity
of misses.
An adjusted scheme that linearly discriminates between zero-, one- and
two-class misses is shown in Table 10.2b, in which a two-class miss is now
regarded as a loss of a hit. To use (10.1) with this scheme requires the
redefinition of E as the expected number of hits with possible loss of hits
from two-class misses taken into account, and corresponding redefinition of
H. To accommodate this and more general redefinitions with more complex
reward/penalty matrices, let the entries for boxes (i, j) in any of the panels
in Table 10.2 be denoted by Sij and nij the number of cases for the box.
Now,
T 3
3
E = - L: Sij and H = L: Sijnij
(10.2)
9 .. 1
.. 1
1,)=
',)=
Note from Figures 10.5c and d (again from Monte Carlo simulations conducted by Barnston, 1992) that this scheme is an improvement on the Heidke
score in two respects: The score is generally closer to the correlation score
and scores for different numbers of classes are quite similar.