172
Chapter 9: Field Intercomparison
Table 9.3: Rejection rate 'using the 5% critial point of tests using the pp
together with the Euclidean distance statistic D 2 , when sampies are taken
from Gaussian Markov processes. (From Zwiers, 1987).
slze
m Pi = 0.3 Pi = 0.75
2
0.12
0.55
4
0.28
0.81
8
0.32
0.91
12
0.40
0.98
24
0.72
1.00
empirical distribution functions that are not representative of the population
of the observed statistic to be tested. The severity of the problem is apparent
in Table 9.3 from Zwiers (1987) which shows the results of modifying the
tests described in Sections 9.3.1 - 9.3.3 so that the elements of X and Y
are independently generated from first-order Markov AR(l) processes with
lag-1 autocorrelations of respectively 0.3 and 0.75. Clearly the tests are now
useless.
Zwiers (1990) examined the effects in more detail with univariate (m = 1)
difference of means tests using pp based on 1000 resamplings each with
sampIes of n = 20,40, and 100 taken from two random series generated from
AR(l) processes with a range of lag-1 autocorrelations, Pi. To assess its
performance pp was performed on 2080 paired series for every combination
of n and Pi. Overall rejection rates are summarized in Figure 9.4 for nominal
levels of 1, 2.5, 5, and 10%. Several things are very clear from the curves
shown, including the insensitivity of the effects of serial correlation to the
sampIe sizes considered and a universal tendency for liberalization of the tests,
i.e. rejection rates higher than their nominal values making it more likely that
the null hypothesis will be erroneously rejected with use of the tests. This
is expected because empirical distribution functions will be narrower than
the populations from which the original sampIes were drawn because of the
destruction of serial correlation by the resampling.
Solutions to the problem of accounting for serial correlation in permutation and Monte Carlo test situations are ofl'ered below. Some are available
only under special circumstances while others can be tried at the discretion
of the analyst. The list is almost certainly not all-inclusive but does represent a synthesis of the ideas presented in the four papers mentioned in the
introduction and the author's experience:
Chapter 9: Field Intercomparison
Table 9.3: Rejection rate 'using the 5% critial point of tests using the pp
together with the Euclidean distance statistic D 2 , when sampies are taken
from Gaussian Markov processes. (From Zwiers, 1987).
slze
m Pi = 0.3 Pi = 0.75
2
0.12
0.55
4
0.28
0.81
8
0.32
0.91
12
0.40
0.98
24
0.72
1.00
empirical distribution functions that are not representative of the population
of the observed statistic to be tested. The severity of the problem is apparent
in Table 9.3 from Zwiers (1987) which shows the results of modifying the
tests described in Sections 9.3.1 - 9.3.3 so that the elements of X and Y
are independently generated from first-order Markov AR(l) processes with
lag-1 autocorrelations of respectively 0.3 and 0.75. Clearly the tests are now
useless.
Zwiers (1990) examined the effects in more detail with univariate (m = 1)
difference of means tests using pp based on 1000 resamplings each with
sampIes of n = 20,40, and 100 taken from two random series generated from
AR(l) processes with a range of lag-1 autocorrelations, Pi. To assess its
performance pp was performed on 2080 paired series for every combination
of n and Pi. Overall rejection rates are summarized in Figure 9.4 for nominal
levels of 1, 2.5, 5, and 10%. Several things are very clear from the curves
shown, including the insensitivity of the effects of serial correlation to the
sampIe sizes considered and a universal tendency for liberalization of the tests,
i.e. rejection rates higher than their nominal values making it more likely that
the null hypothesis will be erroneously rejected with use of the tests. This
is expected because empirical distribution functions will be narrower than
the populations from which the original sampIes were drawn because of the
destruction of serial correlation by the resampling.
Solutions to the problem of accounting for serial correlation in permutation and Monte Carlo test situations are ofl'ered below. Some are available
only under special circumstances while others can be tried at the discretion
of the analyst. The list is almost certainly not all-inclusive but does represent a synthesis of the ideas presented in the four papers mentioned in the
introduction and the author's experience: