Section 9.3: Permutation Procedures
169
In Zwiers (1987) the power to detect known uniform signals was also examined with three sampie sizes, n =5, 10, and 20. This is therefore a test
situation where the effects of sampling resolution can also be studied. The
most important result is that the permutation methods were at least as good
(within estimation error) at detecting the signals (rejecting the null hypothesis) as the exact methods, with pp again slightly superior to BP and the use
of D 2 slightly superior to the counting norm. The other notable result is that
the power of BP and pp both were substantially lower for n = 5, perhaps
because of severe es tim at ion error in the empirical distribution functions. In
this case !!i was as little as ;4' a level at which no method would be expected
to perform weIl, except with a very strong signal.
Obviously, these results are not directly transferable to situations where the
reference distributions are different or unknown, but they lend considerable
confidence to the application of pp and BP in these instances.
9.3.4 Interdependence Among Field Variables
If the m vector elements of X and Y respectively are intercorrelated the
situation is analogous to testing differences in means of maps of climate
data where there is spatial correlation between grid locations. Because BP
and pp build up empirical distribution functions from sam pies in which the
covariance structures of the two fields are preserved the capability of the tests
to estimate appropriate significance levels is unaffected. What is affected is
the power of signal detection as has already been implied in the discussion
in Section 9.2.2.
Zwiers (1987) examines these effects in a modified experiment in which he
imposes cross-correlation on the randomly generated rn-dimensional vectors
in two different ways, one that mimics an expanding network (recall the PNA
vs. hemispheric domain example) and one that mimics increasing resolution
(the summer vs. winter example) as m increases. To estimate power a strong
uniform signal is added to the m components of the sam pie of X. Only pp
with D2 is used for the permutation tests, otherwise the experiments are the
same as before, and the principal result of interest is the rate of rejections of
the null hypothesis. In this experiment, however, the best performing tests
are those with high rates because the null hypothesis is false in all instances.
The exact test becomes a Hotelling-T 2 test applied with known covariance
structure.
The power of the permutation tests is first compared to the exact tests
and the case with independent vector variables. Results of these tests are
presented in Table 9.2 under "Multivariate test". First note the loss of power
of both the pp and exact tests with the addition of spatial correlation. This
is comparable to a reduction in network density.
Second, with spatial correlation the power of the pp test is only slightly
inferior overall to the exact test in the case of the expanding network and is
169
In Zwiers (1987) the power to detect known uniform signals was also examined with three sampie sizes, n =5, 10, and 20. This is therefore a test
situation where the effects of sampling resolution can also be studied. The
most important result is that the permutation methods were at least as good
(within estimation error) at detecting the signals (rejecting the null hypothesis) as the exact methods, with pp again slightly superior to BP and the use
of D 2 slightly superior to the counting norm. The other notable result is that
the power of BP and pp both were substantially lower for n = 5, perhaps
because of severe es tim at ion error in the empirical distribution functions. In
this case !!i was as little as ;4' a level at which no method would be expected
to perform weIl, except with a very strong signal.
Obviously, these results are not directly transferable to situations where the
reference distributions are different or unknown, but they lend considerable
confidence to the application of pp and BP in these instances.
9.3.4 Interdependence Among Field Variables
If the m vector elements of X and Y respectively are intercorrelated the
situation is analogous to testing differences in means of maps of climate
data where there is spatial correlation between grid locations. Because BP
and pp build up empirical distribution functions from sam pies in which the
covariance structures of the two fields are preserved the capability of the tests
to estimate appropriate significance levels is unaffected. What is affected is
the power of signal detection as has already been implied in the discussion
in Section 9.2.2.
Zwiers (1987) examines these effects in a modified experiment in which he
imposes cross-correlation on the randomly generated rn-dimensional vectors
in two different ways, one that mimics an expanding network (recall the PNA
vs. hemispheric domain example) and one that mimics increasing resolution
(the summer vs. winter example) as m increases. To estimate power a strong
uniform signal is added to the m components of the sam pie of X. Only pp
with D2 is used for the permutation tests, otherwise the experiments are the
same as before, and the principal result of interest is the rate of rejections of
the null hypothesis. In this experiment, however, the best performing tests
are those with high rates because the null hypothesis is false in all instances.
The exact test becomes a Hotelling-T 2 test applied with known covariance
structure.
The power of the permutation tests is first compared to the exact tests
and the case with independent vector variables. Results of these tests are
presented in Table 9.2 under "Multivariate test". First note the loss of power
of both the pp and exact tests with the addition of spatial correlation. This
is comparable to a reduction in network density.
Second, with spatial correlation the power of the pp test is only slightly
inferior overall to the exact test in the case of the expanding network and is