160
Chapter 9: Field Intercomparison
assessment of the statistical significance of the results. Here ''fishing expedition" refers to the entire collection of statistics examined without apriori
expectations. For instance, foHowing the example above, if more than two
sets of GCM simulations are searched two-by-two for differences in the average wintertime temperature anomaly pattern over Europe then significance
tests must ac count for the multiple difference maps as weH as the multiple
locations on each map. lncorrect inferences are more likely if such tests are
conducted for only the "best" difference map.
These ideas will be illustrated in Section 9.2 through an extended but
relatively simple example taken from Livezey and Chen (1983). Two strategies for conducting permutation techniques and some of their properties in
a multivariate test environment without serial correlation with and without
interdependence among the variables will be introduced in Section 9.3, which
relies heavily on the work of Zwiers (1987). Next, a discussion of the effects
of serial correlation on permutation techniques and strategies to account for
the effects will be presented in Section 9.4. This last material is a synthesis
and condensation of ideas developed by F.W. Zwiers and colleagues (Zwiers,
1990; Thiebaux and Zwiers, 1984; Zwiers and H. von Storch, 1994) and K.
E. Trenberth (1984).
Climate analysts who must design and/or evaluate studies of variability on
interannual and longer time scales should find the already cited papers enormously useful if not mandatory sources of information and examples pertinent
to the methods of this chapter. Further examples can be found in Livezey
(1985), Zwiers and Boer (1987), Santer and Wigley (1990), Preisendorfer
and Barnett (1983), and Mielke et al. (1981). The latter two references offer
two formal approaches to application of permutation methods. In addition,
Wigley and Santer (1990) present a detailed discussion of the relative merits of measures available for a. variety of experimental situations. Measures
specifically intended for evaluation of forecasts and forecast methodologies
and special considerations for the conduct of these assessments are presented
in Chapter 10.
9.2 Motivation for
Permutation and Monte Carlo Testing
This author's introduction to the type of problem towards which permutation
techniques are directed forms the central focus of Livezey and Chen (1983)
and will be used to illustrate the effects of multiplicity and interdependence
in multivariate hypothesis testing.
The experimental environment of an analysis by W. Y. Chen satisfies all
of the conditions mentioned in Section 9.1 that suggest the use of permutation or Monte Carlo hypothesis testing: Correlations between a SOl and 700
hpa DJF heights from 29 years of data were computed at every point of a
Chapter 9: Field Intercomparison
assessment of the statistical significance of the results. Here ''fishing expedition" refers to the entire collection of statistics examined without apriori
expectations. For instance, foHowing the example above, if more than two
sets of GCM simulations are searched two-by-two for differences in the average wintertime temperature anomaly pattern over Europe then significance
tests must ac count for the multiple difference maps as weH as the multiple
locations on each map. lncorrect inferences are more likely if such tests are
conducted for only the "best" difference map.
These ideas will be illustrated in Section 9.2 through an extended but
relatively simple example taken from Livezey and Chen (1983). Two strategies for conducting permutation techniques and some of their properties in
a multivariate test environment without serial correlation with and without
interdependence among the variables will be introduced in Section 9.3, which
relies heavily on the work of Zwiers (1987). Next, a discussion of the effects
of serial correlation on permutation techniques and strategies to account for
the effects will be presented in Section 9.4. This last material is a synthesis
and condensation of ideas developed by F.W. Zwiers and colleagues (Zwiers,
1990; Thiebaux and Zwiers, 1984; Zwiers and H. von Storch, 1994) and K.
E. Trenberth (1984).
Climate analysts who must design and/or evaluate studies of variability on
interannual and longer time scales should find the already cited papers enormously useful if not mandatory sources of information and examples pertinent
to the methods of this chapter. Further examples can be found in Livezey
(1985), Zwiers and Boer (1987), Santer and Wigley (1990), Preisendorfer
and Barnett (1983), and Mielke et al. (1981). The latter two references offer
two formal approaches to application of permutation methods. In addition,
Wigley and Santer (1990) present a detailed discussion of the relative merits of measures available for a. variety of experimental situations. Measures
specifically intended for evaluation of forecasts and forecast methodologies
and special considerations for the conduct of these assessments are presented
in Chapter 10.
9.2 Motivation for
Permutation and Monte Carlo Testing
This author's introduction to the type of problem towards which permutation
techniques are directed forms the central focus of Livezey and Chen (1983)
and will be used to illustrate the effects of multiplicity and interdependence
in multivariate hypothesis testing.
The experimental environment of an analysis by W. Y. Chen satisfies all
of the conditions mentioned in Section 9.1 that suggest the use of permutation or Monte Carlo hypothesis testing: Correlations between a SOl and 700
hpa DJF heights from 29 years of data were computed at every point of a