4 The Analysis of Event-Related Potentials
75
problem [41, 97] and is very common in ERP studies, where several points in time,
space and frequency are to be investigated. Let M be the number of hypotheses to
be tested and M 0 be the number of true null hypotheses. Testing each hypothesis
independently at the α level, the expectation of false discoveries is M 0 × α. Thus,
if all null hypotheses are actually true, i.e., M 0 M, we expect to commit on the
average (100 × α) % false discoveries. This is, of course, an unacceptable error rate.
Nonetheless, the more hypotheses are false and the more they are correlated, the more
the error rate is reduced. ERP data is highly correlated along adjacent time points,
spatial derivations and frequency. Therefore, special care should be undertaken in
ERP statistical analysis to ensure that the error rate is controlled while preserving
statistical power, that is, while preserving an acceptable chance to detect those null
hypotheses that are false. Two families of statistical procedures have been employed
in ERP studies with this aim: those controlling the family-wise error rate (FWER)
and those controlling the false-discovery rate (FDR).
The family-wise error rate (FWER) is the probability of making one or more false
discoveries among all hypotheses. A procedure controlling the FWER at the α level
ensures that the probability of committing even only one false discovery is less than or
equal to α, regardless the number of tests and how many null hypotheses are actually
true. The popular Bonferroni procedure belongs to this family; each hypothesis is
tested at level α/M instead that at level α. Sequential Bonferroni-like procedures like
the one proposed by Holm [42] also control the FWER, while featuring higher power.
However, all Bonferroni-like procedures fail to take into consideration explicitly the
correlation structure of the hypotheses, thus they are unduly conservative, the more
so the higher the number of hypotheses to be tested.
An important general class of test procedures controlling the FWER is known as
p-min permutation tests [75, 97], tracing back to the seminal work of Fisher [35] and
Pitman [80–82]. Permutation tests are able to account adaptively for any correlation
structure of hypotheses, regardless of its form and degree. Also, they do not need a
distributional model for the observed variables, e.g., Gaussianity, as required by ttests, ANOVA etc. [6, 30, 35, 43, 46, 75, 80–82, 91, 92, 97]. Even more appealing, one
may extract whatever variable from the data and perform a valid test, thus we are not
limited to test on central location, correlation, etc. Depending on the experimental
design, even the random sampling assumption may be relaxed [30]. Given these
characteristics, permutation tests are ideal options for testing hypotheses in ERP
studies and have received much attention in the neuroimaging community [1, 43, 76].
Permutation tests are available for classical correlation, within- and betweensubject mean difference tests, as well as for testing the main effects in ANOVA
designs [30]. However, a straightforward permutation test for interaction effects in
ANOVA designs does not exist, although some solutions have been proposed [75].
This is a major limitation if more than one independent variable is manipulated in the
experiment. Also, like other resampling methods such as bootstrap and Monte Carlo,
permutation tests require intense computations. For large data sets, permutation tests
may be time consuming, although this is rarely a concern with modern computers
and the typical size of data sets in ERP analysis.
75
problem [41, 97] and is very common in ERP studies, where several points in time,
space and frequency are to be investigated. Let M be the number of hypotheses to
be tested and M 0 be the number of true null hypotheses. Testing each hypothesis
independently at the α level, the expectation of false discoveries is M 0 × α. Thus,
if all null hypotheses are actually true, i.e., M 0 M, we expect to commit on the
average (100 × α) % false discoveries. This is, of course, an unacceptable error rate.
Nonetheless, the more hypotheses are false and the more they are correlated, the more
the error rate is reduced. ERP data is highly correlated along adjacent time points,
spatial derivations and frequency. Therefore, special care should be undertaken in
ERP statistical analysis to ensure that the error rate is controlled while preserving
statistical power, that is, while preserving an acceptable chance to detect those null
hypotheses that are false. Two families of statistical procedures have been employed
in ERP studies with this aim: those controlling the family-wise error rate (FWER)
and those controlling the false-discovery rate (FDR).
The family-wise error rate (FWER) is the probability of making one or more false
discoveries among all hypotheses. A procedure controlling the FWER at the α level
ensures that the probability of committing even only one false discovery is less than or
equal to α, regardless the number of tests and how many null hypotheses are actually
true. The popular Bonferroni procedure belongs to this family; each hypothesis is
tested at level α/M instead that at level α. Sequential Bonferroni-like procedures like
the one proposed by Holm [42] also control the FWER, while featuring higher power.
However, all Bonferroni-like procedures fail to take into consideration explicitly the
correlation structure of the hypotheses, thus they are unduly conservative, the more
so the higher the number of hypotheses to be tested.
An important general class of test procedures controlling the FWER is known as
p-min permutation tests [75, 97], tracing back to the seminal work of Fisher [35] and
Pitman [80–82]. Permutation tests are able to account adaptively for any correlation
structure of hypotheses, regardless of its form and degree. Also, they do not need a
distributional model for the observed variables, e.g., Gaussianity, as required by ttests, ANOVA etc. [6, 30, 35, 43, 46, 75, 80–82, 91, 92, 97]. Even more appealing, one
may extract whatever variable from the data and perform a valid test, thus we are not
limited to test on central location, correlation, etc. Depending on the experimental
design, even the random sampling assumption may be relaxed [30]. Given these
characteristics, permutation tests are ideal options for testing hypotheses in ERP
studies and have received much attention in the neuroimaging community [1, 43, 76].
Permutation tests are available for classical correlation, within- and betweensubject mean difference tests, as well as for testing the main effects in ANOVA
designs [30]. However, a straightforward permutation test for interaction effects in
ANOVA designs does not exist, although some solutions have been proposed [75].
This is a major limitation if more than one independent variable is manipulated in the
experiment. Also, like other resampling methods such as bootstrap and Monte Carlo,
permutation tests require intense computations. For large data sets, permutation tests
may be time consuming, although this is rarely a concern with modern computers
and the typical size of data sets in ERP analysis.