15 Statistical Issues in Particle Physics
683
collected for long enough, it is possible to reach any arbitrary level of significance
against a hypothesis that is in fact true.
15.7.9 Combining p-Values
In looking for a given new effect, there may be several separate and uncorrelated
analyses which are relevant. These could correspond to different decay modes for
the new particle; or different experiments looking for the same signal. Thus, if the
p-values for the null hypothesis (i.e. no new physics) for the separate analyses were
10 −6 and 0.1, what is the corresponding p-value for the pair of results? 30
The unambiguous answer is that there is no unique recipe for combining them[92,
93]. There is no single way of taking a uniform distribution in two variables, and
finding a transformation p comb (p 1 , p 2 ) that converts it into a uniform distribution
of the single variable p comb .
Two popular recipes involve asking what is the probability that the smaller pvalue will be 10 −6 or smaller; or that the product is below p 1 ∗ p 2 = 10 −7 . (Note
that these probabilities are not 10 −6 and 10 −7 respectively.) None of the possible
methods has the property that in combining three p-values, the same answer is
obtained if p 1 is first combined with p 2 , and then the result is combined with p 3 ; or
whether some different ordering is used.
Another problem is the lack of other information that might be relevant. For
example, the p-values might arise from χ 2 ’s with different numbers of degrees of
freedom ν e.g. χ 2
1 = 90 for 100 degrees of freedom, and χ 2
2 = 20 for ν = 1.
The second has a very small p-value, so many combination methods (including the
two mentioned above) would conclude that overall the data do not look consistent
with the null hypothesis. However, another plausible-sounding method is to add
the separate χ 2 values and also the individual ν, 31 to obtain a total χ 2 = 110 for
ν = 101, which sounds perfectly satisfactory. The resolution of this discrepancy
of interpretation depends on the nature of the two tests. If the second analysis with
χ 2 = 20 corresponded to just one extra measurement like the previous 100, then it
seems reasonable to combine the χ 2 values and the ν, and to conclude that overall
there is indeed nothing surprising. But on the other hand, if the second measurement
was genuinely different, and an alternative way of looking for some discrepancy,
then it may be more appropriate to combine the p-values by one of the earlier
methods, which suggest that the overall consistency with theory is not good. It
30 Rather than combining p-values, it is of course much better to use the complete sets of original
data (if available) for obtaining the combined result.
31 The method described earlier involving the product of the p-values is equivalent to converting
each p to a χ 2 , assuming that ν = 2, regardless of whether this was the actual number of degrees
of freedom, and then adding the χ 2 and also the ν.
683
collected for long enough, it is possible to reach any arbitrary level of significance
against a hypothesis that is in fact true.
15.7.9 Combining p-Values
In looking for a given new effect, there may be several separate and uncorrelated
analyses which are relevant. These could correspond to different decay modes for
the new particle; or different experiments looking for the same signal. Thus, if the
p-values for the null hypothesis (i.e. no new physics) for the separate analyses were
10 −6 and 0.1, what is the corresponding p-value for the pair of results? 30
The unambiguous answer is that there is no unique recipe for combining them[92,
93]. There is no single way of taking a uniform distribution in two variables, and
finding a transformation p comb (p 1 , p 2 ) that converts it into a uniform distribution
of the single variable p comb .
Two popular recipes involve asking what is the probability that the smaller pvalue will be 10 −6 or smaller; or that the product is below p 1 ∗ p 2 = 10 −7 . (Note
that these probabilities are not 10 −6 and 10 −7 respectively.) None of the possible
methods has the property that in combining three p-values, the same answer is
obtained if p 1 is first combined with p 2 , and then the result is combined with p 3 ; or
whether some different ordering is used.
Another problem is the lack of other information that might be relevant. For
example, the p-values might arise from χ 2 ’s with different numbers of degrees of
freedom ν e.g. χ 2
1 = 90 for 100 degrees of freedom, and χ 2
2 = 20 for ν = 1.
The second has a very small p-value, so many combination methods (including the
two mentioned above) would conclude that overall the data do not look consistent
with the null hypothesis. However, another plausible-sounding method is to add
the separate χ 2 values and also the individual ν, 31 to obtain a total χ 2 = 110 for
ν = 101, which sounds perfectly satisfactory. The resolution of this discrepancy
of interpretation depends on the nature of the two tests. If the second analysis with
χ 2 = 20 corresponded to just one extra measurement like the previous 100, then it
seems reasonable to combine the χ 2 values and the ν, and to conclude that overall
there is indeed nothing surprising. But on the other hand, if the second measurement
was genuinely different, and an alternative way of looking for some discrepancy,
then it may be more appropriate to combine the p-values by one of the earlier
methods, which suggest that the overall consistency with theory is not good. It
30 Rather than combining p-values, it is of course much better to use the complete sets of original
data (if available) for obtaining the combined result.
31 The method described earlier involving the product of the p-values is equivalent to converting
each p to a χ 2 , assuming that ν = 2, regardless of whether this was the actual number of degrees
of freedom, and then adding the χ 2 and also the ν.