15 Statistical Issues in Particle Physics
681
region. Clearly if it is desired to establish a discovery from p-values around 10 −7
or smaller, then γ should be chosen at least an order of magnitude below this.
The properties of these and other methods are compared by Demortier [84], while
Cranmer [88] and Cousins et al.[89] have discussed some of them in the context of
searches at the LHC.
The role of systematic effects is likely to be more serious here than for upper
limits discussed in Sect. 15.4.5. This is because in upper limit situations the number
of events is usually small, and so statistical uncertainties dominate. In contrast,
discovery claims have p-values of 3 ∗ 10 −7 or smaller, and so tails of distributions
are likely to be important.
15.7.7 Why 5σ ?
Unfortunately the usually accepted criterion for claiming a discovery in Particle
Physics is that p should correspond to at least 5σ . Statisticians almost invariably
ask why such a stringent level is used. One answer is past experience: all too often
interesting effects at the 3σ or 4σ level have gone away as more data are collected.
Another is the multiple comparison problem, or “Look Elsewhere Effect” (LEE).
While the chance of obtaining a 5σ effect in one bin of a particular histogram
(“local p-value”) is really small, it is to be remembered that histograms have many
bins, 26 they could be plotted with different selection criteria and different binning, 27
and there are very many other histograms that were or could have been looked at
in the course of the experiment. 28 Thus the chance of a 5σ fluctuation occurring
somewhere in the data (“global p-value”) is much larger than might at first appear.
Calculating a global p-value may require an excessive amount of Monte-Carlo
simulation. Reference [90] circumvents this for asymptotic situations by providing a
formula for extrapolating the LEE correction factor from a lower significance level;
this requires considerably less simulation.
Finally, physicists subconsciously incorporate Bayesian priors in assessing how
likely they feel that they have discovered something new, and hence whether they
26 In calculating a p-value in such a case, it is very desirable to take into account the number of
chances for a statistical fluctuation to occur anywhere in the histogram (or anywhere in the search
procedure, for more complicated analyses). At very least, it should be made clear what the basis of
the calculated p-value is.
27 If a blind analysis is performed, such decisions are made before looking at the data, and so this
aspect of the “look elsewhere” effect is reduced.
28 The extent to which other people’s searches should be included in an allowance for the “look
elsewhere” effect depends on the implied question being addressed. Thus are we considering the
chance of obtaining a statistical fluctuation in any of the analyses we have performed; or by
anyone analysing data in our experiment; or by any Particle Physicist this year? Because of the
ambiguity of which specific question is being addressed, which is often not explicitly mentioned,
we recommend not including an extra “look elsewhere” factor for this.
681
region. Clearly if it is desired to establish a discovery from p-values around 10 −7
or smaller, then γ should be chosen at least an order of magnitude below this.
The properties of these and other methods are compared by Demortier [84], while
Cranmer [88] and Cousins et al.[89] have discussed some of them in the context of
searches at the LHC.
The role of systematic effects is likely to be more serious here than for upper
limits discussed in Sect. 15.4.5. This is because in upper limit situations the number
of events is usually small, and so statistical uncertainties dominate. In contrast,
discovery claims have p-values of 3 ∗ 10 −7 or smaller, and so tails of distributions
are likely to be important.
15.7.7 Why 5σ ?
Unfortunately the usually accepted criterion for claiming a discovery in Particle
Physics is that p should correspond to at least 5σ . Statisticians almost invariably
ask why such a stringent level is used. One answer is past experience: all too often
interesting effects at the 3σ or 4σ level have gone away as more data are collected.
Another is the multiple comparison problem, or “Look Elsewhere Effect” (LEE).
While the chance of obtaining a 5σ effect in one bin of a particular histogram
(“local p-value”) is really small, it is to be remembered that histograms have many
bins, 26 they could be plotted with different selection criteria and different binning, 27
and there are very many other histograms that were or could have been looked at
in the course of the experiment. 28 Thus the chance of a 5σ fluctuation occurring
somewhere in the data (“global p-value”) is much larger than might at first appear.
Calculating a global p-value may require an excessive amount of Monte-Carlo
simulation. Reference [90] circumvents this for asymptotic situations by providing a
formula for extrapolating the LEE correction factor from a lower significance level;
this requires considerably less simulation.
Finally, physicists subconsciously incorporate Bayesian priors in assessing how
likely they feel that they have discovered something new, and hence whether they
26 In calculating a p-value in such a case, it is very desirable to take into account the number of
chances for a statistical fluctuation to occur anywhere in the histogram (or anywhere in the search
procedure, for more complicated analyses). At very least, it should be made clear what the basis of
the calculated p-value is.
27 If a blind analysis is performed, such decisions are made before looking at the data, and so this
aspect of the “look elsewhere” effect is reduced.
28 The extent to which other people’s searches should be included in an allowance for the “look
elsewhere” effect depends on the implied question being addressed. Thus are we considering the
chance of obtaining a statistical fluctuation in any of the analyses we have performed; or by
anyone analysing data in our experiment; or by any Particle Physicist this year? Because of the
ambiguity of which specific question is being addressed, which is often not explicitly mentioned,
we recommend not including an extra “look elsewhere” factor for this.