682
L. Lyons
should claim a discovery. Thus, in deciding between the possibilities of a new
discovery or of an undetected systematic effect, our priors might favour the latter,
and hence strong evidence for discovery is required from the data. 29
However it is not necessarily equitable to use a uniform standard for large
general-purpose experiments and for small ones with a specific aim; or for looking
for a process which is expected (e.g. H 0 → μ + μ − ), as compared with a more
speculative search, such as lepton substructure[91]. But physicists and especially
journal editors seem to like a defined rule rather than a flexible criterion, so this
bolsters the 5σ standard. In any case, it is largely a semantic issue, in that physicists
finding a 4.5σ effect would clearly report it, using judiciously chosen wording to
describe the interpretation of their observation.
Statisticians also ask whether models can really be trusted to describe the extreme
tails of distributions. In general, this may be so—counting experiments are expected
to follow Poisson distributions, with small corrections for possible long timescale drifts in detector calibrations; and particle decays usually are described by
exponential distributions in time. However, the situation is much less clear for
nuisance parameters, where uncertainty estimates may be less rigorous, and their
distribution is often assumed to be Gaussian (or truncated Gaussian) by default. The
effect of these uncertainties on very small p-values needs to be investigated caseby-case.
It is important to remember that p-values merely test the null hypothesis. There
are more sensitive ways of looking for new physics when a specific alternative is
relevant. Thus a very small p-value on its own is usually not enough to make a
convincing case for discovery.
15.7.8 Repetitions in Time
Often experiments accumulate data over several years. The same search for a new
effect may typically be repeated once or twice each year as more data are collected.
Does this constitute another factor of ∼20 in the number of opportunities for a
statistical fluctuation to appear? Our reply is “No”. If there had been a 6σ signal
with the early data (which resulted in a claim for discovery), which had then become
only 3σ with more data, this would be grounds for downplaying the earlier discovery
claim. Thus at any time, there is essentially only one set of data (everything) that is
relevant.
For a p-value to be meaningful, it is important that the time at which the
experiment stops collecting data is determined not by the significance of the
observed signal but by external factors (e.g. accelerator being decommissioned,
ending of funding, etc.). Indeed there is a theorem that states that, provided data is
29 If I were performing an experiment to look for violations of energy conservation, I would require
more than 5σ , because my prior for energy being conserved is very large.
L. Lyons
should claim a discovery. Thus, in deciding between the possibilities of a new
discovery or of an undetected systematic effect, our priors might favour the latter,
and hence strong evidence for discovery is required from the data. 29
However it is not necessarily equitable to use a uniform standard for large
general-purpose experiments and for small ones with a specific aim; or for looking
for a process which is expected (e.g. H 0 → μ + μ − ), as compared with a more
speculative search, such as lepton substructure[91]. But physicists and especially
journal editors seem to like a defined rule rather than a flexible criterion, so this
bolsters the 5σ standard. In any case, it is largely a semantic issue, in that physicists
finding a 4.5σ effect would clearly report it, using judiciously chosen wording to
describe the interpretation of their observation.
Statisticians also ask whether models can really be trusted to describe the extreme
tails of distributions. In general, this may be so—counting experiments are expected
to follow Poisson distributions, with small corrections for possible long timescale drifts in detector calibrations; and particle decays usually are described by
exponential distributions in time. However, the situation is much less clear for
nuisance parameters, where uncertainty estimates may be less rigorous, and their
distribution is often assumed to be Gaussian (or truncated Gaussian) by default. The
effect of these uncertainties on very small p-values needs to be investigated caseby-case.
It is important to remember that p-values merely test the null hypothesis. There
are more sensitive ways of looking for new physics when a specific alternative is
relevant. Thus a very small p-value on its own is usually not enough to make a
convincing case for discovery.
15.7.8 Repetitions in Time
Often experiments accumulate data over several years. The same search for a new
effect may typically be repeated once or twice each year as more data are collected.
Does this constitute another factor of ∼20 in the number of opportunities for a
statistical fluctuation to appear? Our reply is “No”. If there had been a 6σ signal
with the early data (which resulted in a claim for discovery), which had then become
only 3σ with more data, this would be grounds for downplaying the earlier discovery
claim. Thus at any time, there is essentially only one set of data (everything) that is
relevant.
For a p-value to be meaningful, it is important that the time at which the
experiment stops collecting data is determined not by the significance of the
observed signal but by external factors (e.g. accelerator being decommissioned,
ending of funding, etc.). Indeed there is a theorem that states that, provided data is
29 If I were performing an experiment to look for violations of energy conservation, I would require
more than 5σ , because my prior for energy being conserved is very large.