15 Statistical Issues in Particle Physics
661
1887, Michelson and Morley[52] attempted to measure the speed of the Earth with
respect to the aether. No effect was seen, but the experiment was sensitive enough
to lead to the demise of the aether theory.
A simple scenario is a counting experiment where a background b is expected
from conventional sources, together with the possibility of an interesting signal
s. The number of counts n observed is expected to be Poisson distributed with a
mean μ = + b, where b is the expected number of events from background,
and is a factor for converting the basic physics parameter s into the number of
signal events expected in our particular experiment; it thus allows for experimental
inefficiency, the experiment’s running time; etc. Then given a value of n which is
comparable to the expected background, what can we say about s? The true value of
the parameter s is constrained to be non-negative. The problem is interesting enough
if b and are known exactly; it becomes more complicated when only estimates with
uncertainties σ b and σ are available.
An extension of the simple counting scenario is when a search for a new particle
is carried out over a range of masses. This is usually dealt with by performing
separate searches at a series of masses over a specified range. This ‘Raster Scan’
is in contrast with a method that regards the sought-for new particle’s mass and
its production rate as two parameters to be estimated simultaneously. The relative
merits of these two approaches are described in ref. [51].
Even without the nuisance parameters, a variety of methods is available. These
include likelihood, χ 2 , Bayesian with various priors for s, frequentist Neyman
constructions with a variety of ordering rules for n, and various ad hoc approaches.
The methods give different upper limits for the same data. 11 A comparison of several
methods can be found in ref. [53]. The largest discrepancies arise when the observed
n is less than the expected background b, presumably because of a downward
statistical fluctuation. The following different behaviours of the limit (when n < b)
can be obtained:
• Frequentist methods can give empty intervals for s i.e. there are no values of s for
which the data are likely. Particle physicists tend to be unhappy when their years
of work result in an empty interval for the parameter of interest, and it is little
consolation to hear that frequentist statisticians are satisfied with this feature, as
it does not necessarily lead to undercoverage.
When n is not quite small enough to result in an empty interval, the upper
limit might be very small. 12 This could confuse people into thinking that the
experiment was much more sensitive than it really was.
• The Feldman-Cousins frequentist method[54] (see Sect. 15.4.3) that employs a
likelihood-ratio ordering rule gives upper limits which decrease as n gets smaller
11 By coincidence, the upper limits obtained by the Bayesian approach with an (improper) flat prior
for s and by the appropriate Neyman construction agree when b = 0.
12 Bayesian methods that use priors with part of the probability density being a δ-function at s = 0
can result is a posterior with an enhanced δ-function at zero, such that the upper limit contains only
the single point s = 0.
661
1887, Michelson and Morley[52] attempted to measure the speed of the Earth with
respect to the aether. No effect was seen, but the experiment was sensitive enough
to lead to the demise of the aether theory.
A simple scenario is a counting experiment where a background b is expected
from conventional sources, together with the possibility of an interesting signal
s. The number of counts n observed is expected to be Poisson distributed with a
mean μ = + b, where b is the expected number of events from background,
and is a factor for converting the basic physics parameter s into the number of
signal events expected in our particular experiment; it thus allows for experimental
inefficiency, the experiment’s running time; etc. Then given a value of n which is
comparable to the expected background, what can we say about s? The true value of
the parameter s is constrained to be non-negative. The problem is interesting enough
if b and are known exactly; it becomes more complicated when only estimates with
uncertainties σ b and σ are available.
An extension of the simple counting scenario is when a search for a new particle
is carried out over a range of masses. This is usually dealt with by performing
separate searches at a series of masses over a specified range. This ‘Raster Scan’
is in contrast with a method that regards the sought-for new particle’s mass and
its production rate as two parameters to be estimated simultaneously. The relative
merits of these two approaches are described in ref. [51].
Even without the nuisance parameters, a variety of methods is available. These
include likelihood, χ 2 , Bayesian with various priors for s, frequentist Neyman
constructions with a variety of ordering rules for n, and various ad hoc approaches.
The methods give different upper limits for the same data. 11 A comparison of several
methods can be found in ref. [53]. The largest discrepancies arise when the observed
n is less than the expected background b, presumably because of a downward
statistical fluctuation. The following different behaviours of the limit (when n < b)
can be obtained:
• Frequentist methods can give empty intervals for s i.e. there are no values of s for
which the data are likely. Particle physicists tend to be unhappy when their years
of work result in an empty interval for the parameter of interest, and it is little
consolation to hear that frequentist statisticians are satisfied with this feature, as
it does not necessarily lead to undercoverage.
When n is not quite small enough to result in an empty interval, the upper
limit might be very small. 12 This could confuse people into thinking that the
experiment was much more sensitive than it really was.
• The Feldman-Cousins frequentist method[54] (see Sect. 15.4.3) that employs a
likelihood-ratio ordering rule gives upper limits which decrease as n gets smaller
11 By coincidence, the upper limits obtained by the Bayesian approach with an (improper) flat prior
for s and by the appropriate Neyman construction agree when b = 0.
12 Bayesian methods that use priors with part of the probability density being a δ-function at s = 0
can result is a posterior with an enhanced δ-function at zero, such that the upper limit contains only
the single point s = 0.