662
L. Lyons
at constant b. A related effect is the growth of the limit as b decreases at constant
n—this can also occur in other frequentist approaches. Thus if no events are
observed (n = 0), the upper limit of a 90% Feldman-Cousins interval is 1.08
for b = 3.0, but 2.44 for b = 0. This is sometimes presented as a paradox, in
that if a bright graduate student worked hard and discovered how to eliminate the
expected background without much reduction in signal efficiency, the ‘reward’
would be a weaker upper limit. 13 An answer is that although the actual limit
had increased, the sensitivity of the experiment with the smaller background was
better. There are other situations—for example, variants of the random choice of
voltmeter (compare ref. [55])—where a measurement with better sensitivity can
on occasion give a less precise result.
• In the Bayesian approach, the dependence of the limit on b is weaker. Indeed
when n = 0, the limit does not depend on b.
• Sen et al. [56] consider a related problem, of a physical non-negative parameter
λ producing a measurement x, which is distributed about λ as a Gaussian of
variance σ 2 . As the observable x becomes more and more negative, the upper
limit on λ increases, because it is deduced that σ must in fact be larger than its
quoted value.
In trying to assess which of the methods is best, one first needs a list of desirable
properties. These include:
• Coverage: Even though coverage is a frequentist concept, most Bayesian particle
physicists would like the coverage of their intervals to match their reported
credibility, at least approximately.
Because the data in counting experiments is discrete, it is impossible in any
sensible way to achieve exact coverage for all μ (see Sect. 15.1.3.4). However, it
is not completely obvious that even Frequentists need coverage for every possible
value of μ, since different experiments will have different values of b and of
Thus even for a constant value of the physical parameter s, different experiments
will have different μ = ∗ s + b. Thus it would appear that, if coverage in some
average (over μ) sense were satisfactory, the frequentist requirement for intervals
to contain the true value at the requisite rate would be maintained. This, however,
is not the generally accepted view by particle physicists, who would like not to
undercover for any μ.
• Not too much overcoverage: Because coverage varies with μ, for methods that
aim not to undercover anywhere, some overcoverage is inevitable. This corresponds to having some upper limits which are high, and this leads to undesirable
loss of power in rejecting alternative hypotheses about the parameter’s value.
13 The n = 0 situation is perhaps a special case, as the number of observed events cannot decrease
as further selections are imposed to reduce the expected background. For non-zero observed events,
if n decreases with the tighter cuts (as expected for reduced background), the upper limit is likely
to go down, in agreement with intuition. But if n stays constant, that could be because the observed
events contain signal, so it is perhaps not surprising that the upper limit increases.
L. Lyons
at constant b. A related effect is the growth of the limit as b decreases at constant
n—this can also occur in other frequentist approaches. Thus if no events are
observed (n = 0), the upper limit of a 90% Feldman-Cousins interval is 1.08
for b = 3.0, but 2.44 for b = 0. This is sometimes presented as a paradox, in
that if a bright graduate student worked hard and discovered how to eliminate the
expected background without much reduction in signal efficiency, the ‘reward’
would be a weaker upper limit. 13 An answer is that although the actual limit
had increased, the sensitivity of the experiment with the smaller background was
better. There are other situations—for example, variants of the random choice of
voltmeter (compare ref. [55])—where a measurement with better sensitivity can
on occasion give a less precise result.
• In the Bayesian approach, the dependence of the limit on b is weaker. Indeed
when n = 0, the limit does not depend on b.
• Sen et al. [56] consider a related problem, of a physical non-negative parameter
λ producing a measurement x, which is distributed about λ as a Gaussian of
variance σ 2 . As the observable x becomes more and more negative, the upper
limit on λ increases, because it is deduced that σ must in fact be larger than its
quoted value.
In trying to assess which of the methods is best, one first needs a list of desirable
properties. These include:
• Coverage: Even though coverage is a frequentist concept, most Bayesian particle
physicists would like the coverage of their intervals to match their reported
credibility, at least approximately.
Because the data in counting experiments is discrete, it is impossible in any
sensible way to achieve exact coverage for all μ (see Sect. 15.1.3.4). However, it
is not completely obvious that even Frequentists need coverage for every possible
value of μ, since different experiments will have different values of b and of
Thus even for a constant value of the physical parameter s, different experiments
will have different μ = ∗ s + b. Thus it would appear that, if coverage in some
average (over μ) sense were satisfactory, the frequentist requirement for intervals
to contain the true value at the requisite rate would be maintained. This, however,
is not the generally accepted view by particle physicists, who would like not to
undercover for any μ.
• Not too much overcoverage: Because coverage varies with μ, for methods that
aim not to undercover anywhere, some overcoverage is inevitable. This corresponds to having some upper limits which are high, and this leads to undesirable
loss of power in rejecting alternative hypotheses about the parameter’s value.
13 The n = 0 situation is perhaps a special case, as the number of observed events cannot decrease
as further selections are imposed to reduce the expected background. For non-zero observed events,
if n decreases with the tighter cuts (as expected for reduced background), the upper limit is likely
to go down, in agreement with intuition. But if n stays constant, that could be because the observed
events contain signal, so it is perhaps not surprising that the upper limit increases.