15 Statistical Issues in Particle Physics
665
15.4.4 Sensitivity
It is useful to quote the sensitivity of a procedure, as well as the actual upper
limit as derived from the observed data. 15 For upper limits or for uncertainties on
measurements, this can be defined as the median value that would be obtained if the
procedure was repeated a large number of times. 16 Using the median is preferable
to the mean because (a) it is metric independent (i.e. the median lifetime upper limit
would be the reciprocal of the median decay rate lower limit); and (b) it is much less
sensitive to a few anomalously large upper limits or uncertainty estimates.
It is common to present not only the median of the expected distribution, but also
values corresponding to 16th and 84th percentiles (commonly referred to as ±1 σ )
and also the 2.5% and 97.5% ones (±2σ ). This enables a check to be made that the
observed result is reasonable.
Punzi [57] has drawn attention to the fact that this choice of definition for
sensitivity has some undesirable features. Thus designing an analysis procedure to
minimise the median upper limit for a search in the absence of a signal provides a
different optimisation from maximising the median number of standard deviations
for the significance of a discovery when the signal is present. Also there is only
a 50% chance of achieving the median result or better. Instead, for pre-defined
levels α and confidence level CL, Punzi determines at what signal strength there
is a probability of at least CL for establishing a discovery at a significance level α.
This is what he quotes as the sensitivity, and is the signal strength at which we are
sure to be able either to claim a discovery or to exclude its existence. Below this,
the presence or otherwise of a signal makes too little difference, and we may remain
uncertain (see Fig. 15.5).
15.4.5 Nuisance Parameters
For calculating upper limits in the simple counting experiment described in
Sect. 15.4.1, the nuisance parameters arise from the uncertainties in the background
rate b and the acceptance . These uncertainties are usually quoted as σ b and σ
(e.g. b = 3.1 ± 0.5), and the question arises of what these uncertainties mean.
Sometimes they encapsulate the results of a subsidiary measurement, performed to
estimate b or , and then they would express the width of the Bayesian posterior
or of the frequentist interval obtained for the nuisance parameters. However, in
15 The sensitivity on its own will not do, because it is independent of the data.
16 Instead of using a large number of simulations in order to extract the median, sometimes the
‘Asimov’ data set is used. This is the single data set that would be obtained if statistical fluctuations
were suppressed. i.e. if a model predicted 11.3 events in a particular bin, the Asimov data set for
that model would contain 11.3 events in that bin. The Asimov data set and the median of the toys
usually but not always produce similar results.
665
15.4.4 Sensitivity
It is useful to quote the sensitivity of a procedure, as well as the actual upper
limit as derived from the observed data. 15 For upper limits or for uncertainties on
measurements, this can be defined as the median value that would be obtained if the
procedure was repeated a large number of times. 16 Using the median is preferable
to the mean because (a) it is metric independent (i.e. the median lifetime upper limit
would be the reciprocal of the median decay rate lower limit); and (b) it is much less
sensitive to a few anomalously large upper limits or uncertainty estimates.
It is common to present not only the median of the expected distribution, but also
values corresponding to 16th and 84th percentiles (commonly referred to as ±1 σ )
and also the 2.5% and 97.5% ones (±2σ ). This enables a check to be made that the
observed result is reasonable.
Punzi [57] has drawn attention to the fact that this choice of definition for
sensitivity has some undesirable features. Thus designing an analysis procedure to
minimise the median upper limit for a search in the absence of a signal provides a
different optimisation from maximising the median number of standard deviations
for the significance of a discovery when the signal is present. Also there is only
a 50% chance of achieving the median result or better. Instead, for pre-defined
levels α and confidence level CL, Punzi determines at what signal strength there
is a probability of at least CL for establishing a discovery at a significance level α.
This is what he quotes as the sensitivity, and is the signal strength at which we are
sure to be able either to claim a discovery or to exclude its existence. Below this,
the presence or otherwise of a signal makes too little difference, and we may remain
uncertain (see Fig. 15.5).
15.4.5 Nuisance Parameters
For calculating upper limits in the simple counting experiment described in
Sect. 15.4.1, the nuisance parameters arise from the uncertainties in the background
rate b and the acceptance . These uncertainties are usually quoted as σ b and σ
(e.g. b = 3.1 ± 0.5), and the question arises of what these uncertainties mean.
Sometimes they encapsulate the results of a subsidiary measurement, performed to
estimate b or , and then they would express the width of the Bayesian posterior
or of the frequentist interval obtained for the nuisance parameters. However, in
15 The sensitivity on its own will not do, because it is independent of the data.
16 Instead of using a large number of simulations in order to extract the median, sometimes the
‘Asimov’ data set is used. This is the single data set that would be obtained if statistical fluctuations
were suppressed. i.e. if a model predicted 11.3 events in a particular bin, the Asimov data set for
that model would contain 11.3 events in that bin. The Asimov data set and the median of the toys
usually but not always produce similar results.