15 Statistical Issues in Particle Physics
669
In practice there are formidable difficulties in writing a program to do this in
a reasonable amount of time. Another problem is that, unless a clever ordering
rule is used for producing the acceptance region in data space for fixed values of
the parameters, the projection phase leads to overcoverage, which can become
larger as the number of nuisance parameters increases. Good ordering rules have
been found for a version of the Poisson counting experiment[63], and also for
the ratio of Poisson means[64], where the confidence intervals are tighter than
those obtained by conditioning on the sum of the numbers of counts in the two
observations.
For the fully frequentist method, it is guaranteed that there will be no undercoverage for any combination of parameter true values. This is not so for any
other method, and so most particle physicists would like assurance that the
technique used does indeed provide reasonable coverage, at least for s. There is
usually lively debate between frequentists and Bayesians as to whether coverage
is desirable for all values of the nuisance parameter(s), or whether one should
be happy with no or little undercoverage when experiments are averaged, for
example, over the nuisance parameter true values.
• Mixed
Because of the difficulty of performing a fully frequentist analysis in all but the
simplest problems, an alternative approach[65] is to use Bayesian averaging over
the nuisance parameters, but then to employ a frequentist approach for s. The
hope is that for most experiments setting upper limits, the statistical uncertainties
on the low n data are relatively large and so, provided the uncertainties in the
nuisance parameters are not too large, the effect of the systematics on the upper
limits will not be too dramatic, and an approximate method of dealing with them
may be reasonable.
Although such an approach cannot be justified from fundamentals, it provides a
practical method whose properties can be checked, and is often satisfactory.
15.4.6 Banff Challenges
Given the large number of techniques available for extracting upper limits from
data, especially in the presence of nuisance parameters, it was decided at the
Banff meeting[6] that it would be useful to compare the properties of the different
approaches under comparable conditions. This led to the setting up of the ‘Banff
Challenge’, which consisted of providing common data sets for anyone to calculate
their upper limits. This was organised by Joel Heinrich, who reported on the
performance of the various methods at the PHYSTAT-LHC meeting[66].
669
In practice there are formidable difficulties in writing a program to do this in
a reasonable amount of time. Another problem is that, unless a clever ordering
rule is used for producing the acceptance region in data space for fixed values of
the parameters, the projection phase leads to overcoverage, which can become
larger as the number of nuisance parameters increases. Good ordering rules have
been found for a version of the Poisson counting experiment[63], and also for
the ratio of Poisson means[64], where the confidence intervals are tighter than
those obtained by conditioning on the sum of the numbers of counts in the two
observations.
For the fully frequentist method, it is guaranteed that there will be no undercoverage for any combination of parameter true values. This is not so for any
other method, and so most particle physicists would like assurance that the
technique used does indeed provide reasonable coverage, at least for s. There is
usually lively debate between frequentists and Bayesians as to whether coverage
is desirable for all values of the nuisance parameter(s), or whether one should
be happy with no or little undercoverage when experiments are averaged, for
example, over the nuisance parameter true values.
• Mixed
Because of the difficulty of performing a fully frequentist analysis in all but the
simplest problems, an alternative approach[65] is to use Bayesian averaging over
the nuisance parameters, but then to employ a frequentist approach for s. The
hope is that for most experiments setting upper limits, the statistical uncertainties
on the low n data are relatively large and so, provided the uncertainties in the
nuisance parameters are not too large, the effect of the systematics on the upper
limits will not be too dramatic, and an approximate method of dealing with them
may be reasonable.
Although such an approach cannot be justified from fundamentals, it provides a
practical method whose properties can be checked, and is often satisfactory.
15.4.6 Banff Challenges
Given the large number of techniques available for extracting upper limits from
data, especially in the presence of nuisance parameters, it was decided at the
Banff meeting[6] that it would be useful to compare the properties of the different
approaches under comparable conditions. This led to the setting up of the ‘Banff
Challenge’, which consisted of providing common data sets for anyone to calculate
their upper limits. This was organised by Joel Heinrich, who reported on the
performance of the various methods at the PHYSTAT-LHC meeting[66].