670
L. Lyons
At the second Banff meeting[8], the challenge was set by Tom Junk and consisted
of participants trying to distinguish between histograms, some of which contained
only background and others which contained a background and signal, which
appeared as a peak (compare Sect. 15.7.5)
15.4.7 Recommendations
It would be incorrect to say that there is one method that must be used. Many Particle
Physicists’ ideal would be to use a frequentist approach if viable software were
available for problems with several parameters and items of data. Otherwise they
would be prepared to settle for a Bayesian approach, with studies of the sensitivity of
the upper limit to the choice of priors, and of the coverage; or for a profile likelihood
method, again with coverage studies. What is important is that the procedure should
be fully defined before the data are analysed; and that when the experimental result
and the sensitivity of the search are reported, the method used should be fully
explained.
The CDF Statistics Committee [67] also suggests that it is useful to use
a technique that has been employed by other experiments studying the same
phenomenon; this makes for easier comparison. They tend to favour a Bayesian
approach, chiefly because of the ease of incorporating nuisance parameters.
15.5 Combining Results
This section deals with the combination of the results from two or more measurements of a single (or several) parameters of interest. It is not possible to
combine upper limits (UL). This is because an 84% UL of 1.5 could come from
a measurement of 1.4 ± 0.1, or 0.5 ± 1.0; these would give very different results
when combined with some other measurement.
The combination of p-values is discussed in Sect. 15.7.9.
15.5.1 Single Parameter
An interesting question is whether it is possible to combine two measurements
of a single quantity, each with uncertainty ±10, such that the uncertainty on the
combined best estimate is ±1? The answer can be deduced later.
L. Lyons
At the second Banff meeting[8], the challenge was set by Tom Junk and consisted
of participants trying to distinguish between histograms, some of which contained
only background and others which contained a background and signal, which
appeared as a peak (compare Sect. 15.7.5)
15.4.7 Recommendations
It would be incorrect to say that there is one method that must be used. Many Particle
Physicists’ ideal would be to use a frequentist approach if viable software were
available for problems with several parameters and items of data. Otherwise they
would be prepared to settle for a Bayesian approach, with studies of the sensitivity of
the upper limit to the choice of priors, and of the coverage; or for a profile likelihood
method, again with coverage studies. What is important is that the procedure should
be fully defined before the data are analysed; and that when the experimental result
and the sensitivity of the search are reported, the method used should be fully
explained.
The CDF Statistics Committee [67] also suggests that it is useful to use
a technique that has been employed by other experiments studying the same
phenomenon; this makes for easier comparison. They tend to favour a Bayesian
approach, chiefly because of the ease of incorporating nuisance parameters.
15.5 Combining Results
This section deals with the combination of the results from two or more measurements of a single (or several) parameters of interest. It is not possible to
combine upper limits (UL). This is because an 84% UL of 1.5 could come from
a measurement of 1.4 ± 0.1, or 0.5 ± 1.0; these would give very different results
when combined with some other measurement.
The combination of p-values is discussed in Sect. 15.7.9.
15.5.1 Single Parameter
An interesting question is whether it is possible to combine two measurements
of a single quantity, each with uncertainty ±10, such that the uncertainty on the
combined best estimate is ±1? The answer can be deduced later.