680
L. Lyons
amount of data, we expect the difference in χ 2 to have one degree of freedom, so a
value larger than around 5 would be unlikely.
A contrast is provided by a smooth background C(x) compared with a background plus peak, C(x) + A exp [−0.5 ∗ (x − x 0 ) 2 /σ 2 ]. The extra parameters for
the peak are its amplitude, position and width: A, x 0 and σ respectively. Again the
hypotheses are nested, in that C(x) is just a special case of the peak plus background,
with A = 0. However, although A is defined in the background only case, x 0 and
σ are not, as their values become completely irrelevant when A = 0. Furthermore,
unless the peak plus background fit allows A to be negative, zero is on the boundary
of its allowed region. We thus should not expect the difference of the χ 2 quantities
itself to be distributed as a χ 2 [83–85]. To assess the significance of a particular χ 2
difference, this unfortunately means that we have to obtain its distribution ourselves,
presumably by Monte Carlo. If we want to find out probabilities of statistical
fluctuations at the 10 −6 level, this requires a lot of simulation, and probably needs
us to use something better than brute force.
The problem of non-standard limiting distributions for χ 2 tests has a substantial
statistical literature (see, for example, refs. [86] and [87].)
15.7.6 Incorporating Nuisance Parameters
The calculation of p-values is complicated in practice by the existence of nuisance
parameters. (For the simple situation described in Sect. 15.7.2, there could be
some uncertainty in the estimated background.) There are numerous ways of
incorporating them. These include:
• Conditioning: For example, with a single nuisance parameter, it may be possible
to condition on the sum of the number of counts in the main and the subsidiary
experiments, and then to use the binomial distribution to obtain the p-value.
• Plug-in p-value: The best estimate of the nuisance parameter under the null
hypothesis is used to calculate p.
• Prior predictive p-value: The p-values are averaged over the nuisance parameters, weighted by their prior distributions. This is in the spirit of the Cousins and
Highland approach[65] for upper limits.
• Posterior predictive p-value: This time, the posterior distributions of the nuisance
parameters are used for weighting.
• Supremum p-value: The largest p-value for any possible value of the nuisance
parameter is used. This is likely to be useful only when the nuisance parameter is
forced to be within some range; or when there is only a small number of possible
alternative theoretical interpretations.
• Confidence interval: A region of frequentist confidence 1 − γ is used for the
nuisance parameter(s), and then the adjusted p-value is p max + γ , where p max
is the largest p-value as the nuisance parameters are varied over their confidence
L. Lyons
amount of data, we expect the difference in χ 2 to have one degree of freedom, so a
value larger than around 5 would be unlikely.
A contrast is provided by a smooth background C(x) compared with a background plus peak, C(x) + A exp [−0.5 ∗ (x − x 0 ) 2 /σ 2 ]. The extra parameters for
the peak are its amplitude, position and width: A, x 0 and σ respectively. Again the
hypotheses are nested, in that C(x) is just a special case of the peak plus background,
with A = 0. However, although A is defined in the background only case, x 0 and
σ are not, as their values become completely irrelevant when A = 0. Furthermore,
unless the peak plus background fit allows A to be negative, zero is on the boundary
of its allowed region. We thus should not expect the difference of the χ 2 quantities
itself to be distributed as a χ 2 [83–85]. To assess the significance of a particular χ 2
difference, this unfortunately means that we have to obtain its distribution ourselves,
presumably by Monte Carlo. If we want to find out probabilities of statistical
fluctuations at the 10 −6 level, this requires a lot of simulation, and probably needs
us to use something better than brute force.
The problem of non-standard limiting distributions for χ 2 tests has a substantial
statistical literature (see, for example, refs. [86] and [87].)
15.7.6 Incorporating Nuisance Parameters
The calculation of p-values is complicated in practice by the existence of nuisance
parameters. (For the simple situation described in Sect. 15.7.2, there could be
some uncertainty in the estimated background.) There are numerous ways of
incorporating them. These include:
• Conditioning: For example, with a single nuisance parameter, it may be possible
to condition on the sum of the number of counts in the main and the subsidiary
experiments, and then to use the binomial distribution to obtain the p-value.
• Plug-in p-value: The best estimate of the nuisance parameter under the null
hypothesis is used to calculate p.
• Prior predictive p-value: The p-values are averaged over the nuisance parameters, weighted by their prior distributions. This is in the spirit of the Cousins and
Highland approach[65] for upper limits.
• Posterior predictive p-value: This time, the posterior distributions of the nuisance
parameters are used for weighting.
• Supremum p-value: The largest p-value for any possible value of the nuisance
parameter is used. This is likely to be useful only when the nuisance parameter is
forced to be within some range; or when there is only a small number of possible
alternative theoretical interpretations.
• Confidence interval: A region of frequentist confidence 1 − γ is used for the
nuisance parameter(s), and then the adjusted p-value is p max + γ , where p max
is the largest p-value as the nuisance parameters are varied over their confidence