15 Statistical Issues in Particle Physics
679
• p 1 is ruled out: The uncertainty on p is estimated by seeing how much it must
change from its optimum value in order to make χ 2 increase by 1. For this data,
χ 2 (p 1 ) is 25 units larger than χ 2 (p 0 ), and so, assuming that the behaviour of
χ 2 in the neighbourhood of the minimum is parabolic, p 1 is ruled out at the ∼5
standard deviation level.
Unfortunately, many physicists, over-impressed by the fact that χ 2 (p 1 ) appears
to be satisfactory, are reluctant to accept that p 0 is strongly favoured by the data.
A similar argument applies to comparing a given set of data with 2 separate
hypotheses e.g. fitting a histogram with an exponential or a straight line. Again the
difference between the χ 2 quantities provides better discrimination between the
hypotheses than do the individual χ 2 values[78]. Another example of using the
difference in χ 2 ’s is given in the next section.
There are of course other ways available for comparing two hypotheses. e.g.
likelihood ratio, Bayes factor, Bayesian information criterion, etc. For a fuller
discussion, see ref. [79]. A description of their application in cosmology can be
found in ref. [80]. Problems in choosing priors for the Bayes factor approach for
selecting among hypotheses are discussed by Heinrich[81].
15.7.5 Peak Above Smooth Background
When comparing two hypotheses with our data, we can use the numerical values of
the two χ 2 quantities with a view to making some decision about the hypotheses.
For example, we may be fitting a smooth distribution by a power series, and wonder
whether we need a quadratic term, or whether a linear expression would suffice.
Alternatively we may want to assess whether a mass spectrum favours the existence
of a peak on top of a smooth background, as compared with just the smooth
background. Qualitatively, if the extra term(s) are unnecessary, they will result in
a relatively small reduction in χ 2 , while if they really are required, the reduction
could be larger.
It is sometimes possible to be quantitative about the expected reduction when
the extra terms are not needed[82]. If we are in the asymptotic regime, and if the
hypotheses are nested, 25 and if the extra parameters of the larger hypothesis are
defined under the smaller one, and in that case do not lie on the boundary of their
allowed region, then the difference in χ 2 should itself be distributed as a χ 2 , with
the number of degrees of freedom equal to the number of extra parameters.
An example that satisfies this is provided by the different order polynomials. The
hypotheses are nested, in that the linear situation is a special case of a quadratic,
where the coefficient of the quadratic term is zero. Thus the extra parameter is
defined and within the (infinite) allowed range. Then, provided we have a large
25 This means that for suitable values of the parameters the larger hypothesis reduces to the smaller
one.
679
• p 1 is ruled out: The uncertainty on p is estimated by seeing how much it must
change from its optimum value in order to make χ 2 increase by 1. For this data,
χ 2 (p 1 ) is 25 units larger than χ 2 (p 0 ), and so, assuming that the behaviour of
χ 2 in the neighbourhood of the minimum is parabolic, p 1 is ruled out at the ∼5
standard deviation level.
Unfortunately, many physicists, over-impressed by the fact that χ 2 (p 1 ) appears
to be satisfactory, are reluctant to accept that p 0 is strongly favoured by the data.
A similar argument applies to comparing a given set of data with 2 separate
hypotheses e.g. fitting a histogram with an exponential or a straight line. Again the
difference between the χ 2 quantities provides better discrimination between the
hypotheses than do the individual χ 2 values[78]. Another example of using the
difference in χ 2 ’s is given in the next section.
There are of course other ways available for comparing two hypotheses. e.g.
likelihood ratio, Bayes factor, Bayesian information criterion, etc. For a fuller
discussion, see ref. [79]. A description of their application in cosmology can be
found in ref. [80]. Problems in choosing priors for the Bayes factor approach for
selecting among hypotheses are discussed by Heinrich[81].
15.7.5 Peak Above Smooth Background
When comparing two hypotheses with our data, we can use the numerical values of
the two χ 2 quantities with a view to making some decision about the hypotheses.
For example, we may be fitting a smooth distribution by a power series, and wonder
whether we need a quadratic term, or whether a linear expression would suffice.
Alternatively we may want to assess whether a mass spectrum favours the existence
of a peak on top of a smooth background, as compared with just the smooth
background. Qualitatively, if the extra term(s) are unnecessary, they will result in
a relatively small reduction in χ 2 , while if they really are required, the reduction
could be larger.
It is sometimes possible to be quantitative about the expected reduction when
the extra terms are not needed[82]. If we are in the asymptotic regime, and if the
hypotheses are nested, 25 and if the extra parameters of the larger hypothesis are
defined under the smaller one, and in that case do not lie on the boundary of their
allowed region, then the difference in χ 2 should itself be distributed as a χ 2 , with
the number of degrees of freedom equal to the number of extra parameters.
An example that satisfies this is provided by the different order polynomials. The
hypotheses are nested, in that the linear situation is a special case of a quadratic,
where the coefficient of the quadratic term is zero. Thus the extra parameter is
defined and within the (infinite) allowed range. Then, provided we have a large
25 This means that for suitable values of the parameters the larger hypothesis reduces to the smaller
one.