15 Statistical Issues in Particle Physics
647
uncertainty might well be that due to the limited number of neutrino interactions
observed in the detector. On the other hand, there are likely to be systematics from
limited knowledge of neutrino cross-sections in the detector material, the energy
calibration of the detector, neutrino oscillation parameters, models of energy convection in the sun, etc. If some calibration measurement or subsidiary experiment
can be performed, this effectively converts a systematic uncertainty into a statistical
one. Whether this source of uncertainty is quoted as statistical or systematic is
not crucial; what is important is that possible sources of correlation between
uncertainties here and in other measurements (in this or in other experiments) are
well understood.
The magnitude of systematic effects in a parameter-determination situation can
be assessed by fitting the data with different values of the nuisance parameter(s),
and seeing how much the result changes 1 when the nuisance parameter value is
varied by its uncertainty. Alternatively the nuisance parameter(s) for systematic
effects can be incorporated into the likelihood or χ 2 for the fit; or a Bayesian method
involving the prior probability distribution for the nuisance parameter can be used.
(See Sects. 15.4.5 and 15.7.6 for ways of incorporating nuisance parameters in upper
limit and in p-value calculations respectively).
How to assess systematics was much discussed at the first Banff meeting[6] and
at PHYSTAT-LHC[24–26]. A special session of the recent PHYSTATν meeting at
CERN[12] was devoted to systematics. Many reviews of this complex subject exist
and can be traced back via ref. [27].
In general, much more effort is involved in estimating systematic uncertainties
than for parameter determination and the corresponding statistical uncertainties; this
is especially the case when the systematics dominate the statistical uncertainty.
Cowan[35] has considered the effect of having an uncertainty in magnitude of a
systematic effect. As Cox has remarked[36], there is a difference in knowing that a
correction has almost precisely a 20% uncertainty, or that it is somewhere between
0% and 40%.
15.1.3 Bayes and Frequentism
These are two fundamental approaches to making inferences about parameters or
whether data support particular hypotheses. There are also other methods which do
not correspond to either of these philosophies; the use of χ 2 or the likelihood are
examples.
Particle physicists tend to favour a frequentist method. This is because in many
cases we really believe that our data are representative as samples drawn according
to the model we are using (decay time distributions often are exponential; the counts
1 If the simulation yields a change in the result of a ± b, there is much discussion about how the
contribution to the systematic uncertainty should be assessed in terms of a and b—see ref. [27].
647
uncertainty might well be that due to the limited number of neutrino interactions
observed in the detector. On the other hand, there are likely to be systematics from
limited knowledge of neutrino cross-sections in the detector material, the energy
calibration of the detector, neutrino oscillation parameters, models of energy convection in the sun, etc. If some calibration measurement or subsidiary experiment
can be performed, this effectively converts a systematic uncertainty into a statistical
one. Whether this source of uncertainty is quoted as statistical or systematic is
not crucial; what is important is that possible sources of correlation between
uncertainties here and in other measurements (in this or in other experiments) are
well understood.
The magnitude of systematic effects in a parameter-determination situation can
be assessed by fitting the data with different values of the nuisance parameter(s),
and seeing how much the result changes 1 when the nuisance parameter value is
varied by its uncertainty. Alternatively the nuisance parameter(s) for systematic
effects can be incorporated into the likelihood or χ 2 for the fit; or a Bayesian method
involving the prior probability distribution for the nuisance parameter can be used.
(See Sects. 15.4.5 and 15.7.6 for ways of incorporating nuisance parameters in upper
limit and in p-value calculations respectively).
How to assess systematics was much discussed at the first Banff meeting[6] and
at PHYSTAT-LHC[24–26]. A special session of the recent PHYSTATν meeting at
CERN[12] was devoted to systematics. Many reviews of this complex subject exist
and can be traced back via ref. [27].
In general, much more effort is involved in estimating systematic uncertainties
than for parameter determination and the corresponding statistical uncertainties; this
is especially the case when the systematics dominate the statistical uncertainty.
Cowan[35] has considered the effect of having an uncertainty in magnitude of a
systematic effect. As Cox has remarked[36], there is a difference in knowing that a
correction has almost precisely a 20% uncertainty, or that it is somewhere between
0% and 40%.
15.1.3 Bayes and Frequentism
These are two fundamental approaches to making inferences about parameters or
whether data support particular hypotheses. There are also other methods which do
not correspond to either of these philosophies; the use of χ 2 or the likelihood are
examples.
Particle physicists tend to favour a frequentist method. This is because in many
cases we really believe that our data are representative as samples drawn according
to the model we are using (decay time distributions often are exponential; the counts
1 If the simulation yields a change in the result of a ± b, there is much discussion about how the
contribution to the systematic uncertainty should be assessed in terms of a and b—see ref. [27].