15 Statistical Issues in Particle Physics
685
• The signal region is defined by a certain part of multi-dimensional space, and this
is kept hidden, but all other regions, including those adjacent to the signal, are
available for inspection.
• Keep the Monte Carlo parameters hidden. This is a technique suggested by the
TWIST experiment in their high statistics precision determination of parameters
associated with muon decay. The procedure involves comparing the data with
various simulated sets, generated with a series of different parameter values.
The data and the simulations are both visible, but the parameter values used to
generate the simulations are kept hidden.
• Keep visible only a fraction of the contents of each bin of a histogram. This is
used by the MINOS experiment searching for neutrino oscillations; these would
affect the energy distribution of the observed events. By keeping visible different
unknown fractions of the data in each bin, the energy spectral shape cannot be
determined from the visible part of the data.
If several different groups within the same collaboration are performing similar
analyses for extracting some specific parameter, then it is desirable to fix the
procedure for selecting which result to present, or alternatively how to combine
the separate results. This should be done before the results are seen, and is worth
doing even if the individual analyses were not “blind”.
A question that arises with blind analyses is whether it should be permitted to
modify the analysis after the data had been unblinded. It is generally agreed that
this should not be done, unless everyone would regard it as ridiculous not to do
so. For example, if a search for rare events yielded 10 candidates over the course
of a year’s run, all of which occurred on Sunday mornings at precisely 1.17 a.m.,
it would be prudent to do some further investigation before publishing. If ‘postunblinding’ modification of the procedure is performed, this should be made clear
in any publication.
15.9 Topics that Deserve More Attention
15.9.1 Statistical Software
Particle physicists tend to write their own software for performing statistical computations. Although this has educational merits, it is inefficient use of one’s time. The
data-manipulation system of programmes ROOT/RooFit/RooStats contains many
useful statistical routines[96]. Tools also exist for implementing many methods for
separating signal from background[43, 44].
A problem with these is that they are too easy to use. In the hands of a noncritical user, the required input data instructions may contain some error, with
the consequence that they will produce the solution to a different procedure than
the intended one. It is very important to check that the result obtained is not
unreasonable.
685
• The signal region is defined by a certain part of multi-dimensional space, and this
is kept hidden, but all other regions, including those adjacent to the signal, are
available for inspection.
• Keep the Monte Carlo parameters hidden. This is a technique suggested by the
TWIST experiment in their high statistics precision determination of parameters
associated with muon decay. The procedure involves comparing the data with
various simulated sets, generated with a series of different parameter values.
The data and the simulations are both visible, but the parameter values used to
generate the simulations are kept hidden.
• Keep visible only a fraction of the contents of each bin of a histogram. This is
used by the MINOS experiment searching for neutrino oscillations; these would
affect the energy distribution of the observed events. By keeping visible different
unknown fractions of the data in each bin, the energy spectral shape cannot be
determined from the visible part of the data.
If several different groups within the same collaboration are performing similar
analyses for extracting some specific parameter, then it is desirable to fix the
procedure for selecting which result to present, or alternatively how to combine
the separate results. This should be done before the results are seen, and is worth
doing even if the individual analyses were not “blind”.
A question that arises with blind analyses is whether it should be permitted to
modify the analysis after the data had been unblinded. It is generally agreed that
this should not be done, unless everyone would regard it as ridiculous not to do
so. For example, if a search for rare events yielded 10 candidates over the course
of a year’s run, all of which occurred on Sunday mornings at precisely 1.17 a.m.,
it would be prudent to do some further investigation before publishing. If ‘postunblinding’ modification of the procedure is performed, this should be made clear
in any publication.
15.9 Topics that Deserve More Attention
15.9.1 Statistical Software
Particle physicists tend to write their own software for performing statistical computations. Although this has educational merits, it is inefficient use of one’s time. The
data-manipulation system of programmes ROOT/RooFit/RooStats contains many
useful statistical routines[96]. Tools also exist for implementing many methods for
separating signal from background[43, 44].
A problem with these is that they are too easy to use. In the hands of a noncritical user, the required input data instructions may contain some error, with
the consequence that they will produce the solution to a different procedure than
the intended one. It is very important to check that the result obtained is not
unreasonable.