684
L. Lyons
is this extra information about the nature of the two tests that determines which
combination method might be appropriate.
It is clearly important to decide in advance what combination method should be
used, without reference to the specific data being analysed.
15.8 Blind Analyses
These are becoming increasingly popular as a means of avoiding personal bias
affecting the result. They involve keeping part of the data unseen by the analysers,
until the data selection procedure and the analysis method have been completely
defined, all correction procedures specified, etc.
One of the early suggestions to use a blind analysis in a Particle Physics
experiment was due to Luis Alvarez. An experiment at Stanford had looked for
quarks, by measuring the residual charge on small spheres that were levitated in
a superconducting magnet. If a single free quark were present in a sphere, the
residual charge would be a third or two-thirds of the electron’s charge. Several of
the balls tested indeed yielded such values[94]. A potential problem was that large
corrections had to be applied to the raw data in order to extract the final result for
the charge. The suspicion was that maybe the experimenters were (subconsciously)
applying corrections until the value turned out to be ‘satisfactory’. The blind
approach involved the computer adding a random number to the raw value of the
charge, which would then be corrected until the experimentalists were satisfied, and
only then would the computer subtract the random number to reveal the final answer
for that sphere. 32
There are various methods of performing blind analyses[95] most of which aim
to allow the experimentalists to look at some of the real data, in order to perform
checks that nothing is terribly wrong. Some of these are:
• The computer adds a random number to the data, which is only subtracted after
all corrections are applied. This was the method suggested by Alvarez.
• Use only Monte Carlo to define the procedure. This completely avoids the danger
of allowing the data to determine the procedure to be used, but suffers from the
drawback that the data cannot be compared with the Monte Carlo, to check that
the latter is reasonable.
• Use only a fraction of the data for defining the procedure, which then is held
fixed for the remainder of the data. In principle, an optimisation can be employed
to determine the fraction to be kept open, but in practice this is often decided by
choosing a semi-arbitrary time after which the future data is kept blind.
32 This suggestion was implemented, but in fact no subsequent results were published. The current
consensus is that this ‘discovery’ of free quarks is probably spurious.
L. Lyons
is this extra information about the nature of the two tests that determines which
combination method might be appropriate.
It is clearly important to decide in advance what combination method should be
used, without reference to the specific data being analysed.
15.8 Blind Analyses
These are becoming increasingly popular as a means of avoiding personal bias
affecting the result. They involve keeping part of the data unseen by the analysers,
until the data selection procedure and the analysis method have been completely
defined, all correction procedures specified, etc.
One of the early suggestions to use a blind analysis in a Particle Physics
experiment was due to Luis Alvarez. An experiment at Stanford had looked for
quarks, by measuring the residual charge on small spheres that were levitated in
a superconducting magnet. If a single free quark were present in a sphere, the
residual charge would be a third or two-thirds of the electron’s charge. Several of
the balls tested indeed yielded such values[94]. A potential problem was that large
corrections had to be applied to the raw data in order to extract the final result for
the charge. The suspicion was that maybe the experimenters were (subconsciously)
applying corrections until the value turned out to be ‘satisfactory’. The blind
approach involved the computer adding a random number to the raw value of the
charge, which would then be corrected until the experimentalists were satisfied, and
only then would the computer subtract the random number to reveal the final answer
for that sphere. 32
There are various methods of performing blind analyses[95] most of which aim
to allow the experimentalists to look at some of the real data, in order to perform
checks that nothing is terribly wrong. Some of these are:
• The computer adds a random number to the data, which is only subtracted after
all corrections are applied. This was the method suggested by Alvarez.
• Use only Monte Carlo to define the procedure. This completely avoids the danger
of allowing the data to determine the procedure to be used, but suffers from the
drawback that the data cannot be compared with the Monte Carlo, to check that
the latter is reasonable.
• Use only a fraction of the data for defining the procedure, which then is held
fixed for the remainder of the data. In principle, an optimisation can be employed
to determine the fraction to be kept open, but in practice this is often decided by
choosing a semi-arbitrary time after which the future data is kept blind.
32 This suggestion was implemented, but in fact no subsequent results were published. The current
consensus is that this ‘discovery’ of free quarks is probably spurious.