15 Statistical Issues in Particle Physics
677
The conventional method of claiming new particle production would be if the
observed t fell well above the main peak of the H 0 distribution; typically a p 0 value
corresponding to 5σ would be required (see Sect. 15.7.7). In a similar way, new
particle production would be excluded if t were below the main part of the H 1
distribution. Typically a 95% exclusion region would be chosen (i.e. p 1 ≤ 0.05),
where p 1 is by convention the left-hand tail of the H 1 distribution, as shown in
Fig. 15.5b.
The CL s method aims to provide protection against a downward fluctuation of
t in Fig. 15.5a resulting in a claim of exclusion in a situation where the experiment
has no sensitivity to the production of the new particle; this could happen in 5% of
experiments. It achieves this by defining 24
CL s = p 1 /(1 − p 0 ),
(15.17)
and requiring CL s to be below 0.05. From its definition, it is clear that CL s cannot
be smaller than p 1 , and hence is a conservative version of the frequentist quantity
p 1 . It tends to p 1 when t lies above the H 0 distribution, and to unity when the H 0
and H 1 distributions are very similar. The reduced CL s exclusion region is shown by
the dotted diagonal line in Fig. 15.7; the price to pay for the protection provided by
CL s is that there is built-in conservatism when p 1 is small but p 0 has intermediate
values i.e. there are more cases in which no decision is made. Most statisticians are
appalled by the use of CL s , because they consider that it is meaningless to take the
ratio of two p-values.
It is deemed not to be necessary to protect against statistical fluctuations giving
rise to discovery claims in situations with no sensitivity, because that should happen
only at the 3 ∗ 10 −7 rate (the one-sided 5σ Gaussian tail area).
Figure 15.7 is also useful for understanding the Punzi sensitivity definition (see
Sect. 15.4.4). For any specified distributions of the statistic t for H 0 and H 1 , the
possible (p 0 , p 1 ) values lie on a curve or straight line which extends from (0,1) to
(1,0). With more data, the t distributions separate, and the curve moves closer to the
p 0 and p 1 axes. The amount of data required to satisfy the Punzi requirement of
always claiming a discovery or an exclusion is when no part of the curve is in the
“no decision” region of Fig. 15.7.
24 Given the fact that CL s is the ratio of two p-values, the choice of symbol CL s (standing for
‘confidence level of signal’) is not optimal. Another source of confusion is that in definitions of
CL s the ways the p-values are defined vary, so the formulae can look different but the underlying
concept is the same.
A subtlety with Eq. (15.17) is that p 0 there is the probability of obtaining a measurement
greater than the observed one, rather than the usual ‘greater than or equal to’. This is to make
1 − p 0 the probability of a value smaller than or equal to the observed one, in analogy with the
definition of p 1 . It makes a difference when the observation is a small discrete number.
677
The conventional method of claiming new particle production would be if the
observed t fell well above the main peak of the H 0 distribution; typically a p 0 value
corresponding to 5σ would be required (see Sect. 15.7.7). In a similar way, new
particle production would be excluded if t were below the main part of the H 1
distribution. Typically a 95% exclusion region would be chosen (i.e. p 1 ≤ 0.05),
where p 1 is by convention the left-hand tail of the H 1 distribution, as shown in
Fig. 15.5b.
The CL s method aims to provide protection against a downward fluctuation of
t in Fig. 15.5a resulting in a claim of exclusion in a situation where the experiment
has no sensitivity to the production of the new particle; this could happen in 5% of
experiments. It achieves this by defining 24
CL s = p 1 /(1 − p 0 ),
(15.17)
and requiring CL s to be below 0.05. From its definition, it is clear that CL s cannot
be smaller than p 1 , and hence is a conservative version of the frequentist quantity
p 1 . It tends to p 1 when t lies above the H 0 distribution, and to unity when the H 0
and H 1 distributions are very similar. The reduced CL s exclusion region is shown by
the dotted diagonal line in Fig. 15.7; the price to pay for the protection provided by
CL s is that there is built-in conservatism when p 1 is small but p 0 has intermediate
values i.e. there are more cases in which no decision is made. Most statisticians are
appalled by the use of CL s , because they consider that it is meaningless to take the
ratio of two p-values.
It is deemed not to be necessary to protect against statistical fluctuations giving
rise to discovery claims in situations with no sensitivity, because that should happen
only at the 3 ∗ 10 −7 rate (the one-sided 5σ Gaussian tail area).
Figure 15.7 is also useful for understanding the Punzi sensitivity definition (see
Sect. 15.4.4). For any specified distributions of the statistic t for H 0 and H 1 , the
possible (p 0 , p 1 ) values lie on a curve or straight line which extends from (0,1) to
(1,0). With more data, the t distributions separate, and the curve moves closer to the
p 0 and p 1 axes. The amount of data required to satisfy the Punzi requirement of
always claiming a discovery or an exclusion is when no part of the curve is in the
“no decision” region of Fig. 15.7.
24 Given the fact that CL s is the ratio of two p-values, the choice of symbol CL s (standing for
‘confidence level of signal’) is not optimal. Another source of confusion is that in definitions of
CL s the ways the p-values are defined vary, so the formulae can look different but the underlying
concept is the same.
A subtlety with Eq. (15.17) is that p 0 there is the probability of obtaining a measurement
greater than the observed one, rather than the usual ‘greater than or equal to’. This is to make
1 − p 0 the probability of a value smaller than or equal to the observed one, in analogy with the
definition of p 1 . It makes a difference when the observation is a small discrete number.