15 Statistical Issues in Particle Physics
657
15.3 Separating Signal from Background
Almost every Particle Physics analysis uses some technique for separating possible
signal from background. First some simple ‘cuts’ are applied; these are generally
loose selections on single variables, which are designed to remove a large fraction
of the background while barely reducing the real or potential signal. Then to
obtain a better separation of signal from background in the multi-dimensional
space of the event observables, methods like Fisher discriminants, decision trees,
artificial neural networks (including Bayesian nets and more recently deep neural
nets), support vector machines, etc. are used[41, 42]. Extensions of these methods
involve bagging, boosting and random forests, which have been used to achieve
improved performance of the separation as seen on a plot of signal efficiency
against background mis-acceptance rate. A description of the software available for
implementing some of these techniques can be found in the talks by Narsky[43] and
by Tegenfeldt[44] at the PHYSTAT-LHC Workshop.
More recently, deep learning techniques are rapidly becoming popular. In Particle
Physics, they have been used for on-line triggering, tracking, fast simulation, object
identification, image recognition, and event-by-event separation of signal from
background. Reference [45] provides good introductions to the use of these methods
for Particle Physics. There are now regular workshops and lectures on Machine
Learning at CERN and at Fermilab (see refs. [46] and [47]), as well as at many
universities.
The signal-to-background ratio before this multivariate stage can vary widely, as
can the signal purity after it. If some large statistics study is being performed (e.g.
to use a large sample of events to obtain an accurate measurement of the lifetime of
some particle), then it is not a disaster if there is some level of background in the
finally selected events, provided that it can be accurately assessed and allowed for in
the subsequent analysis. At the other extreme, the separation technique may be used
to see if there is any evidence for the existence of some hypothesised particle (the
potential signal), in the presence of background from well-known sources. Then the
actual data may in fact contain no observable signal.
These techniques are usually ‘taught’ to recognise signal and background by
being given examples consisting of large numbers of events of each type. These
may be produced by Monte Carlo simulation, but then there is a problem of trying
to verify that the simulation is a sufficiently accurate representation of reality. It is
better to use real data for this, but the difficulty then is to obtain sufficiently pure
samples of background and signal. Indeed, for the search for a new particle, true
data examples do not exist. However, it is the accurate representation of background
that is likely to pose a more serious problem.
The way that, for example, neural networks are trained is to present the software
with approximately equal numbers of signal and background events 10 and then
10 For searches for rare processes, it is clearly inappropriate to use the actual fractions expected
in the data to determine the ratio of signal to background Monte Carlo events to be used as the
657
15.3 Separating Signal from Background
Almost every Particle Physics analysis uses some technique for separating possible
signal from background. First some simple ‘cuts’ are applied; these are generally
loose selections on single variables, which are designed to remove a large fraction
of the background while barely reducing the real or potential signal. Then to
obtain a better separation of signal from background in the multi-dimensional
space of the event observables, methods like Fisher discriminants, decision trees,
artificial neural networks (including Bayesian nets and more recently deep neural
nets), support vector machines, etc. are used[41, 42]. Extensions of these methods
involve bagging, boosting and random forests, which have been used to achieve
improved performance of the separation as seen on a plot of signal efficiency
against background mis-acceptance rate. A description of the software available for
implementing some of these techniques can be found in the talks by Narsky[43] and
by Tegenfeldt[44] at the PHYSTAT-LHC Workshop.
More recently, deep learning techniques are rapidly becoming popular. In Particle
Physics, they have been used for on-line triggering, tracking, fast simulation, object
identification, image recognition, and event-by-event separation of signal from
background. Reference [45] provides good introductions to the use of these methods
for Particle Physics. There are now regular workshops and lectures on Machine
Learning at CERN and at Fermilab (see refs. [46] and [47]), as well as at many
universities.
The signal-to-background ratio before this multivariate stage can vary widely, as
can the signal purity after it. If some large statistics study is being performed (e.g.
to use a large sample of events to obtain an accurate measurement of the lifetime of
some particle), then it is not a disaster if there is some level of background in the
finally selected events, provided that it can be accurately assessed and allowed for in
the subsequent analysis. At the other extreme, the separation technique may be used
to see if there is any evidence for the existence of some hypothesised particle (the
potential signal), in the presence of background from well-known sources. Then the
actual data may in fact contain no observable signal.
These techniques are usually ‘taught’ to recognise signal and background by
being given examples consisting of large numbers of events of each type. These
may be produced by Monte Carlo simulation, but then there is a problem of trying
to verify that the simulation is a sufficiently accurate representation of reality. It is
better to use real data for this, but the difficulty then is to obtain sufficiently pure
samples of background and signal. Indeed, for the search for a new particle, true
data examples do not exist. However, it is the accurate representation of background
that is likely to pose a more serious problem.
The way that, for example, neural networks are trained is to present the software
with approximately equal numbers of signal and background events 10 and then
10 For searches for rare processes, it is clearly inappropriate to use the actual fractions expected
in the data to determine the ratio of signal to background Monte Carlo events to be used as the