14 Distributed Computing
627
The output of the reconstruction arises from the application of the techniques
described in the contribution “Patter Recognition and Reconstruction” and consists
of data structures containing tracks, clusters, etc. These data structures are written as
event records, which may or may not have a one-to-one correspondence to the event
records read on input. For example, the reconstruction program may disentangle
data from different collisions contained in a single raw data event record and write
the corresponding reconstruction output as separate event records for each collision.
The reconstruction program needs detector configuration and calibration data
that often varies with time, though reasonably slowly compared to the event rate. 58
Therefore, the reconstruction program accesses a “conditions database” in order to
fetch the configuration and calibration data matching the conditions under which a
particular event was acquired. Alternatively, if stable and reliable calibration data
are available at the time of acquisition, it may be stored within the raw data files,
interspersed with the event records.
Event records are, in principle, all independent from one another and therefore
suitable for loosely-coupled parallel processing. Early implementations 59 would
process one raw data file at a time, in essence treating the raw data file as a
buffer and implementing an “event server” to distribute individual events via the
network to CPU servers. This methodology has been largely abandoned for event
reconstruction, although it remains a good choice for parallel processing of analysis
data sets. Modern event reconstruction clusters or “farms” are configured so that
each CPU server executes jobs that process one or more complete data files at a time.
The CPU servers read raw and calibration data and write reconstructed data in an
autonomous fashion to the Disk and Database servers. Hundreds or even thousands
of jobs can run simultaneously in a single cluster.
Care must be taken to handle aspects that may introduce coupling and break
the highly parallel processing scheme. Consider, for example, one of the most
common pitfalls that breaks parallelism: simultaneous access to the same file by
many jobs. For example, the executable image of the reconstruction program is the
same for all jobs and may be quite large. Therefore, each job that starts must read
the image file from a Network File server. In an environment where thousands of
jobs are executed, the number of jobs simultaneously reading the image file may be
58 There are exceptions to this rule, especially for neutrino beam or non-accelerator events, where
calibration data may in fact be updated faster than the event rate.
59 Event processing in parallel dates back to the 1970s when physicists used their experience in
building digital electronic boards for detectors to design inexpensive processor boards, called
emulators, which executed the instruction set of the IBM System 370 mainframe computer and
interfaced to its input-output channel by emulating a tape drive. Starting in the mid-1980s, parallel
event processing was mostly done using specially configured clusters (called “farms”) of scientific
workstations with Reduced Instruction Set CPUs (RISC) running the VMS operating system
from Digital Equipment Corp. and a number of variants of Unix, interconnected by proprietary
networks or by daisy-chained Ethernet networks. Starting in the mid-1990s to date, essentially all
reconstruction is run on clusters of widely marketed, industry standard “commodity” computers
using the x86 instruction set supported by Intel and Advance Micro Devices CPU chips and
interconnected by switched Ethernet networks.
627
The output of the reconstruction arises from the application of the techniques
described in the contribution “Patter Recognition and Reconstruction” and consists
of data structures containing tracks, clusters, etc. These data structures are written as
event records, which may or may not have a one-to-one correspondence to the event
records read on input. For example, the reconstruction program may disentangle
data from different collisions contained in a single raw data event record and write
the corresponding reconstruction output as separate event records for each collision.
The reconstruction program needs detector configuration and calibration data
that often varies with time, though reasonably slowly compared to the event rate. 58
Therefore, the reconstruction program accesses a “conditions database” in order to
fetch the configuration and calibration data matching the conditions under which a
particular event was acquired. Alternatively, if stable and reliable calibration data
are available at the time of acquisition, it may be stored within the raw data files,
interspersed with the event records.
Event records are, in principle, all independent from one another and therefore
suitable for loosely-coupled parallel processing. Early implementations 59 would
process one raw data file at a time, in essence treating the raw data file as a
buffer and implementing an “event server” to distribute individual events via the
network to CPU servers. This methodology has been largely abandoned for event
reconstruction, although it remains a good choice for parallel processing of analysis
data sets. Modern event reconstruction clusters or “farms” are configured so that
each CPU server executes jobs that process one or more complete data files at a time.
The CPU servers read raw and calibration data and write reconstructed data in an
autonomous fashion to the Disk and Database servers. Hundreds or even thousands
of jobs can run simultaneously in a single cluster.
Care must be taken to handle aspects that may introduce coupling and break
the highly parallel processing scheme. Consider, for example, one of the most
common pitfalls that breaks parallelism: simultaneous access to the same file by
many jobs. For example, the executable image of the reconstruction program is the
same for all jobs and may be quite large. Therefore, each job that starts must read
the image file from a Network File server. In an environment where thousands of
jobs are executed, the number of jobs simultaneously reading the image file may be
58 There are exceptions to this rule, especially for neutrino beam or non-accelerator events, where
calibration data may in fact be updated faster than the event rate.
59 Event processing in parallel dates back to the 1970s when physicists used their experience in
building digital electronic boards for detectors to design inexpensive processor boards, called
emulators, which executed the instruction set of the IBM System 370 mainframe computer and
interfaced to its input-output channel by emulating a tape drive. Starting in the mid-1980s, parallel
event processing was mostly done using specially configured clusters (called “farms”) of scientific
workstations with Reduced Instruction Set CPUs (RISC) running the VMS operating system
from Digital Equipment Corp. and a number of variants of Unix, interconnected by proprietary
networks or by daisy-chained Ethernet networks. Starting in the mid-1990s to date, essentially all
reconstruction is run on clusters of widely marketed, industry standard “commodity” computers
using the x86 instruction set supported by Intel and Advance Micro Devices CPU chips and
interconnected by switched Ethernet networks.