628
M. Delfino
high enough to overload the Network File server, therefore creating a performance
bottleneck. 60 Caching of the image file on a local disk of each CPU server can
be used to restore parallelism more economically compared to deploying a more
powerful Network File server. A similar situation arises from the access by all jobs
to a unique conditions database. Parallelism is harder to restore in this case and
requires tuning of the database server and the caching algorithm, or deployment of
very powerful (and expensive) replicated database servers.
Raw data files must often be retrieved from tape, and parallelism may be broken
from the fact that a single tape cartridge is used to store multiple raw data files.
Many HSM systems are unable to handle multiple, asynchronous, closely spaced
requests for files on the same tape. They queue the requests and mount and dismount
the tape cartridge in order to read each of the files, a slow procedure as it involves
mechanical actions. As individual raw file sizes have historically increased more
slowly than total raw data volume, the number of files per tape cartridge has
increased and parallelism breakage by tape access has become quite common. In
order to circumvent these problems, production managers run special jobs to “prefetch” in an efficient way lists of specific datasets from tape to Disk servers. Efficient
pre-fetching requires detailed knowledge of the HSM software and the contents of
the tape cartridges in order to minimize the number of tape mounts.
Most modern experiments strive to run a production workflow for reconstructing
the raw data that keeps up with the data acquisition, called “quasi-online” or
“prompt” reconstruction. The quality of the calibration constants which are available
for this workflow is often limited and therefore the reconstructed output may be
of limited value for final physics results; the output is very valuable, however, to
monitor detector performance and to derive more definitive calibration constants.
Frequently, reconstruction of a particular raw data file must be repeated a few
times, in order to incorporate improvements or correct errors in calibration constants
and reconstruction algorithms. This is called “reprocessing” and applies to both
detector and simulated data.
14.4.2 Event Simulation
Production of simulated events usually starts with the generation of physics finalstates using an “event generator”. The CPU time needed for this step is usually
60 To understand how parallelism is broken, consider that the Network File server handles read
requests in parallel up to a certain limit and will serialize into a queue all requests exceeding
this limit, breaking parallelism. To understand how coupling is introduced, consider that the
event records are coupled through the fact that they are all being reconstructed using the same
reconstruction program image file. This can be exacerbated by additional coupling at the hardware
level, for example when many cores compete for the same Ethernet network interface in modern
multi-core computers.
M. Delfino
high enough to overload the Network File server, therefore creating a performance
bottleneck. 60 Caching of the image file on a local disk of each CPU server can
be used to restore parallelism more economically compared to deploying a more
powerful Network File server. A similar situation arises from the access by all jobs
to a unique conditions database. Parallelism is harder to restore in this case and
requires tuning of the database server and the caching algorithm, or deployment of
very powerful (and expensive) replicated database servers.
Raw data files must often be retrieved from tape, and parallelism may be broken
from the fact that a single tape cartridge is used to store multiple raw data files.
Many HSM systems are unable to handle multiple, asynchronous, closely spaced
requests for files on the same tape. They queue the requests and mount and dismount
the tape cartridge in order to read each of the files, a slow procedure as it involves
mechanical actions. As individual raw file sizes have historically increased more
slowly than total raw data volume, the number of files per tape cartridge has
increased and parallelism breakage by tape access has become quite common. In
order to circumvent these problems, production managers run special jobs to “prefetch” in an efficient way lists of specific datasets from tape to Disk servers. Efficient
pre-fetching requires detailed knowledge of the HSM software and the contents of
the tape cartridges in order to minimize the number of tape mounts.
Most modern experiments strive to run a production workflow for reconstructing
the raw data that keeps up with the data acquisition, called “quasi-online” or
“prompt” reconstruction. The quality of the calibration constants which are available
for this workflow is often limited and therefore the reconstructed output may be
of limited value for final physics results; the output is very valuable, however, to
monitor detector performance and to derive more definitive calibration constants.
Frequently, reconstruction of a particular raw data file must be repeated a few
times, in order to incorporate improvements or correct errors in calibration constants
and reconstruction algorithms. This is called “reprocessing” and applies to both
detector and simulated data.
14.4.2 Event Simulation
Production of simulated events usually starts with the generation of physics finalstates using an “event generator”. The CPU time needed for this step is usually
60 To understand how parallelism is broken, consider that the Network File server handles read
requests in parallel up to a certain limit and will serialize into a queue all requests exceeding
this limit, breaking parallelism. To understand how coupling is introduced, consider that the
event records are coupled through the fact that they are all being reconstructed using the same
reconstruction program image file. This can be exacerbated by additional coupling at the hardware
level, for example when many cores compete for the same Ethernet network interface in modern
multi-core computers.