752
D. Froidevaux
Streams can be used for a variety of purposes. The primary use, as mentioned
previously, is to allow the prioritisation of the processing of the data. As an example,
having the di-muon dataset as a independent stream obviously results in a much
faster turnaround on any analysis that relies on these data. Streams can also be useful
in the commissioning phase, to debug both the software and the overall online and
offline computing systems. As an example, a special “debug” stream is dedicated
to problematic events, e.g. failing in the HLT step, to facilitate the understanding of
errors in the system. Obviously, such streams will be created as the need arises, will
be rate-limited, and may even be withdrawn once the primary motivation for them
is no longer present.
The first step before full-fledged prompt reconstruction is the actual processing
of the calibration data in the shortest possible time. The plan calls for a short 1 to 2day latency in completing this task. Once the calibration and alignment constants are
in place, a first-pass (or prompt) reconstruction is run on the primary event streams,
and the resulting reconstructed data (ESD and AOD formats) are archived into the
CERN mass storage system.
Upon completion of this step, the data are distributed to the Tier-1 centres. Each
Tier-1 site assumes responsibility for a fraction of the reconstructed data. Most
of the ESD format data are, however, not available on disk for individual user
access. A major role for the Tier-1 centres is the reprocessing of the data, once
more mature calibrations and software are available, typically once or twice every
year. By shifting the burden of reprocessing to the Tier-1 centres, the experiment
can reprocess its data asynchronously and concurrently with data-taking and the
associated prompt processing. The Tier-2 centres can obtain partial or full copies of
the AOD/DPD/TAG format data, which will be the primary tool for physics analysis.
The Tier-2 centres will also be responsible for large-scale simulation tasks, once the
Tier-1 sites will be very busy with data reprocessing.
16.7.2 Software
On the software front, there have been two major issues encountered by the LHC
experiments, which are either new or simply appear to a much greater extent than
in the past: the distributed nature of the development and the maintainability of the
code over long time-scales:
• Software development has had to continue down the path established at LEP and
at the Tevatron: the code is developed in a distributed manner with responsibilities that span multiple individuals, institutions, countries and time zones. While
for the large-scale hardware projects, a factorisation of the overall construction
into substantial units has been possible, software, with its much wider contributor
base within the collaborations, has a larger degree of fragmentation. This has
necessitated the formation of intricate project structures to monitor and steer
D. Froidevaux
Streams can be used for a variety of purposes. The primary use, as mentioned
previously, is to allow the prioritisation of the processing of the data. As an example,
having the di-muon dataset as a independent stream obviously results in a much
faster turnaround on any analysis that relies on these data. Streams can also be useful
in the commissioning phase, to debug both the software and the overall online and
offline computing systems. As an example, a special “debug” stream is dedicated
to problematic events, e.g. failing in the HLT step, to facilitate the understanding of
errors in the system. Obviously, such streams will be created as the need arises, will
be rate-limited, and may even be withdrawn once the primary motivation for them
is no longer present.
The first step before full-fledged prompt reconstruction is the actual processing
of the calibration data in the shortest possible time. The plan calls for a short 1 to 2day latency in completing this task. Once the calibration and alignment constants are
in place, a first-pass (or prompt) reconstruction is run on the primary event streams,
and the resulting reconstructed data (ESD and AOD formats) are archived into the
CERN mass storage system.
Upon completion of this step, the data are distributed to the Tier-1 centres. Each
Tier-1 site assumes responsibility for a fraction of the reconstructed data. Most
of the ESD format data are, however, not available on disk for individual user
access. A major role for the Tier-1 centres is the reprocessing of the data, once
more mature calibrations and software are available, typically once or twice every
year. By shifting the burden of reprocessing to the Tier-1 centres, the experiment
can reprocess its data asynchronously and concurrently with data-taking and the
associated prompt processing. The Tier-2 centres can obtain partial or full copies of
the AOD/DPD/TAG format data, which will be the primary tool for physics analysis.
The Tier-2 centres will also be responsible for large-scale simulation tasks, once the
Tier-1 sites will be very busy with data reprocessing.
16.7.2 Software
On the software front, there have been two major issues encountered by the LHC
experiments, which are either new or simply appear to a much greater extent than
in the past: the distributed nature of the development and the maintainability of the
code over long time-scales:
• Software development has had to continue down the path established at LEP and
at the Tevatron: the code is developed in a distributed manner with responsibilities that span multiple individuals, institutions, countries and time zones. While
for the large-scale hardware projects, a factorisation of the overall construction
into substantial units has been possible, software, with its much wider contributor
base within the collaborations, has a larger degree of fragmentation. This has
necessitated the formation of intricate project structures to monitor and steer