16 Integration of Detectors into a Large Experiment: Examples from ATLAS. . .
751
with the smallest size, has, by construction, the most compact and complete physics
information of the event, and is thus going to be indispensable in carrying out highstatistics analyses.
In preparation for the hopefully soon-to-come high-statistics analysis era,
ATLAS has defined two further formats, namely a condensed data format for
tagging events with certain properties, called TAG, and a Derived Physics Data
format(or DPD), which are intended for use in end-user analyses. TAG data are
event-level metadata, i.e. thumbnail information about each event to enable rapid
and efficient selection for individual analyses. The TAG data are also stored in a
relational database to enable various searches via database queries. The average
size is a few kB per event. The DPD format corresponds to the highest-level of
data representation, with “ntuple”-like content, for direct analysis and display by
analysis programs.
These official data formats have been deployed as the vehicle for running physics
analyses. As an example, the AOD format and its contents have been the subject of
several generations of very extensive sets of tests with different data, conditions,
and subsequent uses. Of course, since the AOD format contains only a subset of the
information in the event, there will always be analyses that need to refer back to
the ESD format. The most critical part of the optimisation of these various formats
over the past few years has therefore been to select appropriately the objects to be
included in the AOD. There is usually a trade-off between storage cost and CPU to
derive the additional objects to be studied, and the details depend very strongly on
the sample size required and the number of times the sample is used.
16.7.1.2 Data Flow and Processing
To maximise the physics reach of the experiment, the HLT farms will write events
at the maximum possible data rate, which can be supported by the computing
resources. Currently, this is expected to be in the range of 200–300 Hz, essentially
independent of the instantaneous luminosity of the accelerator. Trigger thresholds
will be adjusted up or down to match the maximum data rate, in order to maintain
consistency with the data storage and processing capabilities of the offline systems.
Extensive test campaigns have shown that the online-offline link and the Tier-0
centre are able to keep up in real-time with the HLT output rate.
The HLT output is streamed according to trigger type for the subsequent
reconstruction and physics analysis. In addition, specialised calibration streams
allow for independent processing from the bulk of the physics data. These streams
are required to produce calibration and alignment constants of sufficient quality to
allow a useful first-pass processing of the physics streams with minimum latency.
ATLAS also makes use of an express stream, which is a set of physics triggers
corresponding to about 5% of the full data rate. These events are selected to tune the
physics and detector algorithms and also to provide rapid updates to the calibration
and alignment constants required for the first-pass processing.
751
with the smallest size, has, by construction, the most compact and complete physics
information of the event, and is thus going to be indispensable in carrying out highstatistics analyses.
In preparation for the hopefully soon-to-come high-statistics analysis era,
ATLAS has defined two further formats, namely a condensed data format for
tagging events with certain properties, called TAG, and a Derived Physics Data
format(or DPD), which are intended for use in end-user analyses. TAG data are
event-level metadata, i.e. thumbnail information about each event to enable rapid
and efficient selection for individual analyses. The TAG data are also stored in a
relational database to enable various searches via database queries. The average
size is a few kB per event. The DPD format corresponds to the highest-level of
data representation, with “ntuple”-like content, for direct analysis and display by
analysis programs.
These official data formats have been deployed as the vehicle for running physics
analyses. As an example, the AOD format and its contents have been the subject of
several generations of very extensive sets of tests with different data, conditions,
and subsequent uses. Of course, since the AOD format contains only a subset of the
information in the event, there will always be analyses that need to refer back to
the ESD format. The most critical part of the optimisation of these various formats
over the past few years has therefore been to select appropriately the objects to be
included in the AOD. There is usually a trade-off between storage cost and CPU to
derive the additional objects to be studied, and the details depend very strongly on
the sample size required and the number of times the sample is used.
16.7.1.2 Data Flow and Processing
To maximise the physics reach of the experiment, the HLT farms will write events
at the maximum possible data rate, which can be supported by the computing
resources. Currently, this is expected to be in the range of 200–300 Hz, essentially
independent of the instantaneous luminosity of the accelerator. Trigger thresholds
will be adjusted up or down to match the maximum data rate, in order to maintain
consistency with the data storage and processing capabilities of the offline systems.
Extensive test campaigns have shown that the online-offline link and the Tier-0
centre are able to keep up in real-time with the HLT output rate.
The HLT output is streamed according to trigger type for the subsequent
reconstruction and physics analysis. In addition, specialised calibration streams
allow for independent processing from the bulk of the physics data. These streams
are required to produce calibration and alignment constants of sufficient quality to
allow a useful first-pass processing of the physics streams with minimum latency.
ATLAS also makes use of an express stream, which is a set of physics triggers
corresponding to about 5% of the full data rate. These events are selected to tune the
physics and detector algorithms and also to provide rapid updates to the calibration
and alignment constants required for the first-pass processing.