14 Distributed Computing
631
14.4.4 Processing of Calibration Data and Calculation
of Calibration Constants
Reconstruction of data from detectors requires a large amount of calibration
constants, many of them time-varying. Raw calibration data, such as temperatures,
pressures and voltages are recorded periodically during data taking. Calibration data
related to response and alignment of different detector elements can be obtained by
generating artificial events with specialized calibration devices, for example pulsed
lasers. Alternatively, they can be calculated from the raw physics data itself, using
better understood reactions or additional events occurring in parallel to the physics,
such as the passage of cosmic rays.
Data input to calibration procedures often comes from diverse sources, and
handling it poses challenges that are often underestimated. For example, laser
pulses for calibration may be recorded in parallel to the physics data and stored
by the online system interspersed in the raw data files, thereby requiring the offline
to access all raw data files in order to process the laser pulses, a rather tedious
procedure. Conversely, sometimes calibration data are stored in a potpourri of small
files and databases of diverse formats, introducing unnecessary complexity in the
handling of these data.
There is no standardized framework to handle the output of all calibration
procedures, although the recent trend is to centralize it into a single calibration (or
“conditions”) database that is accessed through heavily cached methods by instances
of the reconstruction, DST generation and analysis programs. Care must be taken
to serve this database with adequate resources, and to ensure efficient and cached
access to it in order not to break parallelism as described earlier.
The flow of raw and derived calibration data and the related data processing tasks
should be studied with equal care as that of the physics data, and solutions should
be implemented to ensure efficient and agile data access for calibration purposes.
14.5 Interactive Analysis Using Clusters
The last stages of physics analysis are performed by individual users using
interactive tools such as ROOT, 65 Jupyter Notebooks 66 or SWAN. 67 They apply
65 See http://root.cern.ch
66 See https://jupyter.org/about. Athough Jupyter Notebooks are language agnostic, they are closely
associated with the Python language. They are increasingly popular in particle physics and many
other scientific disciplines, given the rise of the use of Python by them. Jupyter Notebooks must be
combined with suitable data-access modules. For example, the SWAN project at CERN combines
Jupyter Notebook with ROOT input–output modules to create an interactive analysis platform.
67 Danilo Piparo, Enric Tejedor, Pere Mato, Luca Mascetti, Jakub Moscicki, Massimo Lamanna,
SWAN: A service for interactive analysis in the cloud, Future Generation Computer Systems, Volume 78, Part 3, 2018, Pages 1071–1078, ISSN 0167-739X, https://doi.org/10.1016/
j.future.2016.11.035
631
14.4.4 Processing of Calibration Data and Calculation
of Calibration Constants
Reconstruction of data from detectors requires a large amount of calibration
constants, many of them time-varying. Raw calibration data, such as temperatures,
pressures and voltages are recorded periodically during data taking. Calibration data
related to response and alignment of different detector elements can be obtained by
generating artificial events with specialized calibration devices, for example pulsed
lasers. Alternatively, they can be calculated from the raw physics data itself, using
better understood reactions or additional events occurring in parallel to the physics,
such as the passage of cosmic rays.
Data input to calibration procedures often comes from diverse sources, and
handling it poses challenges that are often underestimated. For example, laser
pulses for calibration may be recorded in parallel to the physics data and stored
by the online system interspersed in the raw data files, thereby requiring the offline
to access all raw data files in order to process the laser pulses, a rather tedious
procedure. Conversely, sometimes calibration data are stored in a potpourri of small
files and databases of diverse formats, introducing unnecessary complexity in the
handling of these data.
There is no standardized framework to handle the output of all calibration
procedures, although the recent trend is to centralize it into a single calibration (or
“conditions”) database that is accessed through heavily cached methods by instances
of the reconstruction, DST generation and analysis programs. Care must be taken
to serve this database with adequate resources, and to ensure efficient and cached
access to it in order not to break parallelism as described earlier.
The flow of raw and derived calibration data and the related data processing tasks
should be studied with equal care as that of the physics data, and solutions should
be implemented to ensure efficient and agile data access for calibration purposes.
14.5 Interactive Analysis Using Clusters
The last stages of physics analysis are performed by individual users using
interactive tools such as ROOT, 65 Jupyter Notebooks 66 or SWAN. 67 They apply
65 See http://root.cern.ch
66 See https://jupyter.org/about. Athough Jupyter Notebooks are language agnostic, they are closely
associated with the Python language. They are increasingly popular in particle physics and many
other scientific disciplines, given the rise of the use of Python by them. Jupyter Notebooks must be
combined with suitable data-access modules. For example, the SWAN project at CERN combines
Jupyter Notebook with ROOT input–output modules to create an interactive analysis platform.
67 Danilo Piparo, Enric Tejedor, Pere Mato, Luca Mascetti, Jakub Moscicki, Massimo Lamanna,
SWAN: A service for interactive analysis in the cloud, Future Generation Computer Systems, Volume 78, Part 3, 2018, Pages 1071–1078, ISSN 0167-739X, https://doi.org/10.1016/
j.future.2016.11.035