632
M. Delfino
selection criteria to events in DST-like files or in personal or group files derived
from them and written directly in the format of the analysis tools. The basic record
in these files is often a collection of ordered lists of properties, or “n-tuples”.
Distributed computing is used for preparing the input files for the interactive
tools, using the same methodologies described earlier for running reconstruction
and data reduction jobs. In addition, since the application of selection criteria and the
generation of histograms from n-tuple files can often be mapped to highly parallel
tasks (if each n-tuple record is independent of any other), efforts have been made to
provide parallel “back-ends” to interactive analysis tools, for example the PROOF
back-end to ROOT or the use of Hadoop in conjunction with Jupyter Notebooks.
The idea is simple in principle: the user issues a command that specifies some
actions to be performed on all records; the back-end automatically generates and
executes in parallel a number of tasks reading the data and writing partial result
files; when all parallel tasks have finished, the partial results are merged to a
global result which is presented to the user via the interactive front-end. Speed is
gained, in principle, by using many CPUs and, more importantly, many input data
channels in parallel. In addition, since the interactive user pauses to examine the
results, resources can be used more efficiently by serving many users from a single
parallel back-end. Practical implementation with good performance, however, can
be extremely complex, as they involve running a cluster at very large peak input–
output rates and clever scheduling of bursts of parallel tasks.
14.6 Multiple Sites and Grid Computing
Grids were defined in the late 1990s as a new manner of integrating services from
clusters across multiple sites. 68 Practical development and deployment of Grids was
originally led by the particle physics community, especially the LHC collaborations.
Grids are convenient for two main purposes: data distribution to many end-users
who are geographically dispersed, and data processing load sharing across many
computing centers (“resource” centers).
The main idea of a Grid is to abstract the most commonly used interfaces for
authentication, access control, job submission and data access into a unique “metainterface”, which is the only interface exposed to end-users. Therefore, multiple
sites are integrated for the end-user under the illusion of a single “meta-cluster”.
Specialized Grid Information and Control servers are deployed in order to translate
from the abstract meta-interface into the specific interface used by each cluster.
Grid interfaces and protocols have not been standardized. After an intense period
of development during the 2000s, a handful of partially compatible Grid schemes
are now deployed. Many claims of success in providing resources on the Grid
68 “The grid: blueprint for a new computing infrastructure”, Ian Foster and Carl Kesselman, editors.
San Francisco: Morgan Kaufmann Publishers, c1999. ISBN 1558604758.
M. Delfino
selection criteria to events in DST-like files or in personal or group files derived
from them and written directly in the format of the analysis tools. The basic record
in these files is often a collection of ordered lists of properties, or “n-tuples”.
Distributed computing is used for preparing the input files for the interactive
tools, using the same methodologies described earlier for running reconstruction
and data reduction jobs. In addition, since the application of selection criteria and the
generation of histograms from n-tuple files can often be mapped to highly parallel
tasks (if each n-tuple record is independent of any other), efforts have been made to
provide parallel “back-ends” to interactive analysis tools, for example the PROOF
back-end to ROOT or the use of Hadoop in conjunction with Jupyter Notebooks.
The idea is simple in principle: the user issues a command that specifies some
actions to be performed on all records; the back-end automatically generates and
executes in parallel a number of tasks reading the data and writing partial result
files; when all parallel tasks have finished, the partial results are merged to a
global result which is presented to the user via the interactive front-end. Speed is
gained, in principle, by using many CPUs and, more importantly, many input data
channels in parallel. In addition, since the interactive user pauses to examine the
results, resources can be used more efficiently by serving many users from a single
parallel back-end. Practical implementation with good performance, however, can
be extremely complex, as they involve running a cluster at very large peak input–
output rates and clever scheduling of bursts of parallel tasks.
14.6 Multiple Sites and Grid Computing
Grids were defined in the late 1990s as a new manner of integrating services from
clusters across multiple sites. 68 Practical development and deployment of Grids was
originally led by the particle physics community, especially the LHC collaborations.
Grids are convenient for two main purposes: data distribution to many end-users
who are geographically dispersed, and data processing load sharing across many
computing centers (“resource” centers).
The main idea of a Grid is to abstract the most commonly used interfaces for
authentication, access control, job submission and data access into a unique “metainterface”, which is the only interface exposed to end-users. Therefore, multiple
sites are integrated for the end-user under the illusion of a single “meta-cluster”.
Specialized Grid Information and Control servers are deployed in order to translate
from the abstract meta-interface into the specific interface used by each cluster.
Grid interfaces and protocols have not been standardized. After an intense period
of development during the 2000s, a handful of partially compatible Grid schemes
are now deployed. Many claims of success in providing resources on the Grid
68 “The grid: blueprint for a new computing infrastructure”, Ian Foster and Carl Kesselman, editors.
San Francisco: Morgan Kaufmann Publishers, c1999. ISBN 1558604758.