634
M. Delfino
Grid batch system. Directives may be given to the CE specifying job requirements
such as minimum amount of memory, length of execution, etc. A challenge in
configuring very large Grids is to reach agreement on the normalizations used
to measure cluster resources, for example the speed of processors. Users submit
their jobs through an additional server, called a Resource Broker, which accesses
information in tables maintained by all available CEs and submits the jobs as
appropriate to balance the load amongst the multiple sites conforming the Grid.
Large projects, such as the LHC experiments, have developed job management
services which in practice replace the Resource Brokers, interacting directly with
CEs.
The Grid data access services described above offer functionality similar to the
low levels of operating systems, but with the great advantage of generating coherent
behavior across many Resource Centers. In practice, these services are often too
low-level for practical use by applications. Hence, additional software, services and
APIs are provided by middleware distributions at a higher level. One example is
the File Transfer Service (FTS), 72 designed to support applications which require
automated management, monitoring and error correction of replication of large
numbers of files and large volumes of data, such as the LHC experiments. It
uses a database back end to store replication requests and instances of finite state
machines to reliably accomplish the corresponding data replications, including error
correction and reporting. In addition, FTS implements the concept of a channel (a
virtual path between two Resource Centers) which may be managed by operators,
adjusting parameters such as the number of simultaneous files being transferred, the
maximum bandwidth used, etc.
Large scientific projects, such as particle physics experiments, can have specific
needs that go beyond the scope or functionality of basic middleware (such as UMD)
and basic Grid services (such as those offered by EGI or OSG). In these cases, the
projects themselves provide their own enhancements. 73
14.7 The Worldwide LHC Computing Grid (WLCG)
The largest data processing capacity deployed so far using clusters linked via
Grid infrastructures is the Worldwide LHC Computing Grid (WLCG or LCG). 74
WLCG links some 170 clusters located in 42 different countries into a single
72 Andrey Kiryanov, Alejandro Alvarez Ayllon and Oliver Keeble, “FTS3/WebFTS – A Powerful
File Transfer Service for Scientific Communities”, Procedia Computer Science, Volume 66, 2015,
Pages 670–678, ISSN 1877-0509, https://doi.org/10.1016/j.procs.2015.11.076
73 Readers should inquire within their own projects about these enhancements. They usually fall
into two categories: Job management enhancements, for example the DIRAC package used by the
LHCb experiment, and File or Dataset management enhancements, such as the AAA package used
in CMS and the Rucio package used by ATLAS.
74 See http://www.cern.ch/lcg
M. Delfino
Grid batch system. Directives may be given to the CE specifying job requirements
such as minimum amount of memory, length of execution, etc. A challenge in
configuring very large Grids is to reach agreement on the normalizations used
to measure cluster resources, for example the speed of processors. Users submit
their jobs through an additional server, called a Resource Broker, which accesses
information in tables maintained by all available CEs and submits the jobs as
appropriate to balance the load amongst the multiple sites conforming the Grid.
Large projects, such as the LHC experiments, have developed job management
services which in practice replace the Resource Brokers, interacting directly with
CEs.
The Grid data access services described above offer functionality similar to the
low levels of operating systems, but with the great advantage of generating coherent
behavior across many Resource Centers. In practice, these services are often too
low-level for practical use by applications. Hence, additional software, services and
APIs are provided by middleware distributions at a higher level. One example is
the File Transfer Service (FTS), 72 designed to support applications which require
automated management, monitoring and error correction of replication of large
numbers of files and large volumes of data, such as the LHC experiments. It
uses a database back end to store replication requests and instances of finite state
machines to reliably accomplish the corresponding data replications, including error
correction and reporting. In addition, FTS implements the concept of a channel (a
virtual path between two Resource Centers) which may be managed by operators,
adjusting parameters such as the number of simultaneous files being transferred, the
maximum bandwidth used, etc.
Large scientific projects, such as particle physics experiments, can have specific
needs that go beyond the scope or functionality of basic middleware (such as UMD)
and basic Grid services (such as those offered by EGI or OSG). In these cases, the
projects themselves provide their own enhancements. 73
14.7 The Worldwide LHC Computing Grid (WLCG)
The largest data processing capacity deployed so far using clusters linked via
Grid infrastructures is the Worldwide LHC Computing Grid (WLCG or LCG). 74
WLCG links some 170 clusters located in 42 different countries into a single
72 Andrey Kiryanov, Alejandro Alvarez Ayllon and Oliver Keeble, “FTS3/WebFTS – A Powerful
File Transfer Service for Scientific Communities”, Procedia Computer Science, Volume 66, 2015,
Pages 670–678, ISSN 1877-0509, https://doi.org/10.1016/j.procs.2015.11.076
73 Readers should inquire within their own projects about these enhancements. They usually fall
into two categories: Job management enhancements, for example the DIRAC package used by the
LHCb experiment, and File or Dataset management enhancements, such as the AAA package used
in CMS and the Rucio package used by ATLAS.
74 See http://www.cern.ch/lcg