14 Distributed Computing
641
executing data-intensive tasks. A similar project named ARCHIVER 90 will explore
in 2019–2021 the commercial provision of cloud mass storage services for scientific
data archiving and preservation.
Cloud computing deployment, especially on commercial services, is quite
complex at the infrastructure and platform level. Fortunately, past investments in
Grid computing can be re-used in order to hide these complexities from the vast
majority of users. Job and data management packaged used for Grid computing
have already been deployed on private and commercial clouds, as well as portals
for analysis using higher level tools such as Jupyter notebooks and Hadoop data
servers.
14.9 Future Challenges and Directions
Particle physics has historically been at the forefront of innovation in the adaptation
of computing systems to fulfill its growing needs, as well as to accommodate
tight budgets. It is rare, however, that the configurations developed in the particle
physics context can directly be taken over by commercial systems. They should
be considered as “precursors” or “early adoptions” of advances to come into
general use. This situation has become a challenge with the growing time spans
for experiments. If not properly managed, it may result in huge (often hidden) costs
for maintenance and the lack of adoption of other innovations.
A first consideration for the future development is the number of scientific
projects requiring wide-area distributed processing. In the short term, LHC will
continue to produce unprecedented amounts of data and will be joined by a few
experiments with similar needs, such as the Square Kilometer Array (SKA) 91
radiotelescope. However, many other experiments, such as neutrino detectors and
astroparticle experiments, may produce orders of magnitude smaller data volumes,
due to their own instrumental nature or to advances in data reduction within the
data acquisition platforms. For example, even though 100 Gbps WAN will become
common place by the 2020s, the baseline design of the Cherenkov Telescope Array
(CTA) 92 requires only a 1 Gbps network interface for the sites (a capacity already
available to home users in many countries). This is because CTA plans to use
powerful clusters housed in compact containerized data centers placed at the sites
to reduce the data at the instrument, thus avoiding the recording of large amounts of
90 ARCHIVER – Archiving and Preservation for Research Environments project has received
funding from the European Union’s Horizon 2020 research and innovation programme under grant
agreement No 824516. More information at https://archiver-project.eu
91 Dewdney, Peter E., et al. “The square kilometre array.” Proceedings of the IEEE 97.8 (2009):
1482–1496.
92 Actis, M., et al. “Design concepts for the Cherenkov Telescope Array CTA: an advanced facility
for ground-based high-energy gamma-ray astronomy.” Experimental Astronomy 32.3 (2011): 193–
316.
641
executing data-intensive tasks. A similar project named ARCHIVER 90 will explore
in 2019–2021 the commercial provision of cloud mass storage services for scientific
data archiving and preservation.
Cloud computing deployment, especially on commercial services, is quite
complex at the infrastructure and platform level. Fortunately, past investments in
Grid computing can be re-used in order to hide these complexities from the vast
majority of users. Job and data management packaged used for Grid computing
have already been deployed on private and commercial clouds, as well as portals
for analysis using higher level tools such as Jupyter notebooks and Hadoop data
servers.
14.9 Future Challenges and Directions
Particle physics has historically been at the forefront of innovation in the adaptation
of computing systems to fulfill its growing needs, as well as to accommodate
tight budgets. It is rare, however, that the configurations developed in the particle
physics context can directly be taken over by commercial systems. They should
be considered as “precursors” or “early adoptions” of advances to come into
general use. This situation has become a challenge with the growing time spans
for experiments. If not properly managed, it may result in huge (often hidden) costs
for maintenance and the lack of adoption of other innovations.
A first consideration for the future development is the number of scientific
projects requiring wide-area distributed processing. In the short term, LHC will
continue to produce unprecedented amounts of data and will be joined by a few
experiments with similar needs, such as the Square Kilometer Array (SKA) 91
radiotelescope. However, many other experiments, such as neutrino detectors and
astroparticle experiments, may produce orders of magnitude smaller data volumes,
due to their own instrumental nature or to advances in data reduction within the
data acquisition platforms. For example, even though 100 Gbps WAN will become
common place by the 2020s, the baseline design of the Cherenkov Telescope Array
(CTA) 92 requires only a 1 Gbps network interface for the sites (a capacity already
available to home users in many countries). This is because CTA plans to use
powerful clusters housed in compact containerized data centers placed at the sites
to reduce the data at the instrument, thus avoiding the recording of large amounts of
90 ARCHIVER – Archiving and Preservation for Research Environments project has received
funding from the European Union’s Horizon 2020 research and innovation programme under grant
agreement No 824516. More information at https://archiver-project.eu
91 Dewdney, Peter E., et al. “The square kilometre array.” Proceedings of the IEEE 97.8 (2009):
1482–1496.
92 Actis, M., et al. “Design concepts for the Cherenkov Telescope Array CTA: an advanced facility
for ground-based high-energy gamma-ray astronomy.” Experimental Astronomy 32.3 (2011): 193–
316.