14 Distributed Computing
643
or OpenStack swift 96 Object Storage. Therefore, a likely scenario is that current
storage access techniques used by particle physics will become unsupported in the
future. Of course, virtualization and Cloud techniques will make it possible for the
particle physics community to deploy these legacy technologies themselves, with
the corresponding expenses in personnel. A better way forward is to collaborate
with industry, and between various Resource Centers, to test alternative tools and
methods. Realistic evaluations are far from trivial, however, as they may require
substantial investments by all parties, for example to provide UMD Grid or Cloud
interfaces.
Another looming issue is the efficiency of use of the CPUs under data-intensive
conditions. One basic assumption of the distributed architecture used by the particle
physics community, as described above, is the existence of an infinitely powerful
network connecting all elements, which in effect assumes infinitely powerful
data servers as well as data ingestion by CPUs. Ever increasing data processing
requirements and more and more powerful multi-core CPU nodes, however, are
revealing data serving performance bottlenecks in clusters coming from the network
and disk and CPU server limitations. Removing these bottlenecks may require more
sophisticated architectures to be deployed.
A much more severe issue is related to the simplistic, often naïve, manner
in which large particle physics projects pretend to use Grids and Clouds. They
essentially desire them to behave as a very large cluster perfectly tuned to their
needs, ignoring that imposing unreasonably high requirements in peak network
and data serving rates within and between clusters can be very expensive. A Grid
would better be viewed as a loosely connected federation of largely autonomous,
self-sufficient clusters, with some of these clusters possibly hosted on commercial
services.
These and other future challenges are being addressed in a coherent manner by
the particle physics community through the HEP Software Foundation 97,98 and will
certainly require a vigorous new cycle of research and development, in collaboration
with computer scientists and engineers, the supercomputing and HPC communities
and industry.
96 Arnold, Joe. Openstack swift: Using, administering, and developing for swift object storage.
“O’Reilly Media, Inc.”, 2014.
97 Alves Jr, Antonio Augusto. A Roadmap for HEP Software and Computing R&D for the
2020s. No. HSF-CWP-2017-001; HSF-CWP-2017-01; FERMILAB-PUB-17-607-CD; arXiv:
1712.06982. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Brookhaven
National Laboratory (BNL), Upton, NY (United States); Lawrence Berkeley National Lab.
(LBNL), Berkeley, CA (United States); SLAC National Accelerator Lab., Menlo Park, CA (United
States); Thomas Jefferson National Accelerator Facility (TJNAF), Newport News, VA (United
States); Argonne National Lab. (ANL), Argonne, IL (United States), 2017.
98 Slides from the Computing in High Energy Physics 2018 conference (proceedings to
be published) https://indico.cern.ch/event/587955/contributions/3012294/attachments/1681524/
2708636/CHEP18_-_CWP_Lessons_and_Future_Work.pdf
643
or OpenStack swift 96 Object Storage. Therefore, a likely scenario is that current
storage access techniques used by particle physics will become unsupported in the
future. Of course, virtualization and Cloud techniques will make it possible for the
particle physics community to deploy these legacy technologies themselves, with
the corresponding expenses in personnel. A better way forward is to collaborate
with industry, and between various Resource Centers, to test alternative tools and
methods. Realistic evaluations are far from trivial, however, as they may require
substantial investments by all parties, for example to provide UMD Grid or Cloud
interfaces.
Another looming issue is the efficiency of use of the CPUs under data-intensive
conditions. One basic assumption of the distributed architecture used by the particle
physics community, as described above, is the existence of an infinitely powerful
network connecting all elements, which in effect assumes infinitely powerful
data servers as well as data ingestion by CPUs. Ever increasing data processing
requirements and more and more powerful multi-core CPU nodes, however, are
revealing data serving performance bottlenecks in clusters coming from the network
and disk and CPU server limitations. Removing these bottlenecks may require more
sophisticated architectures to be deployed.
A much more severe issue is related to the simplistic, often naïve, manner
in which large particle physics projects pretend to use Grids and Clouds. They
essentially desire them to behave as a very large cluster perfectly tuned to their
needs, ignoring that imposing unreasonably high requirements in peak network
and data serving rates within and between clusters can be very expensive. A Grid
would better be viewed as a loosely connected federation of largely autonomous,
self-sufficient clusters, with some of these clusters possibly hosted on commercial
services.
These and other future challenges are being addressed in a coherent manner by
the particle physics community through the HEP Software Foundation 97,98 and will
certainly require a vigorous new cycle of research and development, in collaboration
with computer scientists and engineers, the supercomputing and HPC communities
and industry.
96 Arnold, Joe. Openstack swift: Using, administering, and developing for swift object storage.
“O’Reilly Media, Inc.”, 2014.
97 Alves Jr, Antonio Augusto. A Roadmap for HEP Software and Computing R&D for the
2020s. No. HSF-CWP-2017-001; HSF-CWP-2017-01; FERMILAB-PUB-17-607-CD; arXiv:
1712.06982. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Brookhaven
National Laboratory (BNL), Upton, NY (United States); Lawrence Berkeley National Lab.
(LBNL), Berkeley, CA (United States); SLAC National Accelerator Lab., Menlo Park, CA (United
States); Thomas Jefferson National Accelerator Facility (TJNAF), Newport News, VA (United
States); Argonne National Lab. (ANL), Argonne, IL (United States), 2017.
98 Slides from the Computing in High Energy Physics 2018 conference (proceedings to
be published) https://indico.cern.ch/event/587955/contributions/3012294/attachments/1681524/
2708636/CHEP18_-_CWP_Lessons_and_Future_Work.pdf