14 Distributed Computing
639
to move virtual machines between different hardware units. This can be useful for
providing uninterruptable services or for saving energy in environments with large
load variations.
Traditional offline batch computing in particle physics, however, predominantly
uses virtualization for simply packaging together an application and its environment.
This reveals a weakness of traditional virtualization, as the many copies of operating
systems in the virtual machines require large amounts of memory (RAM) to execute
efficiently, increasing the costs of the computing clusters. This has recently been
circumvented by a new virtualization package, Singularity, 80 which has been rapidly
adopted by the scientific community, and large portions of WLCG now support the
execution of Singularity images. An additional advantage of Singularity is that it
can be easily supported on supercomputers.
Containers, in their present form, represent a slightly different approach to
virtualization. The current de-facto standard is Docker, 81 which is widely used
in commercial applications. Containers are closely associated to Linux operating
systems and to Cloud Computing environments (see below). Containers implement
virtual machines which are less hermetic than traditional ones and which have
certain limitations. On the other hand, containers can be rapidly created and
destroyed, and packages exist, such as Kubernetes, 82 for orchestrating a set of
containers which together can implement sophisticated applications using microservice architectures.
In a somewhat unexpected way, Singularity and container environments have
recently emerged as the leading approach for preserving data processing environments in order to ensure reproducibility of scientific results.
14.8.2 Cloud Computing and the Use of Commercial Data
Processing Services
Cloud Computing 83 is the currently used term 84 to describe the deployment of
computing resources and higher level computing services in a shareable, user
configurable way. It is the new paradigm that replaced client-server computing in the
80 Kurtzer GM, Sochat V, and Bauer MW (2017) Singularity: Scientific containers for mobility of
compute. PLoS ONE 12(5): e0177459. https://doi.org/10.1371/journal.pone.0177459
81 Boettiger, Carl. “An introduction to Docker for reproducible research.” ACM SIGOPS Operating
Systems Review 49.1 (2015): 71–79.
82 Bernstein, David. “Containers and cloud: From lxc to docker to kubernetes.” IEEE Cloud
Computing 3 (2014): 81–84.
83 Mell, Peter, Grance, Tim, “The NIST Definition of Cloud Computing”, Special Publication 800145, National Institute of Standards and Technology, U.S. Department of Commerce (September
2011) https://doi.org/10.6028/NIST.SP.800-145
84 Regalado, Antonio. “Who coined ‘cloud computing’.” Technology Review 31 (2011). https://
www.technologyreview.com/s/425970/who-coined-cloud-computing/Retrieved 3 February 2019.
639
to move virtual machines between different hardware units. This can be useful for
providing uninterruptable services or for saving energy in environments with large
load variations.
Traditional offline batch computing in particle physics, however, predominantly
uses virtualization for simply packaging together an application and its environment.
This reveals a weakness of traditional virtualization, as the many copies of operating
systems in the virtual machines require large amounts of memory (RAM) to execute
efficiently, increasing the costs of the computing clusters. This has recently been
circumvented by a new virtualization package, Singularity, 80 which has been rapidly
adopted by the scientific community, and large portions of WLCG now support the
execution of Singularity images. An additional advantage of Singularity is that it
can be easily supported on supercomputers.
Containers, in their present form, represent a slightly different approach to
virtualization. The current de-facto standard is Docker, 81 which is widely used
in commercial applications. Containers are closely associated to Linux operating
systems and to Cloud Computing environments (see below). Containers implement
virtual machines which are less hermetic than traditional ones and which have
certain limitations. On the other hand, containers can be rapidly created and
destroyed, and packages exist, such as Kubernetes, 82 for orchestrating a set of
containers which together can implement sophisticated applications using microservice architectures.
In a somewhat unexpected way, Singularity and container environments have
recently emerged as the leading approach for preserving data processing environments in order to ensure reproducibility of scientific results.
14.8.2 Cloud Computing and the Use of Commercial Data
Processing Services
Cloud Computing 83 is the currently used term 84 to describe the deployment of
computing resources and higher level computing services in a shareable, user
configurable way. It is the new paradigm that replaced client-server computing in the
80 Kurtzer GM, Sochat V, and Bauer MW (2017) Singularity: Scientific containers for mobility of
compute. PLoS ONE 12(5): e0177459. https://doi.org/10.1371/journal.pone.0177459
81 Boettiger, Carl. “An introduction to Docker for reproducible research.” ACM SIGOPS Operating
Systems Review 49.1 (2015): 71–79.
82 Bernstein, David. “Containers and cloud: From lxc to docker to kubernetes.” IEEE Cloud
Computing 3 (2014): 81–84.
83 Mell, Peter, Grance, Tim, “The NIST Definition of Cloud Computing”, Special Publication 800145, National Institute of Standards and Technology, U.S. Department of Commerce (September
2011) https://doi.org/10.6028/NIST.SP.800-145
84 Regalado, Antonio. “Who coined ‘cloud computing’.” Technology Review 31 (2011). https://
www.technologyreview.com/s/425970/who-coined-cloud-computing/Retrieved 3 February 2019.