14 Distributed Computing
623
servers are those based on “Big Data” tools, such as NoSQL databases 37 with
underlying Hadoop storage management systems. 38
• “Data servers”, based on products such as dCache, 39 GPFS, 40 Lustre, 41 EOS 42
and DPM, 43 are used to hold the large-size binary files which contain physics
data. They use a set of specific Control and Information Servers to present
to the other elements of the cluster, in particular to the CPU servers, an
interface approximating a conventional (POSIX-compliant) disk resource, but
with some important differences in order to achieve the large capacities and
high performance needed for particle physics applications (and data-intensive
applications in other fields). Much of the optimization is possible because of
the particular usage pattern of particle physics data, which is essentially written
once and then read many times. 44 The key point is to separate 45 the “namespace”
information (directory tree or file folder structures, file attributes such as name,
creation date, etc.), the “file storage” information (physical location of the blocks
that comprise a file, whether it is open for read, write or append, etc.) and the “file
data” information (the actual data blocks that comprise the file). This separation
makes it possible to serve the data blocks from a large number of Data servers
with a combined capacity far exceeding what can be provided by a conventional
file system. In addition, the data blocks of a given file may be stored in multiple
servers in order to enhance read performance or provide high-availability (for
reading). 46 The cost of this separation (apart from the complexity of the multiple
37 Han, Jing, et al. “Survey on NoSQL database.” Pervasive computing and applications (ICPCA),
2011 6th international conference on. IEEE, 2011.
38 Shvachko, Konstantin, et al. “The hadoop distributed file system.” Mass storage systems and
technologies (MSST), 2010 IEEE 26th symposium on. Ieee, 2010.
39 Millar, Paul, et al. “Storage for advanced scientific use-cases and beyond.” Parallel, Distributed
and Network-based Processing (PDP), 2018 26th Euromicro International Conference on. IEEE,
2018.
40 Schmuck, Frank B., and Roger L. Haskin. “GPFS: A Shared-Disk File System for Large
Computing Clusters.” FAST. Vol. 2. No. 19. 2002.
41 Schwan, Philip. “Lustre: Building a file system for 1000-node clusters.” Proceedings of the 2003
Linux symposium. Vol. 2003. 2003.
42 Peters, A. J., E. A. Sindrilaru, and G. Adde. “EOS as the present and future solution for data
storage at CERN.” Journal of Physics: Conference Series. Vol. 664. No. 4. IOP Publishing, 2015.
43 Alvarez, Alejandro, et al. “DPM: future proof storage.” Journal of Physics: Conference Series.
Vol. 396. No. 3. IOP Publishing, 2012.
44 Aside from the write-once/read-many access pattern, particle physics datasets almost never have
data appended to them nor have their already existing records updated. For an extreme contrast,
consider the access pattern of bank account datasets, which constantly have data appended and
updated.
45 Conventional file systems, whether local or networked, tightly bind the management of the three
types of information described in order to optimize file open/close operations and implement
sharing of files in read/write/append mode.
46 This is conceptually similar to the “striping” of files across many disks performed by RAID
controllers on DAS devices.
623
servers are those based on “Big Data” tools, such as NoSQL databases 37 with
underlying Hadoop storage management systems. 38
• “Data servers”, based on products such as dCache, 39 GPFS, 40 Lustre, 41 EOS 42
and DPM, 43 are used to hold the large-size binary files which contain physics
data. They use a set of specific Control and Information Servers to present
to the other elements of the cluster, in particular to the CPU servers, an
interface approximating a conventional (POSIX-compliant) disk resource, but
with some important differences in order to achieve the large capacities and
high performance needed for particle physics applications (and data-intensive
applications in other fields). Much of the optimization is possible because of
the particular usage pattern of particle physics data, which is essentially written
once and then read many times. 44 The key point is to separate 45 the “namespace”
information (directory tree or file folder structures, file attributes such as name,
creation date, etc.), the “file storage” information (physical location of the blocks
that comprise a file, whether it is open for read, write or append, etc.) and the “file
data” information (the actual data blocks that comprise the file). This separation
makes it possible to serve the data blocks from a large number of Data servers
with a combined capacity far exceeding what can be provided by a conventional
file system. In addition, the data blocks of a given file may be stored in multiple
servers in order to enhance read performance or provide high-availability (for
reading). 46 The cost of this separation (apart from the complexity of the multiple
37 Han, Jing, et al. “Survey on NoSQL database.” Pervasive computing and applications (ICPCA),
2011 6th international conference on. IEEE, 2011.
38 Shvachko, Konstantin, et al. “The hadoop distributed file system.” Mass storage systems and
technologies (MSST), 2010 IEEE 26th symposium on. Ieee, 2010.
39 Millar, Paul, et al. “Storage for advanced scientific use-cases and beyond.” Parallel, Distributed
and Network-based Processing (PDP), 2018 26th Euromicro International Conference on. IEEE,
2018.
40 Schmuck, Frank B., and Roger L. Haskin. “GPFS: A Shared-Disk File System for Large
Computing Clusters.” FAST. Vol. 2. No. 19. 2002.
41 Schwan, Philip. “Lustre: Building a file system for 1000-node clusters.” Proceedings of the 2003
Linux symposium. Vol. 2003. 2003.
42 Peters, A. J., E. A. Sindrilaru, and G. Adde. “EOS as the present and future solution for data
storage at CERN.” Journal of Physics: Conference Series. Vol. 664. No. 4. IOP Publishing, 2015.
43 Alvarez, Alejandro, et al. “DPM: future proof storage.” Journal of Physics: Conference Series.
Vol. 396. No. 3. IOP Publishing, 2012.
44 Aside from the write-once/read-many access pattern, particle physics datasets almost never have
data appended to them nor have their already existing records updated. For an extreme contrast,
consider the access pattern of bank account datasets, which constantly have data appended and
updated.
45 Conventional file systems, whether local or networked, tightly bind the management of the three
types of information described in order to optimize file open/close operations and implement
sharing of files in read/write/append mode.
46 This is conceptually similar to the “striping” of files across many disks performed by RAID
controllers on DAS devices.