36
S. P. Donegan and M. A. Groeber
For a raveled attribute matrix with scalar dimensions, the rows are comprised
of specific attribute arrays, while the columns represent particular objects. This
storage scheme generalizes to higher dimensions, where attribute arrays are stored
in hyper-rows and objects are denoted by the hyper-columns. Note that an attribute
matrix is extensible: new attribute arrays may simply be appended to the matrix
without the need for resizing. Attribute matrices may be interrogated in either
direction: obtaining arrays along (hyper)-rows, which return information about a
given attribute for each object, or property vectors along each (hyper)-column,
yielding a list of attributes for a specific objet.
Attribute arrays, the (hyper-)rows of attribute matrices, are the final leaves
of the overall data tree. These arrays store dense, heavy data. Arrays may be
multicomponent, defining a depth dimension at each tuple. The overall array shape,
tuple dimensions, is inherited from the dimensions of its parent attribute matrix.
SIMPL allows for any fundamental data type to be stored within an attribute array,
including various precision integers and floating point numbers. Attribute arrays
are stored compactly within attribute matrices, even if the arrays do not share the
same component dimensions: therefore, an attribute matrix is sparse in its depth
dimension, as shown in Fig. 9.
The SIMPL data structure is highly flexible and customizable. In order to
serialize it to storage, a data format must be used that is similarly flexible. SIMPL
utilizes the Hierarchical Data Format, or HDF5, as its data format [54]. HDF5 is
a binary file format whose data model allows for explicit hierarchy by organizing
information into groups, similar to folders on a file system, while dense data are
stored in datasets. SIMPL takes advantage of this model by mapping its data
container arrays, data containers, and attribute matrices to groups in an HDF5 file,
with attribute arrays being stored in datasets. An example mapping for a SIMPL
data file is shown in Fig. 10. HDF5 is an open standard which enables easy crossplatform data sharing and data transfer to toolsets other than SIMPL or DREAM.3D.
Similar to SIMPL, HDF5 allows for dense data to have arbitrary shape and
component dimensions and store any fundamental data type. Critically, the analysis
workflow used to generate the SIMPL data structure is stored along with the data
within the SIMPL file, enabling reproducibility and archival.
5.2 Filters, Pipelines, and Plugins
SIMPL defines a standard interface for interacting with the data structure through
the concept of filters. A filter is simply a self-contained function that performs some
operational interaction with the data structure, such as creating a new object (i.e.,
computing some new information) or modifying an existing object. Filters adhere
to a standardized interface defined in an abstract base class. A critical feature of
filters is their ability to request parameters from a user and translate these requests
into queries of the data structure. For example, a user may select an attribute array
by supplying a filter with a path; the filter interface will then utilize this path to
S. P. Donegan and M. A. Groeber
For a raveled attribute matrix with scalar dimensions, the rows are comprised
of specific attribute arrays, while the columns represent particular objects. This
storage scheme generalizes to higher dimensions, where attribute arrays are stored
in hyper-rows and objects are denoted by the hyper-columns. Note that an attribute
matrix is extensible: new attribute arrays may simply be appended to the matrix
without the need for resizing. Attribute matrices may be interrogated in either
direction: obtaining arrays along (hyper)-rows, which return information about a
given attribute for each object, or property vectors along each (hyper)-column,
yielding a list of attributes for a specific objet.
Attribute arrays, the (hyper-)rows of attribute matrices, are the final leaves
of the overall data tree. These arrays store dense, heavy data. Arrays may be
multicomponent, defining a depth dimension at each tuple. The overall array shape,
tuple dimensions, is inherited from the dimensions of its parent attribute matrix.
SIMPL allows for any fundamental data type to be stored within an attribute array,
including various precision integers and floating point numbers. Attribute arrays
are stored compactly within attribute matrices, even if the arrays do not share the
same component dimensions: therefore, an attribute matrix is sparse in its depth
dimension, as shown in Fig. 9.
The SIMPL data structure is highly flexible and customizable. In order to
serialize it to storage, a data format must be used that is similarly flexible. SIMPL
utilizes the Hierarchical Data Format, or HDF5, as its data format [54]. HDF5 is
a binary file format whose data model allows for explicit hierarchy by organizing
information into groups, similar to folders on a file system, while dense data are
stored in datasets. SIMPL takes advantage of this model by mapping its data
container arrays, data containers, and attribute matrices to groups in an HDF5 file,
with attribute arrays being stored in datasets. An example mapping for a SIMPL
data file is shown in Fig. 10. HDF5 is an open standard which enables easy crossplatform data sharing and data transfer to toolsets other than SIMPL or DREAM.3D.
Similar to SIMPL, HDF5 allows for dense data to have arbitrary shape and
component dimensions and store any fundamental data type. Critically, the analysis
workflow used to generate the SIMPL data structure is stored along with the data
within the SIMPL file, enabling reproducibility and archival.
5.2 Filters, Pipelines, and Plugins
SIMPL defines a standard interface for interacting with the data structure through
the concept of filters. A filter is simply a self-contained function that performs some
operational interaction with the data structure, such as creating a new object (i.e.,
computing some new information) or modifying an existing object. Filters adhere
to a standardized interface defined in an abstract base class. A critical feature of
filters is their ability to request parameters from a user and translate these requests
into queries of the data structure. For example, a user may select an attribute array
by supplying a filter with a path; the filter interface will then utilize this path to