394
preservation/resources/rfs/) and provides detailed format description documents for
different categories of digital data, including data sets, images, and geospatial data.
Once data characteristics for long-term storage are known, one can define shortterm storage structures and the workflow leading from raw data collection, to
cleaned-up data, to preliminary and final results and products. Data backups should
match the original data structure so recovery requires only a few minutes. Ideally,
backups are done automatically, continuously, and incrementally, and data history is
preserved. Data recovery should be tested on a regular basis. Additionally, it is
important to check how long data backups are being sustained. When storage space
is limited, it makes sense to use a time-dependent structure, such as keeping daily
backups for a year, biweekly backups for 3 years, and monthly backups thereafter.
Several resources provide details about good data management practices, including
the Oak Ridge National Laboratory Distributed Active Archive Center (https://daac.
ornl.gov/datamanagement/) and the rOpenSci initiative (https://ropensci.github.io/
reproducibility-guide/).
Fundamentals of data organization include (see, e.g., Cook et al. 2018):
• Definition of file/folder content: Keep similar measurements in one data file or
folder (e.g., if the documentation/metadata for data are the same, then the data
products should all be part of one data set).
• Variable definition: Describe the variable name and explicitly state the units and
formats in the metadata; use commonly accepted names, units, and formats and
provide details on the standards used (e.g., SI units, ISO standards, nomenclature
standards); use format consistently throughout the file; use a consistent code
(e.g., −9999) for missing values; and use only one variable per measurement
(e.g., avoid reporting coordinates in more than one coordinate system or time in
several time zones).
• Consistent data organization: Do not change or rearrange columns in the original data; include header rows (first row should contain file name, data set title,
author, date, and companion file names); use column headings to describe the
content of each column; include one row for variable names and one for variable
units; and make sure either each row in a file represents a complete record (with
columns representing all variables that make up the record) or each variable is
placed in an individual row (e.g., for relational databases).
• Stability of file formats: Avoid proprietary formats, and prefer formats encoding
information with a lossless algorithm (e.g., text, comma/tab-separated values,
SQL, XML, HTML, TIFF, PNG, GIF, WAV, postscript formats).
• Descriptive file names: Use descriptive, unique file names; use ASCII characters
only and avoid spaces (e.g., start with ISO date, followed by descriptive file
name: 20180430_siteA_plotB_vegSurvey); remember that file names are not a
replacement for metadata; explain naming structure of files in metadata; organize
files logically; and make sure directory structure and file names are both humanand machine-readable (check operating or database system limitations on file
name length and allowed characters).
A. K. Schweiger
preservation/resources/rfs/) and provides detailed format description documents for
different categories of digital data, including data sets, images, and geospatial data.
Once data characteristics for long-term storage are known, one can define shortterm storage structures and the workflow leading from raw data collection, to
cleaned-up data, to preliminary and final results and products. Data backups should
match the original data structure so recovery requires only a few minutes. Ideally,
backups are done automatically, continuously, and incrementally, and data history is
preserved. Data recovery should be tested on a regular basis. Additionally, it is
important to check how long data backups are being sustained. When storage space
is limited, it makes sense to use a time-dependent structure, such as keeping daily
backups for a year, biweekly backups for 3 years, and monthly backups thereafter.
Several resources provide details about good data management practices, including
the Oak Ridge National Laboratory Distributed Active Archive Center (https://daac.
ornl.gov/datamanagement/) and the rOpenSci initiative (https://ropensci.github.io/
reproducibility-guide/).
Fundamentals of data organization include (see, e.g., Cook et al. 2018):
• Definition of file/folder content: Keep similar measurements in one data file or
folder (e.g., if the documentation/metadata for data are the same, then the data
products should all be part of one data set).
• Variable definition: Describe the variable name and explicitly state the units and
formats in the metadata; use commonly accepted names, units, and formats and
provide details on the standards used (e.g., SI units, ISO standards, nomenclature
standards); use format consistently throughout the file; use a consistent code
(e.g., −9999) for missing values; and use only one variable per measurement
(e.g., avoid reporting coordinates in more than one coordinate system or time in
several time zones).
• Consistent data organization: Do not change or rearrange columns in the original data; include header rows (first row should contain file name, data set title,
author, date, and companion file names); use column headings to describe the
content of each column; include one row for variable names and one for variable
units; and make sure either each row in a file represents a complete record (with
columns representing all variables that make up the record) or each variable is
placed in an individual row (e.g., for relational databases).
• Stability of file formats: Avoid proprietary formats, and prefer formats encoding
information with a lossless algorithm (e.g., text, comma/tab-separated values,
SQL, XML, HTML, TIFF, PNG, GIF, WAV, postscript formats).
• Descriptive file names: Use descriptive, unique file names; use ASCII characters
only and avoid spaces (e.g., start with ISO date, followed by descriptive file
name: 20180430_siteA_plotB_vegSurvey); remember that file names are not a
replacement for metadata; explain naming structure of files in metadata; organize
files logically; and make sure directory structure and file names are both humanand machine-readable (check operating or database system limitations on file
name length and allowed characters).
A. K. Schweiger