Data Management, Retention and Availability
With such large volumes of data being generated, particularly via high-throughput
and automated approaches, facilities are now having to seriously consider their
position with respect to commitments to users and the broader community with
respect to policies regarding the data they collect. Not only does the location of data
have to be managed over time, but also its accessibility – and this applies to all scales
of operation. National-level facilities, being centralised and multi-user operations,
generally have data statements which illustrate the stance they have adopted. Beyond
these statements, such facilities are in varying states of readiness with respect to
having formal policies in place to address them. As the requirement to make a greater
commitment to data management rises, the resourcing required is becoming more
supported, and accordingly policies are becoming more widespread and meaningful.
For example, the Diamond Light Source Data Management policy [50] statement
declares ‘The purpose of this policy is to provide Users conducting Peer Reviewed
Research with information and guidance on Experimental Data ownership, storage,
access and management and to ensure that Experimental Data is managed and used
in ways that maximises public benefit’. Backing this up, the policy states that all data
and metadata generated by facility users will be kept live for a minimum of 30 days
after which a long-term archive copy is created – for the first 3 years, access to this is
exclusive to the experiment team, but after this period, the data may be made openly
available (under a CC-BY-4.0 licence). The ESRF data policy [51] has the same
3-year embargo period and commits to definite long-term storage of 5 years, but
striving for 10 years. However, the ESRF policy goes further in that it makes a
commitment to underpinning the publication process via provision of a DOI that can
be cited in articles publishing the results derived from this raw data – this implies a
much longer-term retention for data relating to publications. Conversely the
Advanced Photon Source data retention policy [52] is rather sparse and makes no
commitment to long-term storage, stating ‘Experiment data and metadata collected
at APS beamlines may be stored at and retrieved from the facility for at least
3 months’.
While such policies are generally the concern of central, national facilities, these
commitments indicate the longer-term view and requirements of the research community. It will therefore become increasingly more important for smaller-scale
facilities to consider their commitments to data management. This consideration is
made increasingly more important by the imposed requirements of the funders and
publishers of research for more transparent and rigorous access and management of
data. Amongst the smaller-scale chemical crystallography facilities, there are varying attitudes and states of readiness towards these aspects of data management, and it
will therefore take longer to address. It is clear that additional resources, such as
finance and expertise, will need to be found in order for this to be implemented fully
at any scale facility. It is also worthy of note that databases such as the PDB and CSD
now facilitate and expose links, via DOIs, from records in their databases to raw
diffraction data, which aids the discoverability of raw data.
90
S. J. Coles et al.
With such large volumes of data being generated, particularly via high-throughput
and automated approaches, facilities are now having to seriously consider their
position with respect to commitments to users and the broader community with
respect to policies regarding the data they collect. Not only does the location of data
have to be managed over time, but also its accessibility – and this applies to all scales
of operation. National-level facilities, being centralised and multi-user operations,
generally have data statements which illustrate the stance they have adopted. Beyond
these statements, such facilities are in varying states of readiness with respect to
having formal policies in place to address them. As the requirement to make a greater
commitment to data management rises, the resourcing required is becoming more
supported, and accordingly policies are becoming more widespread and meaningful.
For example, the Diamond Light Source Data Management policy [50] statement
declares ‘The purpose of this policy is to provide Users conducting Peer Reviewed
Research with information and guidance on Experimental Data ownership, storage,
access and management and to ensure that Experimental Data is managed and used
in ways that maximises public benefit’. Backing this up, the policy states that all data
and metadata generated by facility users will be kept live for a minimum of 30 days
after which a long-term archive copy is created – for the first 3 years, access to this is
exclusive to the experiment team, but after this period, the data may be made openly
available (under a CC-BY-4.0 licence). The ESRF data policy [51] has the same
3-year embargo period and commits to definite long-term storage of 5 years, but
striving for 10 years. However, the ESRF policy goes further in that it makes a
commitment to underpinning the publication process via provision of a DOI that can
be cited in articles publishing the results derived from this raw data – this implies a
much longer-term retention for data relating to publications. Conversely the
Advanced Photon Source data retention policy [52] is rather sparse and makes no
commitment to long-term storage, stating ‘Experiment data and metadata collected
at APS beamlines may be stored at and retrieved from the facility for at least
3 months’.
While such policies are generally the concern of central, national facilities, these
commitments indicate the longer-term view and requirements of the research community. It will therefore become increasingly more important for smaller-scale
facilities to consider their commitments to data management. This consideration is
made increasingly more important by the imposed requirements of the funders and
publishers of research for more transparent and rigorous access and management of
data. Amongst the smaller-scale chemical crystallography facilities, there are varying attitudes and states of readiness towards these aspects of data management, and it
will therefore take longer to address. It is clear that additional resources, such as
finance and expertise, will need to be found in order for this to be implemented fully
at any scale facility. It is also worthy of note that databases such as the PDB and CSD
now facilitate and expose links, via DOIs, from records in their databases to raw
diffraction data, which aids the discoverability of raw data.
90
S. J. Coles et al.