6.2 General Conclusions Regarding Data Quality Control
Data quality control is a continuous process, instrumental in achieving large strategic
objectives such as reliable and efficient drinking water system operations. Data
validation for water companies is not an exclusive task of a data scientist. It is a
collaboration between operators who understand the system and data analysts who
must validate large proportions of data to improve models and, as a consequence,
decisions. Currently only a small percentage of the data is validated, but we suggest
that in the near future, there is a need to become even more active in data management at every level of a water company.
All involved water utilities implement individually data validation at different
levels of complexity. Although water companies face similar issues, several customized tools/software are being developed per company. This is because each
company has its own registration and database system, as well as different specifications regarding time steps, units, storage, etc. Working together on specific
guidelines (standards) for the sector to define which datasets and methodologies
are used for validation can facilitate and speed up implementation of DQC systems
and be useful for potential future exchange of data, and, it may facilitate auditing
operations as a nationwide goal.
From the analysed cases it was concluded that there is a need to exchange data
and develop proper data models that consider not only the raw data but also
metadata, formats and characteristics of the platform.
There are several techniques to validate the data. Regarding DQC, there is no
such thing as one technique that fits all. Depending on the monitored event/variable,
different techniques with different parameters should be applied also according the
objective of the validation. Best practices and challenging issues have been identified (see again Sect. 4.3). To overcome the identified challenges a two-way implementation of DQC procedures is needed:
1. Strategic (top-down) by developing frameworks and standards for the water
sector which are compatible with standards of other sectors
2. Operational (bottom-up) by implementing cases, evaluating case studies and
sharing experiences across utilities.
To progress both, it is envisioned that utilities can start with simple cases and
techniques such as the ones presented in this chapter and steadily scale-up as the
needs and goals of the utilities are met.
References
1. Bertrand-Krajewski J, Bardin J, Mourad M, Beranger Y (2003) Accounting for sensor calibration, data validation, measurement and sampling uncertainties in monitoring urban drainage
systems. Water Sci Technol 47:95–102
106
M. Castro-Gama et al.
Data quality control is a continuous process, instrumental in achieving large strategic
objectives such as reliable and efficient drinking water system operations. Data
validation for water companies is not an exclusive task of a data scientist. It is a
collaboration between operators who understand the system and data analysts who
must validate large proportions of data to improve models and, as a consequence,
decisions. Currently only a small percentage of the data is validated, but we suggest
that in the near future, there is a need to become even more active in data management at every level of a water company.
All involved water utilities implement individually data validation at different
levels of complexity. Although water companies face similar issues, several customized tools/software are being developed per company. This is because each
company has its own registration and database system, as well as different specifications regarding time steps, units, storage, etc. Working together on specific
guidelines (standards) for the sector to define which datasets and methodologies
are used for validation can facilitate and speed up implementation of DQC systems
and be useful for potential future exchange of data, and, it may facilitate auditing
operations as a nationwide goal.
From the analysed cases it was concluded that there is a need to exchange data
and develop proper data models that consider not only the raw data but also
metadata, formats and characteristics of the platform.
There are several techniques to validate the data. Regarding DQC, there is no
such thing as one technique that fits all. Depending on the monitored event/variable,
different techniques with different parameters should be applied also according the
objective of the validation. Best practices and challenging issues have been identified (see again Sect. 4.3). To overcome the identified challenges a two-way implementation of DQC procedures is needed:
1. Strategic (top-down) by developing frameworks and standards for the water
sector which are compatible with standards of other sectors
2. Operational (bottom-up) by implementing cases, evaluating case studies and
sharing experiences across utilities.
To progress both, it is envisioned that utilities can start with simple cases and
techniques such as the ones presented in this chapter and steadily scale-up as the
needs and goals of the utilities are met.
References
1. Bertrand-Krajewski J, Bardin J, Mourad M, Beranger Y (2003) Accounting for sensor calibration, data validation, measurement and sampling uncertainties in monitoring urban drainage
systems. Water Sci Technol 47:95–102
106
M. Castro-Gama et al.