applying data validation techniques corresponds to the use of modelling tools to
identify anomalies (see again Fig. 5). Several methodologies are available which
have not been implemented in this work, ranging from statistical, surrogate and
physically based models with the aim to properly define when a sample of data
corresponds to an anomaly or not. Having a calibrated model of a WDN presents the
possibility to estimate deviations between measured data and simulated data and as
such detect anomalies.
6 Conclusions
6.1 Specific Conclusions Based on the Cases Regarding
Faulty Data Detection
Data validation applied to water quality demonstrated the validity of the utility’s flag
system, as it was able to identify most anomalies. In general, data of Company A is
of good quality with a low percentage of flags issued by the system as faulty data.
The implementation of a proposed data validation in this chapter using simple
validation rules showed that the inclusion of a flat value test can improve the current
data validation procedure of the drinking water companies for water quality data.
Additionally for specific datasets such as turbidity time series, changes in the system
can be identified by using jump detection (i.e. drifts or changes in average). Further
development of the proposed data validation is the inclusion of more complex
detection techniques.
From the validation of data for water balance, it can be concluded that the
temporal data resolution is sufficient; however the process of data collection and
aggregation is quite demanding. If a water balance needs to be performed at different
intervals (e.g. 1 day, 1 week, 1 month, 1 year), the tasks of data validation would
require extensive searches from diverse sources to confirm information from logbooks, installation and maintenance records. This is highly time-consuming and it
can be improved by automation by routines.
Expert knowledge from the utility helped to clarify the validity of large collections of data from the last 2 years, i.e. the pumping station maintenance in HLM. The
integration of such expert knowledge in the system has yet to be implemented. This
is evidenced in the fact that most queries of additional data validation were solved by
internal communication ‘via-via’ and not through a complete record of operations in
any database.
A Bird’s-Eye View of Data Validation in the Drinking Water Industry of the. . .
105
identify anomalies (see again Fig. 5). Several methodologies are available which
have not been implemented in this work, ranging from statistical, surrogate and
physically based models with the aim to properly define when a sample of data
corresponds to an anomaly or not. Having a calibrated model of a WDN presents the
possibility to estimate deviations between measured data and simulated data and as
such detect anomalies.
6 Conclusions
6.1 Specific Conclusions Based on the Cases Regarding
Faulty Data Detection
Data validation applied to water quality demonstrated the validity of the utility’s flag
system, as it was able to identify most anomalies. In general, data of Company A is
of good quality with a low percentage of flags issued by the system as faulty data.
The implementation of a proposed data validation in this chapter using simple
validation rules showed that the inclusion of a flat value test can improve the current
data validation procedure of the drinking water companies for water quality data.
Additionally for specific datasets such as turbidity time series, changes in the system
can be identified by using jump detection (i.e. drifts or changes in average). Further
development of the proposed data validation is the inclusion of more complex
detection techniques.
From the validation of data for water balance, it can be concluded that the
temporal data resolution is sufficient; however the process of data collection and
aggregation is quite demanding. If a water balance needs to be performed at different
intervals (e.g. 1 day, 1 week, 1 month, 1 year), the tasks of data validation would
require extensive searches from diverse sources to confirm information from logbooks, installation and maintenance records. This is highly time-consuming and it
can be improved by automation by routines.
Expert knowledge from the utility helped to clarify the validity of large collections of data from the last 2 years, i.e. the pumping station maintenance in HLM. The
integration of such expert knowledge in the system has yet to be implemented. This
is evidenced in the fact that most queries of additional data validation were solved by
internal communication ‘via-via’ and not through a complete record of operations in
any database.
A Bird’s-Eye View of Data Validation in the Drinking Water Industry of the. . .
105