– The way data is represented (i.e. real numbers as pressures and flows, binary as
pump switches, categorical as status of data provided by most systems),
– Data resolution (i.e. both temporal or spatial)
– The method/technique used for validation (see Sect. 3 “Literature Review
on Faulty Data Detection Techniques for Water Utilities”)
A single technique cannot be used for all instances.
The time spent between techniques can vary between pre- and postprocessing.
– The metrics used (i.e. some variables such as pH, temperature and turbidity are
based on a sensor calibration made through laboratory tests)
• The user and objective
– Data may be used for real-time control (RTC) or offline historical analysis.
– Some methods may be used by data warehouse administrators, while for water
accounts managers as end users only performance indicators or data aggregation as post-process are relevant.
2.4 Current Implementation of Data Validation Techniques
by the Drinking Water Companies
Drinking water companies own and manage extensive systems (with several facilities), which are continually monitored at different points, e.g. production, transport
and distribution. For example, Company A has approximately 73,000 variables in
total, measured every second. Currently every company is dealing with data quality
issues. Due to the exponential growth of data and the specific characteristics of each
variable, these cannot be easily manually validated.
Additionally, time series (TS) are becoming increasingly necessary for modelling, such us hydraulic, risks and decision models.
Other emerging drivers are stricter laws and regulations for the definition of
standardization of data models and protocols. For example, the European INSPIRE
directive
4 defines the technical guidelines for data specification of Infrastructure and
its spatial information. Such initiative is currently an invitation for standardization
moving forward (which may facilitate exchange of information), rather than a
mandatory application for future implementations for the drinking water utilities.
Despite these drivers, there are still several barriers to validate the data. The
volume of real-time information has become so extensive that validation of all the
variables by a human becomes unrealistic. To deal with it, in some cases data
4 Commission Regulation (EU) No 1312/2014 of 10 December 2014 amending Regulation (EU) No
1089/2010 implementing Directive 2007/2/EC of the European Parliament and of the Council as
regards interoperability of spatial data services
74
M. Castro-Gama et al.
pump switches, categorical as status of data provided by most systems),
– Data resolution (i.e. both temporal or spatial)
– The method/technique used for validation (see Sect. 3 “Literature Review
on Faulty Data Detection Techniques for Water Utilities”)
A single technique cannot be used for all instances.
The time spent between techniques can vary between pre- and postprocessing.
– The metrics used (i.e. some variables such as pH, temperature and turbidity are
based on a sensor calibration made through laboratory tests)
• The user and objective
– Data may be used for real-time control (RTC) or offline historical analysis.
– Some methods may be used by data warehouse administrators, while for water
accounts managers as end users only performance indicators or data aggregation as post-process are relevant.
2.4 Current Implementation of Data Validation Techniques
by the Drinking Water Companies
Drinking water companies own and manage extensive systems (with several facilities), which are continually monitored at different points, e.g. production, transport
and distribution. For example, Company A has approximately 73,000 variables in
total, measured every second. Currently every company is dealing with data quality
issues. Due to the exponential growth of data and the specific characteristics of each
variable, these cannot be easily manually validated.
Additionally, time series (TS) are becoming increasingly necessary for modelling, such us hydraulic, risks and decision models.
Other emerging drivers are stricter laws and regulations for the definition of
standardization of data models and protocols. For example, the European INSPIRE
directive
4 defines the technical guidelines for data specification of Infrastructure and
its spatial information. Such initiative is currently an invitation for standardization
moving forward (which may facilitate exchange of information), rather than a
mandatory application for future implementations for the drinking water utilities.
Despite these drivers, there are still several barriers to validate the data. The
volume of real-time information has become so extensive that validation of all the
variables by a human becomes unrealistic. To deal with it, in some cases data
4 Commission Regulation (EU) No 1312/2014 of 10 December 2014 amending Regulation (EU) No
1089/2010 implementing Directive 2007/2/EC of the European Parliament and of the Council as
regards interoperability of spatial data services
74
M. Castro-Gama et al.