the companies. Additionally, there is a lack of knowledge about how data is
validated by third parties, e.g. energy companies.
Among the utilities the purpose of validation is very diverse, for example, water
flow, billing, determining the water balance, identifying leaks and changes in
turbidity due to new filter installation. Depending on the objective the requirements
of the validation change. The following issues have been identified that hinder the
potential for implementation of more complex/advanced data validation techniques:
• Known deviations and operations are usually poorly logged by the utilities, or
when this is done, it is very limited to some variables across utilities.
• Lack of specialized manpower to perform this task on a regular basis: a team
consisting of both data scientists and hydraulic engineers is required.
• Specific techniques for data validation are still hard to be adopted because there is
no overview regarding which data are needed for each of them.
Table 7 Issues found during this interviews
Issues
Company
A B C D
Data storage integration. Different databases or a single database. If multiple
DB, then sometimes data is not linked one to one
x x
Lack of overview of metadata: Difficulties to track additional information,
e.g. log books of maintenance, data is still in different databases stored
x x x
Lack of priorities/time to (at least) tag the large number of signals and known
events
x x
Problems related to the interfaces
X
Current approaches are often somewhat ad hoc
x x
A lot of techniques, but still data validation/correction is largely based on expert
opinions
x
X
Own customize system, data models and tools (not compatible with other
companies)
x
x X
Only a small percentage of the data is validated
x x x X
Vulnerability of failure of servers, data from third parties
x x x
Black box in the built-in tools. Automatic filtering of suspicious data and not
clear which rules they use to validate the data
x
x
Table 6 Best practices identified during the interviews
Best practices
Company
A B C D
Data scientist works together with a domain expert
X
Clear responsibilities
x
X
Validation rules reported
X
Implementation of automatized routines which allows continuo validation of
some datasets
x
x X
Validation of aggregated data
X
Implementing pilot projects to learn from it
x
100
M. Castro-Gama et al.
validated by third parties, e.g. energy companies.
Among the utilities the purpose of validation is very diverse, for example, water
flow, billing, determining the water balance, identifying leaks and changes in
turbidity due to new filter installation. Depending on the objective the requirements
of the validation change. The following issues have been identified that hinder the
potential for implementation of more complex/advanced data validation techniques:
• Known deviations and operations are usually poorly logged by the utilities, or
when this is done, it is very limited to some variables across utilities.
• Lack of specialized manpower to perform this task on a regular basis: a team
consisting of both data scientists and hydraulic engineers is required.
• Specific techniques for data validation are still hard to be adopted because there is
no overview regarding which data are needed for each of them.
Table 7 Issues found during this interviews
Issues
Company
A B C D
Data storage integration. Different databases or a single database. If multiple
DB, then sometimes data is not linked one to one
x x
Lack of overview of metadata: Difficulties to track additional information,
e.g. log books of maintenance, data is still in different databases stored
x x x
Lack of priorities/time to (at least) tag the large number of signals and known
events
x x
Problems related to the interfaces
X
Current approaches are often somewhat ad hoc
x x
A lot of techniques, but still data validation/correction is largely based on expert
opinions
x
X
Own customize system, data models and tools (not compatible with other
companies)
x
x X
Only a small percentage of the data is validated
x x x X
Vulnerability of failure of servers, data from third parties
x x x
Black box in the built-in tools. Automatic filtering of suspicious data and not
clear which rules they use to validate the data
x
x
Table 6 Best practices identified during the interviews
Best practices
Company
A B C D
Data scientist works together with a domain expert
X
Clear responsibilities
x
X
Validation rules reported
X
Implementation of automatized routines which allows continuo validation of
some datasets
x
x X
Validation of aggregated data
X
Implementing pilot projects to learn from it
x
100
M. Castro-Gama et al.