2.5 Data Validation: Experiences of a Front Runner –
Company D
One of the drinking water company, which is identified as one of the front runners in
relation to automatization of routines for data validation, collects a lot of measurement data in PI (from OsiSoft), but still it only validates just a small percentage of all
the data.
Company D has a system to validate water volume flows, and it validates the
daily volume flow at measurement points on the boundaries of the DMAs (about
150 locations per day). The validation consists of checking if the difference between
meter readings at the beginning and end of the day is equal to the sum of the
analogue readings during the day. If that does not turn out to be correct, the user
of the system is assisted in correcting the daily quantity for instance by showing
typical values/ranges for this type of day and the historical values of the last 7 or
14 days. Validating always consists of two steps: (1) check whether the data to be
validated is plausible; if not, (2) correct the data. Individual large customers
(>10 Â 10
3 k m
3 /y, approx. 600 units) are validated on a monthly basis. Meter
readings of each month are compared with the previous month. It is also checked
whether the difference between both meter readings is equal to the sum of hourly
values (which are collected to determine peak rates).
Within the data validation process, the responsibilities are well defined: An
employee (from the control centre) validates the measurement points in the net
and checks that the validation actions are carried out by production sites (if they
appear to be necessary). An employee from the industrial water department validates
large customers (>100,000 m
3 /y) and all the industrial water customers. An
employee of the customer contact centre validates customers with drinking water
consumption between 10,000 and 100,000 m
3 /y. The operators on site validate the
outgoing flows of production sites, plus the waste water flows and the incoming
water flows. The validation rules are also reported. Despite some companies having
automation for data validation, this task still represents a lot of work and needs
constant attention. It has become increasingly clear that not only the quality of the
sensor and the data logger but also a good interface to PI are very important aspects
within the process to validate the data.
3 Literature Review on Faulty Data Detection Techniques
for Water Utilities
3.1 Background
The detection of anomalies corresponds to the first line of defence against faulty
data, as it allows near real-time identification of individual values or sets of values
76
M. Castro-Gama et al.
Company D
One of the drinking water company, which is identified as one of the front runners in
relation to automatization of routines for data validation, collects a lot of measurement data in PI (from OsiSoft), but still it only validates just a small percentage of all
the data.
Company D has a system to validate water volume flows, and it validates the
daily volume flow at measurement points on the boundaries of the DMAs (about
150 locations per day). The validation consists of checking if the difference between
meter readings at the beginning and end of the day is equal to the sum of the
analogue readings during the day. If that does not turn out to be correct, the user
of the system is assisted in correcting the daily quantity for instance by showing
typical values/ranges for this type of day and the historical values of the last 7 or
14 days. Validating always consists of two steps: (1) check whether the data to be
validated is plausible; if not, (2) correct the data. Individual large customers
(>10 Â 10
3 k m
3 /y, approx. 600 units) are validated on a monthly basis. Meter
readings of each month are compared with the previous month. It is also checked
whether the difference between both meter readings is equal to the sum of hourly
values (which are collected to determine peak rates).
Within the data validation process, the responsibilities are well defined: An
employee (from the control centre) validates the measurement points in the net
and checks that the validation actions are carried out by production sites (if they
appear to be necessary). An employee from the industrial water department validates
large customers (>100,000 m
3 /y) and all the industrial water customers. An
employee of the customer contact centre validates customers with drinking water
consumption between 10,000 and 100,000 m
3 /y. The operators on site validate the
outgoing flows of production sites, plus the waste water flows and the incoming
water flows. The validation rules are also reported. Despite some companies having
automation for data validation, this task still represents a lot of work and needs
constant attention. It has become increasingly clear that not only the quality of the
sensor and the data logger but also a good interface to PI are very important aspects
within the process to validate the data.
3 Literature Review on Faulty Data Detection Techniques
for Water Utilities
3.1 Background
The detection of anomalies corresponds to the first line of defence against faulty
data, as it allows near real-time identification of individual values or sets of values
76
M. Castro-Gama et al.