and data quality constraints. For instance, tanks have a limited capacity, and
valid water level values have to be below that; likewise, pressure values in a
network are bounded by physical (and pipe strength) constraints. These tests can
be set by setting appropriate thresholds on data before storage to the database or
data warehouses, based on expert judgment, e.g. from both system operators and
data managers. Thresholds can be also based on historical operational values, for
instance, when deriving the operational temperature range of a wastewater
treatment tank.
(d) Jump or leap detection (change of variance), where faulty data are identified
based on leaps or jumps in signal data. As part of these techniques, any subsets
that deviate from a general trend or (seasonal or diurnal) pattern are flagged as
Fig. 6 Examples of simple testing for data validation, with measured samples in red, boundaries as
red lines, and marks of faulty data in cyan blue. Faulty data 1, no flag 0. Panel (a) flat line detection,
panel (b) min-max boundary testing, panel (c) jump-leap detection
A Bird’s-Eye View of Data Validation in the Drinking Water Industry of the. . .
79