correcting faulty data is also known in literature as fault detection and isolation
(FDI) [28].
Primarily, the goal of data validation lies in identifying and extracting the subset
of data which may be considered faulty (Fig. 4), i.e. not representing a valid
measurement of reality, due to a measurement or human error [10]. From the likely
faulty subset of data, some data represent occurrences of irregular/unexpected
processes in the system (i.e. pipe bursts, catastrophes, maintenance downtime
etc.). These data constitute a third type of human error that lies beyond the scope
of this study, as explained in Sect. 1.2. The focal point of this study lies, therefore, in
techniques that can be used to detect the subset of faulty data whose faultiness can be
explained and attributed to measurement or human errors, i.e. the first two types of
errors seen in Sect. 1.1. At the same time, the detection process has to ensure that
irregular but non-faulty data are not classified as faulty. For instance, outliers owing
to extreme events and even unprecedented events such as black swans [9] belong to
the valid data subgroup and should not be classified as faulty data.
As a core process in DQC, data validation is not a new concept and has been
developed heavily in DQC platforms [29], relying largely on algorithms and mathematical techniques of faulty data detection. However, automating the entire process
of data validation is not realistic [30], and expert judgment is still required to crossvalidate the results produced by mathematical methods.
2.3 Data Quality Control in the Context of Drinking Water
The concepts on DQC and validation described in §2.1 can be applied to any data
stream production environment [31], including course drinking water (DW). In that
case, data describing the status of the DW network (i.e. samples of physical variables
such as water quantity, quality, water level, pressure head, etc.) are acquired by
Fig. 4 The data set which is target of validation
72
M. Castro-Gama et al.
(FDI) [28].
Primarily, the goal of data validation lies in identifying and extracting the subset
of data which may be considered faulty (Fig. 4), i.e. not representing a valid
measurement of reality, due to a measurement or human error [10]. From the likely
faulty subset of data, some data represent occurrences of irregular/unexpected
processes in the system (i.e. pipe bursts, catastrophes, maintenance downtime
etc.). These data constitute a third type of human error that lies beyond the scope
of this study, as explained in Sect. 1.2. The focal point of this study lies, therefore, in
techniques that can be used to detect the subset of faulty data whose faultiness can be
explained and attributed to measurement or human errors, i.e. the first two types of
errors seen in Sect. 1.1. At the same time, the detection process has to ensure that
irregular but non-faulty data are not classified as faulty. For instance, outliers owing
to extreme events and even unprecedented events such as black swans [9] belong to
the valid data subgroup and should not be classified as faulty data.
As a core process in DQC, data validation is not a new concept and has been
developed heavily in DQC platforms [29], relying largely on algorithms and mathematical techniques of faulty data detection. However, automating the entire process
of data validation is not realistic [30], and expert judgment is still required to crossvalidate the results produced by mathematical methods.
2.3 Data Quality Control in the Context of Drinking Water
The concepts on DQC and validation described in §2.1 can be applied to any data
stream production environment [31], including course drinking water (DW). In that
case, data describing the status of the DW network (i.e. samples of physical variables
such as water quantity, quality, water level, pressure head, etc.) are acquired by
Fig. 4 The data set which is target of validation
72
M. Castro-Gama et al.