of faulty data of timestamps and consistency current DQC. Data from WQ corresponds a total of four time series and three different variables (i.e. temperature (1),
pH (1) and turbidity (2)), from their filtration processes.
Data collection corresponds to time series in the period between 1 January 2016
and 31 December 2017. After a visit to the facilities of Company A, it was verified
that it is also possible to fetch data directly from their data warehouse. Values
interpolated at different resolutions can be obtained. However, for the time being,
a resolution of 1 min for all variables was selected for further analysis, with the
exception of energy use which is provided by the energy provider at a 15 min
resolution.
Data was provided as CSV files. Data contains four fields (columns), (A) the
sensor ID, (B) the timestamp, (C) the measured value and (D) a status of signal’s
health, established by the system. Such pre-screening DQC is split in four different
categories as flags: (1) Good data, (2) Faulty data, (3) Dubious data and (4) Out of
range. It was not possible to determine the specific rules which drive the definition of
different categories as they are automatically triggered by the DQC system of
Company A.
4.3 Results Obtained
4.3.1 Water Quality Data at WWTP I and II
For each time series, the corresponding number of flags identified in the data is
presented in Table 3. It is evident that there are a limited number of flags identified
by the system. This is indeed a cumbersome task for Company A as to our
knowledge more than 73,000 variables are updated every minute by their system
only for drinking water.
Subsequently the proposed data validation has been applied. The confusion
matrices (Table 4) present the comparison between the observed faulty data in the
current data validation, and the ones identified by applying the proposed data
validation (KWR). If any timestamp sample is identified as faulty data by any of
the simple tests proposed, then data is considered faulty. There are 3 possibilities:
• When both DQC schemes agree in the identification a Yes-Yes coincidence is
identified. This can be understood as a validation of Company A’s validation. For
Table 3 Number of flags present in water quality data from Company A (% ¼ percentage of total
number of timestamps)
Treatment plant
Acronym
pH
Temperature
Turbidity
(À)
%
(C)
%
(%)
%
WWTP I
L01
25
0.00
33
0.00
30
0.00
WWTP II
L02
N/A
–
N/A
–
67
0.01
N/A not available
86
M. Castro-Gama et al.
pH (1) and turbidity (2)), from their filtration processes.
Data collection corresponds to time series in the period between 1 January 2016
and 31 December 2017. After a visit to the facilities of Company A, it was verified
that it is also possible to fetch data directly from their data warehouse. Values
interpolated at different resolutions can be obtained. However, for the time being,
a resolution of 1 min for all variables was selected for further analysis, with the
exception of energy use which is provided by the energy provider at a 15 min
resolution.
Data was provided as CSV files. Data contains four fields (columns), (A) the
sensor ID, (B) the timestamp, (C) the measured value and (D) a status of signal’s
health, established by the system. Such pre-screening DQC is split in four different
categories as flags: (1) Good data, (2) Faulty data, (3) Dubious data and (4) Out of
range. It was not possible to determine the specific rules which drive the definition of
different categories as they are automatically triggered by the DQC system of
Company A.
4.3 Results Obtained
4.3.1 Water Quality Data at WWTP I and II
For each time series, the corresponding number of flags identified in the data is
presented in Table 3. It is evident that there are a limited number of flags identified
by the system. This is indeed a cumbersome task for Company A as to our
knowledge more than 73,000 variables are updated every minute by their system
only for drinking water.
Subsequently the proposed data validation has been applied. The confusion
matrices (Table 4) present the comparison between the observed faulty data in the
current data validation, and the ones identified by applying the proposed data
validation (KWR). If any timestamp sample is identified as faulty data by any of
the simple tests proposed, then data is considered faulty. There are 3 possibilities:
• When both DQC schemes agree in the identification a Yes-Yes coincidence is
identified. This can be understood as a validation of Company A’s validation. For
Table 3 Number of flags present in water quality data from Company A (% ¼ percentage of total
number of timestamps)
Treatment plant
Acronym
pH
Temperature
Turbidity
(À)
%
(C)
%
(%)
%
WWTP I
L01
25
0.00
33
0.00
30
0.00
WWTP II
L02
N/A
–
N/A
–
67
0.01
N/A not available
86
M. Castro-Gama et al.