now, data validation for Company A is already cumbersome, so the possibility of
performing such task for more than 73.000 variables seems impossible to come to
reality. A calibration process of the parameters for each of the variables is required to
be performed on individual basis. In the second case, it is possible that the proposed
data validation has indeed identified additional faulty data. Although this may sound
controversial, some examples are presented to discuss the reliability of DQC of
Company A.
Figure 7 presents the time series for pH at WWTP I. The range of variability of
pH is quite small due to the need by Company A to keep its magnitude within a
narrow band. However, there are some spikes present in the data which are identified
both by Company A’s system and the proposed data validation.
In Fig. 8, time series of temperature in WWTP I is presented. In this case most of
Company A’s system flags are captured by the simple data validation proposed, and
indeed three time windows in which the possibility of faulty data were identified are
presented. Such time windows are centred in 2016 around April 18th, May 22nd and
December 29th.
The case of turbidity is presented in Fig. 9 (in logarithmic scale). Here the most
relevant feature for data validation is that the time series presents jumps at different
periods. Such jumps (drifts or changes in variance) occur during 2016, around July
1st and in 2017, around 3rd of April and 9th of August. This can be due to a
modification in the operational conditions of the treatment plant, given that data
corresponds to the filtration system of the treatment plants. Without further information it was not possible to elaborate a hypothesis on this change of behaviour.
A duplicate analysis for the same variable, this time at WWTP II, (see Fig. 10,
vertical axis in logarithmic scale) shows that Company A’s flag system tends to allow
higher values of turbidity as normal events. An example is the spike in 2016, during
April 1st. This could have an operational reasoning; however this behaviour is not
identified in the logbooks provided by the utility.
On the other hand, there are some time windows in which the simple tests
identified plateau values registered in the raw data, while Company A’s system
was not able to do so (see Fig. 10). Such time windows are identified in 2016 around
December 22nd and in 2017 around September 26th.
It was possible to identify most of the faulty data, without previous knowledge of
the system rules. However, in some cases with the simple tests, some additional
“likely” faulty data was identified among the time series. This does not mean that the
statuses provided by Company A are not reliable enough, rather than one of the
detection rules presented here (i.e. flat value detection) may not be currently
implemented inside their data warehouse.
4.3.2 Pumping Stations Data
There are a total of five pumping stations (PS) in City A system indicated with the
following abbreviations: WPK, AVW, HLW, OSD and HLM (see Fig. 11).
88
M. Castro-Gama et al.
performing such task for more than 73.000 variables seems impossible to come to
reality. A calibration process of the parameters for each of the variables is required to
be performed on individual basis. In the second case, it is possible that the proposed
data validation has indeed identified additional faulty data. Although this may sound
controversial, some examples are presented to discuss the reliability of DQC of
Company A.
Figure 7 presents the time series for pH at WWTP I. The range of variability of
pH is quite small due to the need by Company A to keep its magnitude within a
narrow band. However, there are some spikes present in the data which are identified
both by Company A’s system and the proposed data validation.
In Fig. 8, time series of temperature in WWTP I is presented. In this case most of
Company A’s system flags are captured by the simple data validation proposed, and
indeed three time windows in which the possibility of faulty data were identified are
presented. Such time windows are centred in 2016 around April 18th, May 22nd and
December 29th.
The case of turbidity is presented in Fig. 9 (in logarithmic scale). Here the most
relevant feature for data validation is that the time series presents jumps at different
periods. Such jumps (drifts or changes in variance) occur during 2016, around July
1st and in 2017, around 3rd of April and 9th of August. This can be due to a
modification in the operational conditions of the treatment plant, given that data
corresponds to the filtration system of the treatment plants. Without further information it was not possible to elaborate a hypothesis on this change of behaviour.
A duplicate analysis for the same variable, this time at WWTP II, (see Fig. 10,
vertical axis in logarithmic scale) shows that Company A’s flag system tends to allow
higher values of turbidity as normal events. An example is the spike in 2016, during
April 1st. This could have an operational reasoning; however this behaviour is not
identified in the logbooks provided by the utility.
On the other hand, there are some time windows in which the simple tests
identified plateau values registered in the raw data, while Company A’s system
was not able to do so (see Fig. 10). Such time windows are identified in 2016 around
December 22nd and in 2017 around September 26th.
It was possible to identify most of the faulty data, without previous knowledge of
the system rules. However, in some cases with the simple tests, some additional
“likely” faulty data was identified among the time series. This does not mean that the
statuses provided by Company A are not reliable enough, rather than one of the
detection rules presented here (i.e. flat value detection) may not be currently
implemented inside their data warehouse.
4.3.2 Pumping Stations Data
There are a total of five pumping stations (PS) in City A system indicated with the
following abbreviations: WPK, AVW, HLW, OSD and HLM (see Fig. 11).
88
M. Castro-Gama et al.