4 Signal Treatment
A water sample, whether natural water, drinking water or wastewater, will contain a
wide range of substances in varying concentrations. The signals from all these
chemicals are recorded simultaneously and in many cases overlap. In spectroscopy
employed for smart water systems, the in situ and real-time nature of the monitoring
requires that a sensor automatically extracts relevant information on the composition
of the water from the superposed spectra. Effects that interfere with the signal, and
therefore must be taken into account when analysing spectral data, include the
absorption and scattering of light by particles and/or air bubbles, wear and tear of
optical surfaces of the sensor (scratches, fouling, scaling), variations between measurements due to slight changes in the spectrum and intensity of the light source and
variations in sensitivity as well as noise in the detector.
4.1 Data Validation
To turn raw data into useful information, modern spectrometers make use of
mathematical algorithms. These clean up the signal and use correlations between
the light intensity at various wavelengths and analytical parameters to calculate
concentrations of specific (groups) of chemicals. Whether spectrometric data is
used for the development of an automatic detection algorithm or for real-time
autonomous data interpretation by the sensor system, the first step in analysis is
always data validation. Because of the amount of data generated, and because of the
fact that real-time process control requires making (near) real-time decisions, it is not
possible to manually verify whether data are reliable and valid. For smart water
systems, automatic validation is critical, ensuring only high-quality measurement
results are used in the automated decision-making processes.
Data validation checks whether the sensor system was working properly during
the measurement and whether the sample analysed was representative of the medium
being monitored. If results are valid and representative, the data is considered
reliable. Typical components in the data validation process include sensor status
checks, noise analysis and detection of outliers, drift, gaps and steps in data. Tests to
validate correct sensor operation include checks against realistic range, detection of
constant values and signal-gradient monitoring [5] as well as hardware and software
error messages. Once identified, anomalous or missing sensor data might be
corrected for, e.g. by interpolation, data smoothing and averaging. More advanced
validation methods include data forecasting [6] and the use of distributed algorithms
in sensor networks where the results for one particular sensor can be inferred from
those of its neighbours [7].
For spectral data averaging is a widely applied method: a set of subsequently
recorded spectra is combined to average out fluctuations in the instrument and
Spectroscopic Methods for Online Water Quality Monitoring
289
A water sample, whether natural water, drinking water or wastewater, will contain a
wide range of substances in varying concentrations. The signals from all these
chemicals are recorded simultaneously and in many cases overlap. In spectroscopy
employed for smart water systems, the in situ and real-time nature of the monitoring
requires that a sensor automatically extracts relevant information on the composition
of the water from the superposed spectra. Effects that interfere with the signal, and
therefore must be taken into account when analysing spectral data, include the
absorption and scattering of light by particles and/or air bubbles, wear and tear of
optical surfaces of the sensor (scratches, fouling, scaling), variations between measurements due to slight changes in the spectrum and intensity of the light source and
variations in sensitivity as well as noise in the detector.
4.1 Data Validation
To turn raw data into useful information, modern spectrometers make use of
mathematical algorithms. These clean up the signal and use correlations between
the light intensity at various wavelengths and analytical parameters to calculate
concentrations of specific (groups) of chemicals. Whether spectrometric data is
used for the development of an automatic detection algorithm or for real-time
autonomous data interpretation by the sensor system, the first step in analysis is
always data validation. Because of the amount of data generated, and because of the
fact that real-time process control requires making (near) real-time decisions, it is not
possible to manually verify whether data are reliable and valid. For smart water
systems, automatic validation is critical, ensuring only high-quality measurement
results are used in the automated decision-making processes.
Data validation checks whether the sensor system was working properly during
the measurement and whether the sample analysed was representative of the medium
being monitored. If results are valid and representative, the data is considered
reliable. Typical components in the data validation process include sensor status
checks, noise analysis and detection of outliers, drift, gaps and steps in data. Tests to
validate correct sensor operation include checks against realistic range, detection of
constant values and signal-gradient monitoring [5] as well as hardware and software
error messages. Once identified, anomalous or missing sensor data might be
corrected for, e.g. by interpolation, data smoothing and averaging. More advanced
validation methods include data forecasting [6] and the use of distributed algorithms
in sensor networks where the results for one particular sensor can be inferred from
those of its neighbours [7].
For spectral data averaging is a widely applied method: a set of subsequently
recorded spectra is combined to average out fluctuations in the instrument and
Spectroscopic Methods for Online Water Quality Monitoring
289