Water Analytics and Management with Real-time Linked Dataspaces
11
6 Realization of Waternomics Platform
In this section, we provide a concrete realization of the Real-time Linked Dataspace
using the tools and techniques discussed in previous sections. We have implemented
the dataspace, for the Waternomics project, as a realization of the Lambda architecture. The Lambda architecture was introduced with the aim of allowing seamless
ingestion and processing of streaming events data [31]. It consists of three layers:
the batch layer deals with processing of large quantities of historical data, the speed
layer processes real-time data to minimize latency, and the serving layer provides
combined query access to data from other two layers. Our implementation departs
from the original Lambda architecture due to the central role of catalog service in the
implementation of the batch, speed, and serving layers. The support services in the
dataspace are mainly implemented through customization of following open source
software: Druid, Apache Spark, MySQL, Apache Kafka, and Apache Cassandra.
Fig. 2 Lambda architecture realization of the Real-time Linked Dataspace for Waternomics project
Figure 2 shows the data from a building management system (BMS) and water
sensors in the Galway pilot being processed in the dataspace. All data sources and
entities are defined in the catalog (WKAN). The batch layer is implemented using
Spark SQL when historical data from BMS is fed into the indexer node of Druid.
Real-time data from sensors is fed into the Kafka message broker, which provides
a high availability integration point for speed layer data from the different pilots.
Real-time data from Kafka is processed through Spark Streaming to a real-time
node of Druid. The combined code from Spark Streaming and Spark SQL provides
a standardized way of generating dimensional data that is served using the Druid
cluster. The Druid nodes use Cassandra as deep storage for historical data. The
batch data is made available through the historical node and streaming data is made
11
6 Realization of Waternomics Platform
In this section, we provide a concrete realization of the Real-time Linked Dataspace
using the tools and techniques discussed in previous sections. We have implemented
the dataspace, for the Waternomics project, as a realization of the Lambda architecture. The Lambda architecture was introduced with the aim of allowing seamless
ingestion and processing of streaming events data [31]. It consists of three layers:
the batch layer deals with processing of large quantities of historical data, the speed
layer processes real-time data to minimize latency, and the serving layer provides
combined query access to data from other two layers. Our implementation departs
from the original Lambda architecture due to the central role of catalog service in the
implementation of the batch, speed, and serving layers. The support services in the
dataspace are mainly implemented through customization of following open source
software: Druid, Apache Spark, MySQL, Apache Kafka, and Apache Cassandra.
Fig. 2 Lambda architecture realization of the Real-time Linked Dataspace for Waternomics project
Figure 2 shows the data from a building management system (BMS) and water
sensors in the Galway pilot being processed in the dataspace. All data sources and
entities are defined in the catalog (WKAN). The batch layer is implemented using
Spark SQL when historical data from BMS is fed into the indexer node of Druid.
Real-time data from sensors is fed into the Kafka message broker, which provides
a high availability integration point for speed layer data from the different pilots.
Real-time data from Kafka is processed through Spark Streaming to a real-time
node of Druid. The combined code from Spark Streaming and Spark SQL provides
a standardized way of generating dimensional data that is served using the Druid
cluster. The Druid nodes use Cassandra as deep storage for historical data. The
batch data is made available through the historical node and streaming data is made