4 Architecting IoT Cloud
191
Data
Stream
processing
Real-time
view
Real-time
view
Query
Real-time layer
Serving layer
Fig. 4.10 Kappa architecture
4.5.1.2 Kappa Architecture
Kappa architecture was first designed by Jay Kreps. This architecture is focused on
using stream processing, but it is not meant to replace Lambda architecture unless it
truly fits your use case. In this architecture solution, all IoT data are routed through a
real-time layer and results are moved to the serving layer where they can be queried
(see Fig. 4.10). In other words, there is no batch layer in Kappa architecture [13].
The key idea behind Kappa architecture is to address real-time data processing
by constantly reprocessing data through a single stream processing engine. This
implies that the incoming data can be quickly replayed. For example, in the instance
that there are code changes, a second stream process would then replay data through
the real-time engine and then replace the previous views in the serving layer data.
Kappa architecture achieves simplicity compared to Lambda architecture by
maintaining only one code instead of trying to manage two codes (i.e., one for
batch layer and one for speed layer). Additionally, queries do not have to look in
batch or speed views. Instead, queries look in only one serving location. Drawbacks
to this architecture are centered on processing data in a stream. It is easier to
handle duplicate events, cross-reference events, or manage order operations in batch
processing. In addition, having access to the whole data set in the batch layer
can lead to better optimizations, higher performance, and simple algorithms in
Lambda architecture. Finally, in some sophisticated situations, streaming and batch
algorithms can lead to different results.
4.5.2 Data Processing Frameworks
This section will overview the well-known frameworks designed for data processing.
4.5.2.1 Apache Storm
Apache Storm is a distributed, open-source stream processing framework known for
its reliability and fault tolerance. Storm applications are created as a “topology” in
191
Data
Stream
processing
Real-time
view
Real-time
view
Query
Real-time layer
Serving layer
Fig. 4.10 Kappa architecture
4.5.1.2 Kappa Architecture
Kappa architecture was first designed by Jay Kreps. This architecture is focused on
using stream processing, but it is not meant to replace Lambda architecture unless it
truly fits your use case. In this architecture solution, all IoT data are routed through a
real-time layer and results are moved to the serving layer where they can be queried
(see Fig. 4.10). In other words, there is no batch layer in Kappa architecture [13].
The key idea behind Kappa architecture is to address real-time data processing
by constantly reprocessing data through a single stream processing engine. This
implies that the incoming data can be quickly replayed. For example, in the instance
that there are code changes, a second stream process would then replay data through
the real-time engine and then replace the previous views in the serving layer data.
Kappa architecture achieves simplicity compared to Lambda architecture by
maintaining only one code instead of trying to manage two codes (i.e., one for
batch layer and one for speed layer). Additionally, queries do not have to look in
batch or speed views. Instead, queries look in only one serving location. Drawbacks
to this architecture are centered on processing data in a stream. It is easier to
handle duplicate events, cross-reference events, or manage order operations in batch
processing. In addition, having access to the whole data set in the batch layer
can lead to better optimizations, higher performance, and simple algorithms in
Lambda architecture. Finally, in some sophisticated situations, streaming and batch
algorithms can lead to different results.
4.5.2 Data Processing Frameworks
This section will overview the well-known frameworks designed for data processing.
4.5.2.1 Apache Storm
Apache Storm is a distributed, open-source stream processing framework known for
its reliability and fault tolerance. Storm applications are created as a “topology” in