4 Architecting IoT Cloud
189
• Data in Motion – Data collection is similar to that of data at rest; however, data
analysis happens in real time as the data-creating event occurs. For example,
a connected wristband can constantly collect and record guest activity data in
an event/conference/fair to customize the guest’s visit with activity suggestions
based on individual behavior, enabling personalized user experience in real time.
4.5.1 Data Processing Architectures
In this section we overview two well-known architectures for data processing,
namely, Lambda architecture and Kappa architecture.
4.5.1.1 Lambda Architecture
Nathan Marz and James Warren first proposed Lambda architecture in “Big Data:
Principles and best practices of scalable real-time data systems.” Lambda was
designed as a universal, fault-tolerant, scalable data processing architecture able to
process massive amounts of data using stream processing and batch processing techniques. Figure 4.9 illustrates the three main elements found in Lambda architecture:
the speed/real-time layer, batch layer, and serving layer [12].
• Batch Layer – Responsible for storing raw data and processing data in batch
model to create batch views. The data scope in the batch layer can encompass
hours to years.
IP
devices
Edge/fog
devices
Web
Social
Mobile
Video &
audio
Relational
DBs
Raw data
Batch
processing
Stream processing
Batch view
Real-time
dashboard
BI tool
App
Report
Services
Analytical tools
Alert
Data in motion
Data at rest
Integration layer
Batch layer
Speed/real-time layer
Serving layer
Input
Output
Data
Intelligence
Action
Real-time view
Query
Fig. 4.9 Lambda architecture
189
• Data in Motion – Data collection is similar to that of data at rest; however, data
analysis happens in real time as the data-creating event occurs. For example,
a connected wristband can constantly collect and record guest activity data in
an event/conference/fair to customize the guest’s visit with activity suggestions
based on individual behavior, enabling personalized user experience in real time.
4.5.1 Data Processing Architectures
In this section we overview two well-known architectures for data processing,
namely, Lambda architecture and Kappa architecture.
4.5.1.1 Lambda Architecture
Nathan Marz and James Warren first proposed Lambda architecture in “Big Data:
Principles and best practices of scalable real-time data systems.” Lambda was
designed as a universal, fault-tolerant, scalable data processing architecture able to
process massive amounts of data using stream processing and batch processing techniques. Figure 4.9 illustrates the three main elements found in Lambda architecture:
the speed/real-time layer, batch layer, and serving layer [12].
• Batch Layer – Responsible for storing raw data and processing data in batch
model to create batch views. The data scope in the batch layer can encompass
hours to years.
IP
devices
Edge/fog
devices
Web
Social
Mobile
Video &
audio
Relational
DBs
Raw data
Batch
processing
Stream processing
Batch view
Real-time
dashboard
BI tool
App
Report
Services
Analytical tools
Alert
Data in motion
Data at rest
Integration layer
Batch layer
Speed/real-time layer
Serving layer
Input
Output
Data
Intelligence
Action
Real-time view
Query
Fig. 4.9 Lambda architecture