4 Architecting IoT Cloud
193
data scientists requiring quick iterative access to data, to execute SQL, machine
learning, or streaming workloads. The details of Apache Spark will be discussed in
Chap. 6.
4.6 Data Storage Layer: A Hybrid Architecture
Another challenge to realize holistic IoT Clouds is data storage. Relational databases
were an appropriate place for data storage in the past; however, as enterprises
strategically utilize big data and IoT applications, data persistence should not
necessarily be relational. Ideally, we need to design and deploy a hybrid architecture
that supports three types of repositories in any IoT environment (see Fig. 4.12)
[15]:
• Cold storage – This repository is usually implemented by a data lake based on
distributed file system technologies. Cold storage intends to store structured or
unstructured data that is accessed infrequently.
Cold
•
Data lake
•
Store structured &
Unstructured data
•
Unknown questions &
Unknown answers
•
Store everything
Warm
•
Database & data warehouse
•
Data for analytics
•
Regularly refreshed business
data
•
Raw data to answer known
question
Hot
•
In-memory database
•
Dashboard
•
Aggregated (processed)
data to answer known
question
Volume of
data
Performance
& Direct cost
Fig. 4.12 A hybrid architecture to address IoT data