Chapter 6
Big Data
Natasha Balac
Errors using inadequare data are much less than those using no
data at all.
Charles Babbage
Contents
6.1 Introduction to Big Data .................................................................. 316
6.1.1 Defining Big Data................................................................ 317
6.1.2 Volume ........................................................................... 317
6.1.3 Velocity ........................................................................... 318
6.1.4 Variety ............................................................................ 318
6.1.5 Veracity ........................................................................... 318
6.2 Big Data Management and Computing Platforms ....................................... 319
6.2.1 Big Data System Architecture Components .................................... 320
6.2.2 Hadoop History .................................................................. 321
6.2.3 The Apache Hadoop Framework Components ................................. 322
6.2.4 Hadoop Distributed File System ................................................ 323
6.2.5 MapReduce....................................................................... 326
6.2.6 YARN ............................................................................ 327
6.3 An Introduction to Big Data Modeling and Manipulation .............................. 329
6.3.1 Big Table ......................................................................... 329
6.3.2 Pig ................................................................................ 329
6.3.3 Sqoop ............................................................................. 330
6.3.4 Hive............................................................................... 330
6.3.5 HBase............................................................................. 331
6.3.6 Oozie ............................................................................. 331
6.3.7 Zookeeper ........................................................................ 331
6.3.8 Data Lakes and Warehouses ..................................................... 332
6.4 An Introduction to Spark: An Innovative Paradigm in Big Data........................ 333
6.4.1 The Spark Ecosystem ............................................................ 334
6.4.2 The Core Difference Between Spark and Hadoop ............................. 335
6.4.3 Resilient Distributed Datasets in Spark ......................................... 336
N. Balac ()
University of California San Diego, San Diego, CA, USA
e-mail: nbalac@eng.ucsd.edu
© Springer Nature Switzerland AG 2020
F. Firouzi et al. (eds.), Intelligent Internet of Things,
https://doi.org/10.1007/978-3-030-30367-9_6
315