188
F. Firouzi and B. Farahani
Batch 1
Batch 2
Batch 3
Time
Batch 4
Process
Process
Process
Process
Process
Time
Process
Batch processing
Real-time stream processing
Fig. 4.8 Batch processing versus real-time stream processing
• Streaming Data (Stream) – Stream is a data that is continuously generated and
transferred at a steady, high-speed rate.
• Batch Processing – A batch is made up of data points that have been gathered
within a specified time period often known as a “window of data.” Batch
processing requires that a data set be collected and stored for a certain period
of time. Then, the entire set is processed together at a designated future time.
The parameters for processing can be determined in a number of ways including
a scheduled time interval (e.g., new data is processed every 5 minutes) or through
a specified condition (e.g., batch is processed when it contains five data elements
or when it reaches 1 MB of data). An example of batch processing would include
all of a financial firm’s transactions submitted over the period of a week (see
Fig. 4.8).
• Stream Processing – This type of processing enables real-time data processing
and ascertains conditions in real time. Unlike batch processing, stream processing
processes individual pieces of data as it comes in (arrives) rather than waiting to
process at a specific interval. When it comes to performance, batch processing
latency is minutes to hours while stream processing latency is only milliseconds
to seconds. It should also be noted that although stream processing handles each
new piece of data as an individual unit, many stream processing systems also
allow “window” operations that enable processing to reference data that comes
in during a specific time interval before and/or after current data (see Fig. 4.8).
• Data at Rest – This refers to data collected from a variety of sources but it is
analyzed later. For example, a retail store owner may collect previous monthly
sales data to make strategic business decisions. In this case the analysis of data
occurs separately and after the data collection phase.
F. Firouzi and B. Farahani
Batch 1
Batch 2
Batch 3
Time
Batch 4
Process
Process
Process
Process
Process
Time
Process
Batch processing
Real-time stream processing
Fig. 4.8 Batch processing versus real-time stream processing
• Streaming Data (Stream) – Stream is a data that is continuously generated and
transferred at a steady, high-speed rate.
• Batch Processing – A batch is made up of data points that have been gathered
within a specified time period often known as a “window of data.” Batch
processing requires that a data set be collected and stored for a certain period
of time. Then, the entire set is processed together at a designated future time.
The parameters for processing can be determined in a number of ways including
a scheduled time interval (e.g., new data is processed every 5 minutes) or through
a specified condition (e.g., batch is processed when it contains five data elements
or when it reaches 1 MB of data). An example of batch processing would include
all of a financial firm’s transactions submitted over the period of a week (see
Fig. 4.8).
• Stream Processing – This type of processing enables real-time data processing
and ascertains conditions in real time. Unlike batch processing, stream processing
processes individual pieces of data as it comes in (arrives) rather than waiting to
process at a specific interval. When it comes to performance, batch processing
latency is minutes to hours while stream processing latency is only milliseconds
to seconds. It should also be noted that although stream processing handles each
new piece of data as an individual unit, many stream processing systems also
allow “window” operations that enable processing to reference data that comes
in during a specific time interval before and/or after current data (see Fig. 4.8).
• Data at Rest – This refers to data collected from a variety of sources but it is
analyzed later. For example, a retail store owner may collect previous monthly
sales data to make strategic business decisions. In this case the analysis of data
occurs separately and after the data collection phase.