190
F. Firouzi and B. Farahani
• Speed Layer – Processes data in real time and computes real-time data views.
• Serving Layer – Stores the result of batch layer and speed layer (i.e., batch views
and real-time views) and responds to ad hoc queries by returning previously
computed batch and real-time views or building new views from the batch and
speed layer outputs stored in the serving layer. This approach provides a more
holistic and complete view by obtaining the best of two worlds (i.e., batch and
real time). Batch views can be processed using more complex rules resulting in
higher data quality and less skew. On the other hand, real-time views provide
instant access to data.
The main benefits of Lambda architecture are as below:
• Real Time – The real-time layer can apply simple data processing and machine
learning algorithms to provide real-time insights and alerts.
• Improved Processing Without Data Loss – The batch layer allows data processing
to take place with great precision using complex algorithms without the loss of
alerts, short-term information, or other insights generated by the real-time layer.
• Reduced Storage Needs – Because of the batch layer, the Lambda architecture
minimizes the need for random write storage.
• Tolerance to Human Errors and Hardware Crashes – A well-implemented batch
layer makes it difficult for hardware crashes or human errors to damage stored
data because the system does not allow existing data to be deleted or updated.
The real-time layer is more vulnerable to errors. Data can be lost or corrupted in
this layer because the data stores are variable. However, if incoming writes are
being transmitted to the batch storage area, the data results will eventually catch
up when the next batch is processed. This means no data will be lost even if the
real-time layer encounters an error. While results may be outdated if the real-time
layer experiences failure, the batch layer data records will not be damaged and
the results will sync again when the real-time layer is functioning correctly again.
While there are benefits to Lambda architecture, there are also shortcomings that
should be considered:
• Complexity – Lambda architecture comprised of many layers, and thus maintaining proper syncing between layers can be costly and requires more thoughtful
effort and handling.
• Maintenance and Support – Because this architecture is made up of two
clearly defined, completely distributed layers (speed and batch), support and
maintenance activities can be difficult.
• Technology Mastery – Many technology proficiencies must be used to create
Lambda architecture. Finding and recruiting qualified professionals with expertise in these areas can be difficult.
• Complex Implementation and Deployment – Creating Lambda architecture using
open-source technologies and then deploying it via the Cloud or on-premises
servers can be complicated.
F. Firouzi and B. Farahani
• Speed Layer – Processes data in real time and computes real-time data views.
• Serving Layer – Stores the result of batch layer and speed layer (i.e., batch views
and real-time views) and responds to ad hoc queries by returning previously
computed batch and real-time views or building new views from the batch and
speed layer outputs stored in the serving layer. This approach provides a more
holistic and complete view by obtaining the best of two worlds (i.e., batch and
real time). Batch views can be processed using more complex rules resulting in
higher data quality and less skew. On the other hand, real-time views provide
instant access to data.
The main benefits of Lambda architecture are as below:
• Real Time – The real-time layer can apply simple data processing and machine
learning algorithms to provide real-time insights and alerts.
• Improved Processing Without Data Loss – The batch layer allows data processing
to take place with great precision using complex algorithms without the loss of
alerts, short-term information, or other insights generated by the real-time layer.
• Reduced Storage Needs – Because of the batch layer, the Lambda architecture
minimizes the need for random write storage.
• Tolerance to Human Errors and Hardware Crashes – A well-implemented batch
layer makes it difficult for hardware crashes or human errors to damage stored
data because the system does not allow existing data to be deleted or updated.
The real-time layer is more vulnerable to errors. Data can be lost or corrupted in
this layer because the data stores are variable. However, if incoming writes are
being transmitted to the batch storage area, the data results will eventually catch
up when the next batch is processed. This means no data will be lost even if the
real-time layer encounters an error. While results may be outdated if the real-time
layer experiences failure, the batch layer data records will not be damaged and
the results will sync again when the real-time layer is functioning correctly again.
While there are benefits to Lambda architecture, there are also shortcomings that
should be considered:
• Complexity – Lambda architecture comprised of many layers, and thus maintaining proper syncing between layers can be costly and requires more thoughtful
effort and handling.
• Maintenance and Support – Because this architecture is made up of two
clearly defined, completely distributed layers (speed and batch), support and
maintenance activities can be difficult.
• Technology Mastery – Many technology proficiencies must be used to create
Lambda architecture. Finding and recruiting qualified professionals with expertise in these areas can be difficult.
• Complex Implementation and Deployment – Creating Lambda architecture using
open-source technologies and then deploying it via the Cloud or on-premises
servers can be complicated.