114
Y. Qin et al.
In order to make full use of the heterogeneity of the cluster and solve the
fluctuations and differences in job performance caused by different hardware
configurations in heterogeneous clusters, cluster heterogeneity should be fully
considered when assigning Executors to the application in the process of Spark
resource allocation [3].
2 System Model
Zone-based resource allocation strategy (ZbRAS) enhances Spark for heterogeneous clusters by scheduling tasks based on zone division module and resource
allocation module and can improve the execution speed of the job by assigning
high-performance computing resources to the job.
2.1 The Definition of Zone
We have analyzed the existence of optimization space for Spark scheduling on
heterogeneous clusters. The following cluster consists of three different types of
computers. The three types of computers have the same number of CPU cores
and the same memory size. The CPU performance of the Type-1 nodes is the
highest, and the CPU performance of the Type-2 and Type-3 nodes is second.
Figure 1 shows the general task schedule in Spark. The task to be executed is
randomly assigned to the node in the cluster. Obviously, the default scheduling
strategy does not fully utilize the highest performing Type-1 nodes in the cluster.
Fig. 1. Scheduling in native Spark.
Fig. 2. Scheduling in Spark with
ZbRAS.
To reflect the similarities and differences between computers in a Spark cluster, we introduce a new definition—Zone. Different zones represent computer
clusters of different performance and also represent computing resources of different priorities. With sufficient computing resources, users can schedule highpriority jobs to run in high-performance zones and low-priority jobs to run in
low-performance zones. This strategy makes it possible to use computers in the
cluster more reasonably.
Spark is a big data computing framework which divides cluster resources
by CPU cores and can complete jobs’ running in a limited time. Therefore, in
Y. Qin et al.
In order to make full use of the heterogeneity of the cluster and solve the
fluctuations and differences in job performance caused by different hardware
configurations in heterogeneous clusters, cluster heterogeneity should be fully
considered when assigning Executors to the application in the process of Spark
resource allocation [3].
2 System Model
Zone-based resource allocation strategy (ZbRAS) enhances Spark for heterogeneous clusters by scheduling tasks based on zone division module and resource
allocation module and can improve the execution speed of the job by assigning
high-performance computing resources to the job.
2.1 The Definition of Zone
We have analyzed the existence of optimization space for Spark scheduling on
heterogeneous clusters. The following cluster consists of three different types of
computers. The three types of computers have the same number of CPU cores
and the same memory size. The CPU performance of the Type-1 nodes is the
highest, and the CPU performance of the Type-2 and Type-3 nodes is second.
Figure 1 shows the general task schedule in Spark. The task to be executed is
randomly assigned to the node in the cluster. Obviously, the default scheduling
strategy does not fully utilize the highest performing Type-1 nodes in the cluster.
Fig. 1. Scheduling in native Spark.
Fig. 2. Scheduling in Spark with
ZbRAS.
To reflect the similarities and differences between computers in a Spark cluster, we introduce a new definition—Zone. Different zones represent computer
clusters of different performance and also represent computing resources of different priorities. With sufficient computing resources, users can schedule highpriority jobs to run in high-performance zones and low-priority jobs to run in
low-performance zones. This strategy makes it possible to use computers in the
cluster more reasonably.
Spark is a big data computing framework which divides cluster resources
by CPU cores and can complete jobs’ running in a limited time. Therefore, in