(4) sorted_executor_queue = sortedByWorker(load_worker_queue)
(5) for(taskset in tasksets){
(6) for(LOCALITY in data_locality if !taskset.isEmpty){
(7) for(count in 1 to 2){
(8) for(executor in sorted_executor_queue if !taskset.isEmpty){
(9) while(canAllocateTask(executor)){
(10) if(count == 1 && executor.worker.load > spark.load.threshold){
(11) continue} task = taskAtLocality(taskset, LOCALITY, executor)
(12) taskset.remove(task)
(13) allocateTask(task, executor)
(14) executor.worker.load + = spark.task.cpus/executor.worker.cores
(15) executor.cores − = spark.task.cpus
(16) If(executor.cores < spark.task.cpus){sortedByLoad(load_worker_queue)
(17) sorted_executor_queue = sortedByWorker(load_worker_queue)}}}}}
(18) }
4 Experiment and Result Analysis
4.1 Experimental Design
The purpose of the experiment is to verify that the task scheduling of the dynamic load
schedules the task to the low-load node, makes the distribution in the task re-cluster as
uniform as possible, and accelerates the job operation with the low-load node.
The experimental cluster provides four worker CPUs with four CPU cores. The
maximum task concurrency of the program is 12. In the experiment, it is necessary to
test whether the load scheduling has the ability to avoid high-load nodes and select the
low-load Executor to perform tasks. The test plan is shown in Table 1.
The experimental node sequentially presets the load values shown in Table 2 to
perform experiments. The reason why tests 1, 2 and tests 3, 4 choose two different tasks
is because the degree of concurrency of 1, 2 is exactly the number of all CPU cores of
the three Executors. In this case, Spark will take up all the cores for calculation. The
concurrency of 9 is less than 12, which means that load scheduling can fill up to 4
Tasks on some Executors and only 4 Tasks on other Executors, depending on the load
status of each computer.
Table 1. Experimental test program configuration table
Program name
Log handler
Computer model
Use
Number of computers
Number of processor cores
Lenovo T530
Worker
4
4
ASUS X550V
Driver
1
4
134
Y. Liang et al.
(5) for(taskset in tasksets){
(6) for(LOCALITY in data_locality if !taskset.isEmpty){
(7) for(count in 1 to 2){
(8) for(executor in sorted_executor_queue if !taskset.isEmpty){
(9) while(canAllocateTask(executor)){
(10) if(count == 1 && executor.worker.load > spark.load.threshold){
(11) continue} task = taskAtLocality(taskset, LOCALITY, executor)
(12) taskset.remove(task)
(13) allocateTask(task, executor)
(14) executor.worker.load + = spark.task.cpus/executor.worker.cores
(15) executor.cores − = spark.task.cpus
(16) If(executor.cores < spark.task.cpus){sortedByLoad(load_worker_queue)
(17) sorted_executor_queue = sortedByWorker(load_worker_queue)}}}}}
(18) }
4 Experiment and Result Analysis
4.1 Experimental Design
The purpose of the experiment is to verify that the task scheduling of the dynamic load
schedules the task to the low-load node, makes the distribution in the task re-cluster as
uniform as possible, and accelerates the job operation with the low-load node.
The experimental cluster provides four worker CPUs with four CPU cores. The
maximum task concurrency of the program is 12. In the experiment, it is necessary to
test whether the load scheduling has the ability to avoid high-load nodes and select the
low-load Executor to perform tasks. The test plan is shown in Table 1.
The experimental node sequentially presets the load values shown in Table 2 to
perform experiments. The reason why tests 1, 2 and tests 3, 4 choose two different tasks
is because the degree of concurrency of 1, 2 is exactly the number of all CPU cores of
the three Executors. In this case, Spark will take up all the cores for calculation. The
concurrency of 9 is less than 12, which means that load scheduling can fill up to 4
Tasks on some Executors and only 4 Tasks on other Executors, depending on the load
status of each computer.
Table 1. Experimental test program configuration table
Program name
Log handler
Computer model
Use
Number of computers
Number of processor cores
Lenovo T530
Worker
4
4
ASUS X550V
Driver
1
4
134
Y. Liang et al.