116
Y. Qin et al.
worker result reflects the computing power of the worker, and the heterogeneity difference between different workers is reflected in different worker capacity.
worker result is calculated by running the benchmark program benchmark job
in the worker. After the introduction of benchmark, the definition of zone can
also be expressed as a grouping of computers in the cluster with the same benchmark performance and the same number of CPU cores.
2.3 Zone-Based Resource Allocation Strategy
Algorithm 1 Zone-based Resource Allocation Strategy
Input: spark.zone.level,spark.cores
Output: allocated computing resources
1: function AllocateResource(spark.zone.level, spark.cores)
2:
reqCores ← spark.cores
3:
zoneLevel ← spark.zone.level
4:
targetZone ← getById(zoneLevel)
5:
for worker in targetZone.workerList do
6:
if reqCores >= worker.coreN um − 1 then
7:
allocateResources(app,worker)
8:
reqCores ← reqCores − worker.coreN um
9:
end if
10:
if reqCores <= 0 then
11:
return
12:
end if
13:
end for
14:
zoneList.filter(zoneLevel)
15:
zoneQueue ← zoneList.sortByCpuCapacity().sortByCpuNums()
16:
while reqCores > 0 and !zoneQueue.isEmpty() do
17:
zone ← zoneQueue.pop()
18:
for worker in zone do
19:
if reqCores >= worker.coreN um − 1 then
20:
allocateResources(app,worker)
21:
reqCores ← reqCores − worker.coreN um
22:
end if
23:
end for
24:
end while
25:
return
26: end function
The above section defines zones and the heterogeneity between zones in a
heterogeneous cluster. This part will design an allocation strategy of computing
resource that utilizes cluster heterogeneity to avoid heterogeneity affecting job
execution, improve job execution speed and perform zones scheduling according
to user requirements and make rational use of cluster computing resources.