192
K. Tsugane et al.
Fig. 11 Ping-Pong communication bandwidth with MPI (MPI_Send/Recv) and XMP (coarray)
Table 3 Evaluation of the
weak scaling of
decomposition for each
domain using problems
ranging from 16 to 512
processes
Problem size A
Default Toroidal Radial
Particle
mstep
100
20
20
20
mpsi
90
90
90–2880 90
mzetamax
64
2–64
2
2
Particles per cell 100
100
100
100–3200
to use two ports of Infiniband, but we could get the performance of only single port
of Infiniband. It may be an issue with GASNet library.
The GTC-P simulation size is determined by several important numerical
parameters. Table 3 shows the default parameters for problem size A provided
by GTC-P, where we modified the parameters to evaluate weak scaling based on
problem size A. Strong scaling was evaluated using the minimum parameters in the
decomposition of each domain shown in Table 3, where mstep is the number of
calculation steps, mzetamax is the number of grid points in the toroidal dimension,
and mpsi is the number of grid points in the radial domain. Because the number of
grid points in the poloidal plane and in the toroidal domain must be the same during
decomposition, this was also changed in the parameter set for problem size A.
First, we used up to 32 nodes of HA-PACS where 16 processes ran on each node
and the total number of processes ranged from 16 to 512. The processes mapped to
evaluate the decomposition on each domain are shown in Table 4. As described
above, three problem dimensions were considered: toroidal, radial, and particle.
When we decomposed these dimensions into parallel processes, we always fixed the
decomposition number on two dimensions (e.g., toroidal and radial) as 2 × 2 and
we varied the decomposition size in the other dimension (e.g., particle) from 4 to
128, thereby scaling the total number of processes from 16 to 512. However, during
decomposition on the toroidal dimension, we fixed the decomposition number on the
K. Tsugane et al.
Fig. 11 Ping-Pong communication bandwidth with MPI (MPI_Send/Recv) and XMP (coarray)
Table 3 Evaluation of the
weak scaling of
decomposition for each
domain using problems
ranging from 16 to 512
processes
Problem size A
Default Toroidal Radial
Particle
mstep
100
20
20
20
mpsi
90
90
90–2880 90
mzetamax
64
2–64
2
2
Particles per cell 100
100
100
100–3200
to use two ports of Infiniband, but we could get the performance of only single port
of Infiniband. It may be an issue with GASNet library.
The GTC-P simulation size is determined by several important numerical
parameters. Table 3 shows the default parameters for problem size A provided
by GTC-P, where we modified the parameters to evaluate weak scaling based on
problem size A. Strong scaling was evaluated using the minimum parameters in the
decomposition of each domain shown in Table 3, where mstep is the number of
calculation steps, mzetamax is the number of grid points in the toroidal dimension,
and mpsi is the number of grid points in the radial domain. Because the number of
grid points in the poloidal plane and in the toroidal domain must be the same during
decomposition, this was also changed in the parameter set for problem size A.
First, we used up to 32 nodes of HA-PACS where 16 processes ran on each node
and the total number of processes ranged from 16 to 512. The processes mapped to
evaluate the decomposition on each domain are shown in Table 4. As described
above, three problem dimensions were considered: toroidal, radial, and particle.
When we decomposed these dimensions into parallel processes, we always fixed the
decomposition number on two dimensions (e.g., toroidal and radial) as 2 × 2 and
we varied the decomposition size in the other dimension (e.g., particle) from 4 to
128, thereby scaling the total number of processes from 16 to 512. However, during
decomposition on the toroidal dimension, we fixed the decomposition number on the