The NERSC Cori system
Cori is a new Cray XC40 supercomputer hosted at NERSC at the Lawrence Berkeley National Laboratory (LBNL) in California. The system overview is detailed on the NERSC website. Cori is equipped of 2,004 Intel Haswell computes nodes and 9,304 Intel KNL compute nodes. The system has a theoretical peak of 27.88 TFlops and is at rank 5 in the top500 most powerful supercomputers in November 2016 with a maximum peak performance of 14 TFlops.
Global kernel performance
Global performance of the PICSAR kernel has been measured on a single node of the Haswell and the KNL partitions using the FORTRAN stand-alone kernel. For this aim, the similar test case has been used: a homogeneous thermalized plasma that fulfills the entire domain. the domain is divided into 100x100x100 cells with 2 species (electrons and protons) and 40 particles per cell and per species. The use of the thermalized homogeneous plasma ensures that the load is balanced between processors during the whole simulation time and that same amounts of particles cross all boundaries. For this study, the code has been compiled with Intel Compiler 17 using Intel-MPI. The direct deposition is used for the charge and the current deposition processes.
The time per iteration and per particle on a Haswell node (2 Intel Haswell sockets) is given in Fig. 1 for order 1 and order 3 interpolations. In physical simulations, order 3 is usually preferred for accuracy. This study exhibits a speed-up of 2.4 at order 1 and 1.7 at order 3.
The time per iteration and per particle on a KNL node (2 Intel Haswell sockets) is given in Fig. 2 for order 1 and order 3 interpolations. The system used here is not Cori but Carl, the samll KNl cluster available before the arrival of Cori. This study exhibits a speed-up of 3.7 at order 1 and 5 at order 3.
Optimization efforts perform in PICSAR now enable to have better performance on Many-Integrated Cores architectures (KNL) than on previous multi-core architectures. A speed-up of 1.8 is observed at order 3 between Haswell and KNL. Similar simulation times are obtained at order 1.
OpenMP strong scaling on a single KNL node
A strong scaling of OpenMP has been performed on a node of Carl (Intel compiler with Intel MPI) using a KNL configured in SNC4. For this aim we use 4 MPI ranks, one per NUMA (Non-Uniform Memory Access) domain. The number of OpenMP threads ranges between 1 per MPI rank to 64 if we use hyper-threading. Hyper-threading starts above 17 cores per MPI rank (68 cores). The results are shown in Fig. 3. OpenMP scaling efficiency slowly diminishes and is still good until 16 OpenMP threads with an efficiency of 70%. Above, with hyperscaling, efficiency drops to reach 20% for 64 OpenMP threads per MPI rank. This is mainly due to the communications as shown by the red and green curves. Grid communication scales very badly but represents between 1 and 10% of the simulation time. They represent guard cell exchanges between MPI domains. Particle communication represent between 20 and 40% of the simulation time.
In some cases, using hyper-threading can bring an additional speedup. One should bear in mind that on KNL all cores have to be used to saturate the bandwidth. For codes with a high arithmetic intensity (high flop per byte ratio), hyper-threading can be interesting. However, bandwidth bound codes or code that exploit the totality of the L2 cache can suffer from loss of performance. Using more threads means that L2 will be shared with more processing units leading to potential L2 cache misses in case of saturation.
The study performed on a single KNL node shows that PICSAR can benefit from hyper-threading until 2 threads per core. Using 4 threads per core leads to similar performance to 1 thread per core. as shown in Fig. 4. This behavior is similarly observed for quadrant flat, SNC4 flat and quadrant cache mode.
Although PICSAR can be considered as partially memory bound, the stand-alone Fortran code can be used with 2 threads per core to obtain 20% speedup. As shown in Fig 3. in the previous section, hyper-threading does not scale and efficient drops rapidly for more than a thread per core.
Hybrid parallel configuration
PICSAR uses a hybrid MPI-OpenMP parallelization. In order to determine the best ratio number of MPI ranks/number of OpenMP threads in term of performance, a parametric study has been conducted on quadrant flat, quandrant cache and SNC4 flat mode. The results are shown in Fig. 5.
On a single node, for the homogeneous test case, simulation time does not vary much between 4 MPI ranks/32 OpenMP threads and the full MPI case 64 MPI ranks/2 OpenMP threads. Only the full OpenMP case appears nearly 40% slower. This study also shows that on a single node, performance between the 3 studied KNL configuration are very similar when the simulation entirely fits in MCDRAM (less than 16 GB).
DDR versus MCDRAM
When the simulation can entirely fit in MCDRAM, memory bound codes can highly benefit from the high-bandwidth. This study reveals advantage of MCDRAM for PICSAR stand-alone when the problem is below 16 GB. Fig. 6 shows the time per iteration per particle for a series of 5 simulations with different hybrid parallel configurations (see abscissa) and for 2 KNL configurations: quadrant flat with in one case the use of MCDRAM only (
numactl -m 1, blue) and in the other the use of DDR only (
numaclt -m 0, orange).
For any hybrid parallel configuration using here hyper-threading (2 threads per core), there is an average factor of 2 between simulation time on MCDRAM and in DDR. Using the MCDRAM as a memory or as a cache when the problem can be contained in it is an advantage for the PICSAR simulations.
Effect of L3 cache
Intel KNL processors do not have a L3 cache level (LLC cache) contrary to Intel Haswell processors but have this MCDRAM that can be used in cache mode. The L3 cache is a medium size (40 MB) high-bandwidth memory.
Mathieu Lobet, last update: January 27, 2017