Small inefficiencies, major delays
In high-performance computing (HPC), even small inefficiencies can cause significant delays. I recently encountered such a challenge while supporting a customer in the semiconductor industry who had integrated AMD EPYC systems into their global compute clusters. These systems, known for their high core counts and energy efficiency, were supposed to accelerate simulation workloads. However, unexpected performance issues arose, particularly with multithreaded jobs.
The bottleneck
The customer noticed that certain simulation jobs, particularly those using 4 to 8 cores, had inconsistent runtimes. Some tasks were completed quickly, while others lagged significantly behind, causing delays in the overall workflow. This inconsistency was striking, especially since all clusters had the same hardware and software configuration.
After investigation, I discovered that the cause lay in the interaction between the job scheduler and the Non-Uniform Memory Access (NUMA) architecture of the AMD EPYC processors. Specifically, the scheduler assigned threads without taking into account the L3 cache boundaries inherent in the EPYC design. As a result, threads that shared data were placed on cores with separate L3 caches, leading to higher latency from cross-CCD cache traffic.
Understanding the architecture
AMD EPYC processors use a chiplet-based architecture, in which each chiplet—also known as a Core Complex Die (CCD)—contains multiple cores that share an L3 cache. This design offers advantages in terms of scalability and performance, but also introduces complexity in memory access patterns. When threads that communicate intensively with each other are assigned to cores on different CCDs, they must access data via separate L3 caches. This leads to increased latency and reduced performance.
Implementing a solution
To address this, I developed a strategy to make the scheduler NUMA-aware with respect to L3 cache boundaries. By modifying the topology information that the system presents to the scheduler, I ensured that threads that share data are assigned to cores within the same L3 cache domain. This approach consisted of the following steps:
- Topology adjustment: Adjusting the system's hardware topology information so that the L3 cache boundaries are exposed correctly. This caused the scheduler to group threads within the same cache domain.
- Scheduler configuration: updating the scheduler configuration to prioritize assigning threads to cores within the same L3 cache group, minimizing communication between caches.
- Validation and testing: performing benchmark tests to compare performance before and after the adjustments. The results showed a clear decrease in job runtime variation and improved overall efficiency.
Results and impact
- Consistent job runtimes: the variation in job completion times decreased significantly, resulting in more predictable workflows.
- Improved resource utilization: by reducing communication between caches, CPU and memory usage became more efficient.
- Increased throughput: the total throughput of the compute clusters increased, allowing more simulations to be processed in less time.
This experience underscores the importance of job scheduling strategies that are aligned with the underlying hardware architecture in HPC environments. By making the scheduler aware of the NUMA topology—and, in particular, the L3 cache domains—we can optimize performance and harness the full potential of modern processors such as AMD EPYC.
Conclusion
In high-performance computing, you need to understand the processor architecture and tailor your software accordingly. This case study demonstrates how well-thought-out adjustments to job scheduling, based on hardware topology, can lead to significant performance improvements. HPC systems continue to evolve; optimizations like these remain essential for efficient and reliable workflows.
