Authors: Jonathan Lifflander, Nicole Slattengren, Philippe Pebay, Pierre Pebay, Caleb Schilly, Robert Pfeiffer, Joseph Kotulski

Published: Association for Computing Machinery, 2025

While load balancing in distributed-memory computing has been well-studied, we present an innovative approach to tackle challenges that an electromagnetic application poses due to irregular workloads and tight memory constraints. To this end, we present a unified model for approximating work in a distributed system that combines three key components: computation, communication, and memory. This enables the exploration of complex trade-offs in task placement, such as increased parallelism at the expense of data replication. We then present our new fully distributed load balancing strategy that incorporates this model.

To predict workloads for the matrix assembly of the electromagnetics application, we apply machine learning across an ensemble of executions to train a neural network, which makes online predictions for our task-based decomposition, informing the load balancer of the computational loads. Finally, we demonstrate that our approach, when applied to this application, leads to substantial speedups, up to 2.0x, thereby decreasing time-to-solution for the imbalanced execution.

Authors: Jonathan Lifflander, Nicole Slattengren, Philippe Pebay, Pierre Pebay, Caleb Schilly, Robert Pfeiffer, Joseph Kotulski

Technical Report: DOE NNSA, 2024

While load balancing in distributed-memory computing has been well-studied, we present an innovative approach to this problem: a unified, reduced-order model that combines three key components to describe “work” in a distributed system: computation, communication, and memory. Our model enables an optimizer to explore complex tradeoffs in task placement, such as augmented parallelism, at the expense of data replication increasing memory usage.

We propose a fully distributed, heuristic-based load balancing optimization algorithm, and demonstrate that it quickly finds close-to-optimal solutions. We formalize the complex optimization problem as a mixed-integer linear program, and compare it to our strategy. Finally, we show that when applied to an electromagnetics code, our approach obtains up to 2.3x speedups for the imbalanced execution.

Authors: Nicolas Morales, Reese Jones, Jonathan Lifflander, Philippe P. Pébaÿ, Sean McGovern, Cezary Skrzyński, Caleb Schilly

Conference: Workshop on Asynchronous Many-Task Systems and Applications, 2024

Contact mechanics, or the modeling of the impenetrability of solid objects, is fundamental to computational solid mechanics (CSM) applications yet is oftentimes the most challenging in terms of computational efficiency and performance.

These challenges arise from the irregularity and highly dynamic nature of contact simulation, particularly with algorithms designed for distributed memory architectures. First among these challenges is the inherent load imbalance when distributing contact load across compute nodes. This imbalance is highly problem dependent, and relates to the surface area of contact manifolds and the volume around them, rather than the distribution of the mesh over compute nodes, meaning the application load can vary drastically over different phases. The dynamic nature of contact problems motivates the use of distributed asynchronous many-tasking (AMT) frameworks to efficiently handle irregular workloads.

In this paper, we present our work on distBVH, a distributed contact solution using the DARMA/vt library for asynchronous tasking that is also capable of running on-node Kokkos-based kernels. We explore how distBVH addresses the various challenges of CSM contact problems. We evaluate the use of many of DARMA/vt's dynamic load balancers and demonstrate how our load balancing approach can provide significant performance improvements on various computational solid mechanics benchmarks. Additionally, we show how our approach can take advantage of DARMA/vt for tasking and efficient on-node kernels using Kokkos to scale over hundreds of processing elements.