Domain Decomposition
Domain decomposition partitions a simulation's spatial grid across processors, the foundational strategy for parallel physics codes.
Divide the space
Most physics simulations discretize a spatial domain into a grid or mesh. Domain decomposition splits that domain into subdomains, one per process, so each process updates only its own region. It is the dominant parallelization strategy for grid-based codes because it maps the problem's natural structure onto distributed memory.
Halo exchange
Updating a cell usually requires the values of neighboring cells. At a subdomain boundary, some neighbors live on another process. Each process therefore keeps a border of ghost (halo) cells holding copies of the neighbors' edge data, refreshed each step by exchanging boundary values with adjacent processes. This neighbor communication is the core cost of the method.
Surface-to-volume
Computation scales with a subdomain's volume; communication scales with its surface. As processors increase and subdomains shrink, the surface-to-volume ratio rises, so communication grows relative to computation. This is precisely why strong scaling eventually saturates, and why keeping subdomains reasonably large matters.
Partitioning quality
- Balance work: equal computational cost per subdomain
- Minimize surface: reduce boundary data to exchange
- Preserve locality: keep neighboring subdomains on nearby nodes
Irregular meshes
Structured grids partition into simple blocks. Unstructured meshes need graph partitioners (such as ParMETIS) that cut the mesh to balance elements while minimizing the edge cut. When the workload shifts, as with adaptive refinement or moving plasma features, the partition is periodically recomputed to restore balance.