HPC & Compute

Data Parallelism

Data parallelism applies the same operation to many data elements simultaneously, the dominant pattern in scientific computing and deep learning.

Same operation, many elements

In data parallelism the program performs one operation across a large collection of data at the same time. Adding two vectors of a million elements is embarrassingly data-parallel: every element pair is independent, so the work can be split across cores, vector lanes, or GPU threads with no coordination beyond the split itself.

Hardware that exploits it

Kronos motion — data assimilation

SIMD units apply one instruction to a vector of operands
GPUs run thousands of lightweight threads over array elements
Distributed clusters partition arrays across nodes and operate on local pieces

A concrete example

python

import numpy as np
# One expression, applied to every element in parallel
# under the hood (SIMD / multithreaded BLAS):
a = np.random.rand(1_000_000)
b = np.random.rand(1_000_000)
c = a * b + 3.0   # elementwise, data-parallel

In machine learning

Deep-learning training is often data-parallel across the batch: each device holds a full copy of the model and processes a different slice of the mini-batch, then gradients are averaged with an all-reduce. This scales well until the gradient-synchronization communication starts to dominate, at which point practitioners turn to model or pipeline parallelism as well.

Limits

Pure data parallelism assumes elements are independent. When elements interact, as in a stencil update that reads neighbors, boundaries must be exchanged between the pieces held by different processors. This halo exchange is the communication cost that separates a toy vector-add from a real physics solver, and managing it well is central to scalable simulation.