NUMA Architectures
In non-uniform memory access systems, a core reaches its local memory faster than memory attached to another socket, so data placement affects speed.
Non-uniform memory
A modern compute node usually has several processor sockets, each with its own attached memory. Any core can address all of memory, but reaching memory on its own socket (local) is faster and higher-bandwidth than reaching memory on another socket (remote), which must cross an inter-socket link. This is non-uniform memory access (NUMA). Each socket-plus-memory group is a NUMA node.
First-touch placement
Most operating systems allocate a page of memory on the NUMA node of the core that first writes to it, not the core that allocated it. This first-touch policy means that if a single thread initializes a large array, the whole array lands on one node, and threads on other sockets pay remote-access penalties forever after. The remedy is to initialize data in parallel with the same thread-to-data mapping used later, so each thread touches (and thus places) the data it will work on.
- Local memory access is faster than remote (cross-socket) access.
- First-touch places a page on the node of the first thread to write it.
- Parallel initialization aligns data placement with computation.
- Thread and memory pinning keeps the alignment stable at runtime.
Pinning
Even with good placement, the OS may migrate threads between cores, breaking locality. Affinity settings pin threads to cores and memory to nodes so the alignment holds. Tools and environment variables (thread affinity controls, numactl) express these bindings. Getting affinity and first-touch right can change memory-bound performance on a node by a large factor with no change to the algorithm.
In practice
A hybrid MPI+OpenMP Hyperion solver places one MPI rank per NUMA node and pins its threads to that node's cores, initializing each rank's subdomain with those threads. This keeps hot field arrays local, avoiding the cross-socket traffic that would otherwise throttle a bandwidth-bound stencil.