Interconnects

The interconnect is the network binding compute nodes; its latency and bandwidth often decide whether a large simulation scales.

The network is a first-class component

In a cluster, nodes communicate over a dedicated high-speed network. For communication-heavy codes the interconnect matters as much as the processors, because time lost moving data cannot be recovered by faster arithmetic. HPC interconnects are engineered for low latency and high bandwidth well beyond commodity Ethernet.

Latency and bandwidth

A message's time is approximately latency plus (message size / bandwidth). Latency, the fixed per-message startup, dominates small transfers; bandwidth caps large ones. Modern HPC fabrics reach sub-microsecond latencies and hundreds of gigabits per second per link. Codes are tuned to send fewer, larger messages to amortize latency.

Common fabrics

• InfiniBand: the traditional low-latency HPC fabric
• High-performance Ethernet with RDMA (RoCE)
• Vendor fabrics such as Slingshot and proprietary tori

RDMA and offload

Remote Direct Memory Access lets one node read or write another's memory without involving the remote CPU, cutting latency and freeing processors. Combined with hardware collective offload, RDMA is what allows collectives to scale across tens of thousands of nodes.

Topology matters

How links are arranged, the topology, sets how many hops messages take and how bandwidth is shared under load. Fat-trees, dragonflies, and tori trade cost against bisection bandwidth. Placement of a job's ranks onto physically nearby nodes reduces contention and improves scaling. Because the interconnect is shared among all running jobs, one communication-heavy job can slow others, which is why facilities monitor fabric utilization and schedulers try to allocate compact, well-shaped node sets.