Tensor and Matrix Cores
Tensor cores are hardware units that multiply small matrices in one operation, delivering most of an accelerator's throughput on deep learning.
Hardware for matrix multiply
A tensor core (or matrix engine) performs a small dense matrix multiply-accumulate, for example a 4x4 by 4x4 product added to an accumulator, as a single hardware operation. Because deep-learning training and inference are dominated by matrix multiplication, dedicating silicon to this one pattern yields far higher throughput than general vector units.
How they are used
Large matrix multiplications are tiled so each tile maps onto a tensor-core operation. Inputs are typically low precision (FP16, BF16, or FP8) while the accumulation is kept in higher precision (FP32) to control error. Libraries and frameworks emit tensor-core instructions automatically for supported operations.
Why they are so fast
- Many multiply-accumulates per instruction, amortizing control overhead
- Low-precision inputs move fewer bytes and pack more into registers
- High-precision accumulation preserves usable accuracy
The catch: only for the right shape
Tensor cores accelerate only work cast as dense matrix multiply of suitable size and precision. Elementwise operations, reductions, and irregular kernels see no benefit and run on ordinary units. This shapes model design: architectures that express computation as large dense matrix multiplies extract the most from the hardware.
Beyond machine learning
Because so much scientific computing reduces to dense linear algebra, tensor cores are increasingly exploited outside ML, through mixed-precision solvers that do bulk factorization in low precision on tensor units and refine in high precision. This lets traditional simulation borrow throughput built for deep learning.