Mamba and Selective State-Space Models
Mamba makes state-space model parameters depend on the input, giving content-aware selectivity with linear-time sequence processing and no attention matrix.
Making state-space models selective
Classical structured state-space models use parameters that are fixed across the sequence, so the same linear recurrence is applied to every input regardless of content. This is efficient but limits the model's ability to decide what to keep and what to forget based on what it is currently reading. Mamba addresses this by making the state-space parameters, specifically the input and step-size terms, functions of the input at each position. The recurrence becomes selective: it can focus on relevant tokens and ignore irrelevant ones, an ability previously associated mainly with attention.
The cost of selectivity
Once the parameters vary with the input, the elegant convolution shortcut of S4 no longer applies, because the kernel is no longer fixed. Mamba recovers efficiency with a hardware-aware parallel scan: it computes the recurrence with a work-efficient prefix-scan algorithm and keeps intermediate states in fast on-chip memory rather than writing them to slower memory. This makes the selective recurrence practical at scale despite losing the convolutional form.
- Input-dependent parameters give content-aware filtering of the sequence
- Linear time and memory in sequence length, unlike quadratic attention
- Constant-memory recurrent inference, well suited to long generations
- A parallel scan on-chip preserves training throughput
Architecture
A Mamba block wraps the selective state-space layer with input and output projections and a gating branch, replacing the attention-plus-MLP pattern of a transformer with a single homogeneous block that is repeated. There is no explicit token-to-token attention matrix, so memory does not grow with the square of the sequence length.
Trade-offs
Mamba is competitive with transformers on language and other sequence tasks while scaling far better to long contexts, which makes it attractive for very long documents, audio, and genomic sequences. Its recurrent state is a fixed-size summary of the past, so unlike attention it cannot look back at an arbitrary earlier token with perfect fidelity; it must have stored what it needs. Hybrid designs that interleave Mamba blocks with occasional attention layers aim to combine both strengths.