Memory-Augmented Neural Networks
Memory-augmented networks pair a controller with an external, addressable memory it can read from and write to, separating computation from storage.
Adding an external memory
A recurrent network stores everything it remembers in a fixed-size hidden state, which limits how much it can hold and entangles storage with computation. Memory-augmented neural networks add a separate external memory: a matrix of slots that a controller network reads from and writes to through differentiable operations. This decouples what the network computes from what it stores, giving it a larger, structured place to keep information and retrieve it later, closer to how a computer separates a processor from its memory.
Differentiable addressing
For the whole system to train by gradient descent, the memory access must be differentiable. Rather than reading a single slot with a hard index, the controller produces a set of weights over all slots, typically a soft attention distribution, and reads a weighted average of their contents. Writing works similarly: an erase and an add vector, weighted across slots, update the memory smoothly. Because every operation is continuous, gradients flow back through reads and writes into the controller.
- Storage capacity scales with memory size, independent of the controller
- Content-based addressing retrieves slots by similarity to a query
- Location-based addressing supports iteration and sequential access
- Reading and writing are soft and differentiable, enabling end-to-end training
What it enables
With external memory a network can learn algorithm-like behaviors that require holding and manipulating intermediate results: copying a sequence, sorting, associative recall, and answering questions that need many facts. These tasks are hard for a fixed hidden state because they demand more storage than it provides and a way to address specific pieces of it on demand.
Notable designs
The Neural Turing Machine and its successor the Differentiable Neural Computer are the canonical examples, combining content and location addressing with mechanisms for allocating and freeing memory. The soft, content-based read is closely related to the attention mechanism that later dominated transformers, and the idea of retrieving relevant stored information at inference reappears in retrieval-augmented generation.