Hypernetworks

A hypernetwork is a network that generates the weights of another network, letting one model produce parameters adapted to a context or task.

A network that writes weights

A hypernetwork is a neural network whose output is the weights of a second network, called the target or primary network. Instead of learning the target's parameters directly, training learns the hypernetwork, which then produces those parameters from some input, such as a layer index, a task descriptor, or a context vector. This indirection lets the model generate parameters that adapt to the situation and often expresses many target weights with far fewer hypernetwork parameters, a form of weight compression through sharing.

How it is used

In the simplest case a small hypernetwork takes an embedding for each layer of a deep target network and emits that layer's weight matrix, so the whole target is generated from a compact set of layer embeddings plus the shared hypernetwork. In a conditional case the hypernetwork takes a task or context input and produces weights specialized to it, enabling fast adaptation without storing a separate full model per task. The target network then runs normally on its data using the generated weights.

# generate a target layer's weights from a context vector
W = hypernet(context).view(out_dim, in_dim)   # produced weights
y = x @ W.t()                                   # target uses them

• One hypernetwork can produce weights for many layers or many tasks
• Often far fewer learned parameters than the target it generates
• Enables conditional, context-dependent behavior without swapping models
• Gradients flow through the generated weights back into the hypernetwork

Applications

Hypernetworks appear in several settings. In meta-learning and few-shot learning they generate task-adapted weights from a small support set. In continual learning they can produce per-task weights from task embeddings, reducing interference between tasks. They also underlie some parameter-efficient adaptation methods, where a small network emits the low-rank updates applied to a frozen backbone. Conceptually they share the theme of conditional computation with mixture-of-experts routing, which selects among fixed experts rather than generating weights, and both aim to make a model's behavior depend flexibly on its input.