Encoder-Decoder Architecture

The encoder-decoder pattern separates understanding an input from generating an output, a template reused across many modern architectures.

Two halves, two jobs

An encoder-decoder model splits a task into comprehension and production. The encoder reads the input and builds an internal representation of it. The decoder takes that representation and generates the output, one element at a time. This division suits any problem that maps one structured object to another of possibly different length or form: translating a sentence, captioning an image, transcribing speech, or reconstructing a signal.

The bridge between them

Kronos motion — architecture

How the decoder accesses the encoder's representation defines the architecture. Early sequence models passed a single summary vector, which bottlenecked long inputs. Modern encoder-decoder transformers use cross-attention, letting the decoder attend to all encoder outputs at every generation step. This flexible bridge is why current models handle long and complex inputs far better than their predecessors.

Instantiations across domains

Seq2seq with RNNs: LSTM/GRU encoder and decoder for early machine translation.
Encoder-decoder transformer: attention throughout, used for translation and text-to-text tasks.
Autoencoder: encoder compresses to a code, decoder reconstructs the input.
U-Net: convolutional encoder-decoder with skip connections for dense image prediction.

Encoder-only and decoder-only

Not every task needs both halves. Encoder-only models like BERT build representations for understanding tasks such as classification and retrieval. Decoder-only models like GPT generate text autoregressively and have become the dominant form for large language models, since a single stack can both read a prompt and continue it. The full encoder-decoder remains preferred when input and output are distinct sequences, as in translation.

python

# encoder-decoder generation (sketch)
# memory = encoder(source_tokens)
# y = [BOS]
# while y[-1] != EOS:
#     logits = decoder(y, memory)   # decoder cross-attends to memory
#     y.append(argmax(logits[-1]))

Why the pattern endures

The encoder-decoder split is a durable design because it cleanly separates two concerns and lets each half be built from whatever components suit the domain: recurrence, convolution, or attention. The same skeleton, with different internals, spans translation, image generation, and dense prediction. Understanding it gives a map for reading many architectures as variations on reading, then generating.

Encoder comprehends, decoder produces.
Cross-attention is the flexible bridge between them.
Realized as seq2seq, transformers, autoencoders, U-Net.
Encoder-only and decoder-only variants drop one half.