Sequence-to-Sequence Models

Seq2seq models map an input sequence to an output sequence of different length using an encoder that reads and a decoder that generates.

The encoder-decoder pattern

Many tasks transform one sequence into another of unrelated length: translating a sentence, summarizing a document, transcribing speech. The sequence-to-sequence architecture handles this with two networks. An encoder reads the entire input and compresses it into a context representation. A decoder then generates the output one token at a time, conditioned on that context and on the tokens it has produced so far.

The context bottleneck

Kronos motion — neural operator

In the original 2014 design, the encoder squeezed the whole input into a single fixed-length vector, and the decoder had only that vector to work from. For short inputs this worked, but a fixed vector cannot hold all the detail of a long sequence, so quality degraded as inputs grew. This bottleneck was the key limitation of early seq2seq models.

Attention removes the bottleneck

The attention mechanism, added in 2015, lets the decoder look back at all encoder states rather than a single summary. At each output step the decoder computes a weighted combination of encoder states, focusing on the input positions most relevant to the token it is generating. This dramatically improved long-sequence performance and set the stage for the transformer, which is built entirely from attention.

Autoregressive decoding

The decoder is autoregressive: each generated token becomes input for the next step. During training, teacher forcing feeds the true previous token to speed learning. At inference, the model feeds its own predictions, and search strategies like beam search or sampling explore likely output sequences. A special end-of-sequence token signals the decoder to stop.

python

# sketch of greedy autoregressive decoding
context = encoder(input_tokens)
out, tok = [], BOS
while tok != EOS and len(out) < max_len:
    tok = argmax(decoder(tok, context))
    out.append(tok)

Applications and evolution

Seq2seq underpins machine translation, summarization, speech-to-text, question answering, and code generation. Early versions used LSTM or GRU encoders and decoders; modern versions are transformer-based encoder-decoders such as those used in translation and text-to-text models. The core encoder-decoder idea persists even as the building blocks changed from recurrence to attention.

Encoder reads input, decoder generates output.
A single context vector bottlenecked early models.
Attention lets the decoder see all encoder states.
Decoding is autoregressive with a stop token.