Long Short-Term Memory

LSTM cells add a gated memory pathway that lets recurrent networks carry information across hundreds of steps without vanishing gradients.

The cell state

The central innovation of the LSTM is a separate cell state c_t that runs along the sequence with only minor, gated changes. Information can travel down this pathway almost unchanged, which gives gradients a nearly linear route backward and largely defeats the vanishing gradient problem. The hidden state h_t is derived from the cell state and serves as the cell's output.

The three gates

Kronos motion — rnn lstm

Forget gate: decides what fraction of the old cell state to keep, using a sigmoid.
Input gate: decides how much of a new candidate value to write into the cell state.
Output gate: decides how much of the cell state to expose as the hidden state.

The update equations

Each gate is a sigmoid over the current input and previous hidden state, producing values in (0, 1) that act as soft switches. A tanh produces the candidate update. The cell state updates as c_t = f_t · c_{t-1} + i_t · c_candidate, a gated blend of old memory and new information, and h_t = o_t · tanh(c_t).

python

import numpy as np
def lstm_step(x, h, c, W, U, b):
    z = W @ x + U @ h + b            # stacked gate pre-activations
    i, f, o, g = np.split(z, 4)
    i, f, o = 1/(1+np.exp(-i)), 1/(1+np.exp(-f)), 1/(1+np.exp(-o))
    g = np.tanh(g)
    c = f * c + i * g
    h = o * np.tanh(c)
    return h, c

Why the gates help

Because the forget gate multiplies the cell state and the input gate controls writing, the network learns when to remember, overwrite, or ignore. A gate set near one preserves memory across many steps; near zero it clears it. This adaptive control lets an LSTM hold a fact from early in a sequence until it becomes relevant much later, which a plain RNN cannot reliably do.

Use and legacy

LSTMs dominated sequence modeling for years, powering machine translation, speech recognition, and time-series forecasting before transformers arrived. They remain strong for streaming and low-latency tasks with modest data, where their constant per-step memory and strictly sequential processing are advantages. The GRU is a lighter alternative with fewer gates and comparable performance on many problems.

A gated cell state gives an unbroken gradient highway.
Forget, input, and output gates control memory flow.
Learns when to keep, write, or expose information.
Still competitive for streaming and small-data sequences.