Gated Recurrent Unit

The GRU is a streamlined gated recurrent cell that merges the LSTM's memory and gating into fewer parameters with often-similar performance.

A simpler gated cell

The gated recurrent unit, introduced in 2014, keeps the LSTM's central benefit, gated control of information flow, but with a simpler design. It has no separate cell state; the hidden state itself carries memory. It uses two gates instead of three, which means fewer parameters and slightly faster training while retaining the ability to model long-range dependencies.

The two gates

• Update gate z_t: decides how much of the previous hidden state to keep versus how much new candidate to admit, combining the LSTM's forget and input roles.
• Reset gate r_t: decides how much of the past hidden state to use when forming the new candidate, allowing the cell to drop irrelevant history.

The update

The new hidden state is a linear interpolation controlled by the update gate: h_t = (1 - z_t)·h_{t-1} + z_t·h_candidate. When z_t is near zero the state is copied forward almost unchanged, preserving memory and passing gradients cleanly; when near one it is replaced by fresh information. The candidate is computed with the reset gate modulating the contribution of the previous state.

python

import numpy as np
def gru_step(x, h, Wz, Wr, Wh, Uz, Ur, Uh):
    sig = lambda a: 1/(1+np.exp(-a))
    z = sig(Wz @ x + Uz @ h)
    r = sig(Wr @ x + Ur @ h)
    h_cand = np.tanh(Wh @ x + Uh @ (r * h))
    return (1 - z) * h + z * h_cand

GRU versus LSTM

GRUs have fewer parameters and train a little faster; LSTMs have an extra gate and a dedicated cell state that can help on some tasks with very long dependencies. Empirically the two perform comparably across most sequence problems, and the better choice depends on the dataset. A common practice is to try both and pick whichever validates better.

When to reach for a GRU

GRUs suit smaller datasets where fewer parameters reduce overfitting, latency-sensitive or on-device settings, and time-series or streaming signals where a compact recurrent model is preferable to a transformer. For continuous scientific sensor streams that must be processed in order with bounded memory, a GRU offers a lightweight, capable option.

• Merges LSTM gating into two gates and one state.
• Update gate interpolates old and new state.
• Fewer parameters, faster training, comparable accuracy.
• Good for small data and streaming.