Recurrent Neural Networks

RNNs process sequences one step at a time, carrying a hidden state that acts as memory of everything seen so far.

The recurrent idea

A recurrent neural network reads a sequence element by element, updating a hidden state at each step. The same weights are applied at every time step, so the network is a loop unrolled over time. The hidden state h_t summarizes the past and, combined with the current input x_t, produces the next state: h_t = phi(W_x·x_t + W_h·h_{t-1} + b). This lets one compact model handle sequences of any length.

Outputs and configurations

Kronos motion — confinement time

Many-to-one: read a whole sequence, output one label (sentiment classification).
Many-to-many aligned: output at every step (part-of-speech tagging).
Many-to-many unaligned: read then generate (sequence-to-sequence translation).

Parameter sharing across time

Because the same weight matrices are reused at every step, an RNN has a fixed parameter count regardless of sequence length, and it can generalize a pattern learned at one position to any other position. This is the temporal analog of the weight sharing that convolutional networks apply across space.

python

import numpy as np
def rnn_step(x_t, h_prev, Wx, Wh, b):
    return np.tanh(Wx @ x_t + Wh @ h_prev + b)

The training difficulty

RNNs are trained by backpropagation through time, which unrolls the loop and propagates gradients backward across many steps. Repeatedly multiplying by the recurrent weight matrix makes gradients shrink toward zero (vanishing) or blow up (exploding). Vanishing gradients prevent the network from learning long-range dependencies, the central weakness that motivated gated architectures like LSTM and GRU.

Status today

Transformers have replaced RNNs for most large-scale language tasks because they parallelize across a sequence, while an RNN must process steps in order. RNNs remain relevant for streaming and low-latency settings, small on-device models, and time-series problems where strictly sequential processing and a compact state are advantages. For scientific signals arriving as a continuous stream, an RNN's constant-memory update can be a good fit.

Hidden state carries memory across sequence steps.
Same weights reused at every step.
Trained by backpropagation through time.
Vanishing gradients limit long-range memory.