ReLU and Variants

The rectified linear unit is the default hidden activation of deep learning: cheap, non-saturating, and easy to differentiate.

The rectified linear unit

ReLU is f(x) = max(0, x). It passes positive inputs unchanged and clips negatives to zero. Its derivative is 1 for x > 0 and 0 for x < 0, which makes both the forward and backward passes trivially fast. Because the positive-side gradient is exactly 1, error signal propagates through many layers without shrinking, addressing the vanishing gradient problem that plagued sigmoid networks.

Sparsity and dead units

ReLU produces sparse activations: at any input, roughly half the units may output zero. Sparsity can help representation, but it has a failure mode. A unit whose weights push its pre-activation permanently negative outputs zero for all inputs and receives zero gradient, so it never recovers. This is the dying ReLU problem, often triggered by too large a learning rate.

Leaky and parametric variants

Leaky ReLU replaces the flat negative region with a small slope: f(x) = x for x > 0 and alpha·x for x <= 0, with alpha a small constant like 0.01. PReLU learns alpha as a parameter. Both keep a nonzero gradient on the negative side so dead units can revive. ELU and SELU use smooth exponential curves on the negative side and can center activations closer to zero.

import numpy as np
def relu(x): return np.maximum(0, x)
def leaky_relu(x, a=0.01): return np.where(x > 0, x, a*x)
def elu(x, a=1.0): return np.where(x > 0, x, a*(np.exp(x)-1))

Practical guidance

Plain ReLU remains a strong default for convolutional and fully connected hidden layers. If a large fraction of units die, lower the learning rate, adjust initialization, or switch to Leaky ReLU. Transformers and many modern vision models prefer the smoother GELU or SiLU, which behave like ReLU for large inputs but curve gently near zero, sometimes improving optimization.

• Derivative of 1 on the positive side keeps gradients alive.
• Cheap to compute forward and backward.
• Dead units are the main failure mode.
• Leaky/parametric variants and smooth curves address it.