Activation Functions

Activation functions inject the nonlinearity that gives deep networks their expressive power, and their shape controls how gradients flow.

The role of activations

Between the linear parts of a network sits a nonlinear activation applied element-wise. Without it, stacked layers collapse into a single linear transformation. The activation decides how a neuron responds to its input, whether it saturates, and how large a gradient it passes backward during training. Choice of activation therefore affects both what the network can represent and how easily it learns.

Saturating vs non-saturating

Kronos motion — burner power flow

Early networks used saturating functions like sigmoid and tanh, which flatten for large positive or negative inputs. Flat regions produce near-zero gradients, so signal fades in deep stacks: the vanishing gradient problem. Modern networks favor non-saturating functions like ReLU, whose gradient is a constant 1 for positive inputs, keeping training signal alive through many layers.

A survey of common choices

Sigmoid: squashes to (0,1); useful for probabilities but saturates.
Tanh: squashes to (-1,1); zero-centered, still saturates.
ReLU: max(0, x); cheap, non-saturating, but can produce dead units.
Leaky ReLU / PReLU: small negative slope keeps dead units alive.
GELU / SiLU: smooth curves used in transformers and modern vision models.
Softmax: normalizes a vector into a probability distribution over classes.

Gradient behavior

During backpropagation the derivative of the activation multiplies the incoming gradient. If that derivative is small across many layers, the product shrinks toward zero. ReLU's derivative is exactly 1 on the active side, which is a large part of why deep networks became trainable once ReLU replaced sigmoid in hidden layers.

python

import numpy as np
def relu(x): return np.maximum(0, x)
def sigmoid(x): return 1/(1+np.exp(-x))
def tanh(x): return np.tanh(x)
def gelu(x): return 0.5*x*(1+np.tanh(0.79788*(x+0.044715*x**3)))

Choosing one

As a default, use ReLU or a smooth variant like GELU in hidden layers, sigmoid for a single binary output, and softmax for multi-class outputs. The output activation must match the loss and the meaning of the target; hidden activations are chosen for gradient flow and speed.

Nonlinearity is what makes depth useful.
Saturating functions risk vanishing gradients.
Match the output activation to the task and loss.
ReLU-family functions dominate hidden layers.