Backpropagation Through a Two-Layer Network

Derive and code the gradients of a small neural network by applying the chain rule backward from the loss.

The network

A two-layer net computes h = sigma(W1 x + b1), then y = W2 h + b2, with loss L = (1/2)(y - t)^2 for target t. Training needs the gradient of L with respect to every weight, which backpropagation computes efficiently by reusing intermediate results.

Forward then backward

• Forward: compute z1 = W1 x + b1, h = sigma(z1), y = W2 h + b2, and L.
• Backward: dL/dy = y - t.
• dL/dW2 = (dL/dy) h^T, dL/db2 = dL/dy.
• Propagate: dL/dh = W2^T (dL/dy); dL/dz1 = dL/dh * sigma'(z1); dL/dW1 = (dL/dz1) x^T.

import numpy as np
sig=lambda z:1/(1+np.exp(-z)); dsig=lambda z:sig(z)*(1-sig(z))
x=np.array([[0.5],[0.1]]); t=np.array([[1.0]])
W1=np.random.randn(3,2)*0.5; b1=np.zeros((3,1))
W2=np.random.randn(1,3)*0.5; b2=np.zeros((1,1))
for step in range(2000):
    z1=W1@x+b1; h=sig(z1); y=W2@h+b2
    dy=y-t
    dW2=dy@h.T; db2=dy
    dh=W2.T@dy; dz1=dh*dsig(z1); dW1=dz1@x.T; db1=dz1
    for p,g in [(W1,dW1),(b1,db1),(W2,dW2),(b2,db2)]:
        p-=0.5*g
print('output:',round(float(y),4))  # -> ~1.0

Why backward

Computing each gradient independently would repeat the same forward work many times. Backpropagation is reverse-mode automatic differentiation: one forward pass caches the activations, one backward pass reuses them, and the whole gradient costs about the same as two forward passes regardless of the number of parameters.

Practical notes

Numerically check gradients against finite differences the first time you implement them. Watch for vanishing gradients through saturating sigmoids - modern nets use ReLU activations partly for this reason - and initialize weights with a scale that keeps activations from exploding or collapsing across layers.