Adagrad

Adagrad gives each parameter its own learning rate, shrinking it in proportion to the accumulated size of that parameter's past gradients.

Per-parameter adaptation

Plain gradient descent uses one global learning rate for every parameter, which is a poor fit when different parameters need different step sizes, as in sparse data where some features appear rarely. Adagrad, introduced in 2011, adapts the learning rate individually for each parameter, taking larger steps for infrequently updated parameters and smaller steps for frequently updated ones.

The update rule

Adagrad accumulates the sum of squared gradients per parameter into a running total G. The update divides the global step size by the square root of this accumulator: theta_i <- theta_i - (eta / sqrt(G_i + epsilon)) * g_i, where g_i is the current gradient for parameter i and epsilon prevents division by zero. A parameter that has seen large or frequent gradients builds up a large G_i and therefore takes progressively smaller steps.

import numpy as np

def adagrad(grad_fn, theta, eta=0.1, eps=1e-8, iters=1000):
    G = np.zeros_like(theta)
    for _ in range(iters):
        g = grad_fn(theta)
        G += g*g                       # accumulate squared gradients
        theta -= eta * g / (np.sqrt(G) + eps)
    return theta

Strengths

Adagrad's per-parameter scaling makes it strong on sparse problems such as natural-language and recommendation models, where rare features would otherwise be under-trained. It effectively removes the need to hand-tune a learning-rate schedule for each parameter, since the accumulator adapts automatically to the observed gradient magnitudes. It also has clean regret guarantees when viewed as an online convex optimization algorithm.

The decay problem

Adagrad's defining weakness follows directly from its design: because G only ever grows, the effective learning rate only ever shrinks, eventually becoming so small that learning stalls before reaching a good solution. This monotone decay motivated RMSprop and Adadelta, which replace the ever-growing sum with an exponentially decaying average of squared gradients, keeping the adaptive benefit while preventing the learning rate from vanishing.