Neural Ordinary Differential Equations

Neural ODEs treat a network's depth as continuous time, defining the transformation as the solution of a differential equation parameterized by a neural network.

Depth as continuous time

A residual network updates its hidden state by h(t+1) = h(t) + f(h(t)), which resembles one step of the Euler method for a differential equation. Neural ordinary differential equations take this limit seriously: instead of a fixed number of discrete layers, they define the hidden state's evolution by dh/dt = f(h(t), t, theta), where f is a neural network. The output is the state at the final time, computed by a numerical ODE solver rather than by stacking layers.

The adjoint method

Backpropagating through the internal operations of an ODE solver would require storing every intermediate step, which is memory-heavy. Neural ODEs instead compute gradients with the adjoint sensitivity method, which solves a second, augmented ODE backward in time to recover the gradients. This gives training a memory cost that does not grow with the number of solver steps, at the price of solving another ODE and some numerical error.

• Adaptive solvers spend more steps where the dynamics are stiff and fewer where they are smooth
• A single continuous model can be evaluated at any output time
• Constant memory in the number of function evaluations via the adjoint
• Function evaluations are the natural cost measure, replacing layer count

Continuous normalizing flows

A notable application is density modeling. In a continuous normalizing flow, the change in log-probability along the trajectory is governed by the trace of the Jacobian of f, which the ODE tracks alongside the state. This turns an intractable determinant into an integral, enabling flexible invertible generative models.

from torchdiffeq import odeint
def f(t, h): return net(h)          # net is an nn.Module
h_T = odeint(f, h0, t=torch.tensor([0.0, 1.0]))[-1]

Strengths and limits

Neural ODEs are a natural fit for irregularly sampled time series and for physical systems whose behavior is genuinely continuous, since the model can be queried at arbitrary times. Their limitation is that a single ODE trajectory cannot cross itself, so plain neural ODEs cannot represent certain functions; augmented neural ODEs add extra dimensions to lift this restriction. The link to fixed-depth residual nets is described in residual connections.