Temporal Convolutional Networks
Temporal convolutional networks model sequences with stacked dilated causal convolutions, offering a parallel alternative to recurrent networks for time-series tasks.
Convolutions for sequences
A temporal convolutional network (TCN) applies one-dimensional convolutions along the time axis to model sequential data. It is built from two ingredients: causal convolutions, which ensure the output at time t depends only on inputs at or before t, and dilated convolutions, which skip input positions to expand the receptive field without adding layers or losing resolution. Together they let a TCN see far into the past while training in parallel, unlike a recurrent network that must process one step at a time.
Exponential receptive field
Stacking layers with dilation factors 1, 2, 4, 8, and so on grows the receptive field exponentially with depth. A network with dilations doubling across L layers and kernel size k covers a history of roughly (k-1) times (2^L - 1) time steps. This means a modest stack can capture long-range dependencies that would require many recurrent steps, and it does so with a fixed, predictable memory footprint.
- Parallel training, since all time steps are computed at once
- Stable gradients compared to backpropagation through long recurrences
- A fixed and controllable memory cost regardless of sequence length
- Residual blocks make deep stacks trainable
Residual blocks
Each TCN block typically contains two dilated causal convolutions with weight normalization, a nonlinearity, and dropout, wrapped in a residual connection. The residual path lets very deep TCNs train reliably and lets each block learn a refinement of the sequence representation rather than a full transformation.
TCN versus recurrence and attention
Against recurrent networks, TCNs often match or beat accuracy while training faster and avoiding vanishing-gradient issues. Their receptive field is fixed by architecture, so they cannot in principle attend to an unbounded past the way an RNN's hidden state can carry information indefinitely, though in practice the exponential field usually suffices. For very long or content-dependent context, attention-based models and state-space models are alternatives. The core dilation mechanism is shared with WaveNet.