Dense Connectivity (DenseNet)

DenseNet connects each layer to every earlier layer by concatenation, maximizing feature reuse and gradient flow with few parameters.

Every layer sees every earlier layer

DenseNet takes connectivity further than residual networks. Within a dense block, each layer receives the concatenated feature maps of all preceding layers as its input, and its own output is passed on to every subsequent layer. Where a residual block adds its input to its output, a dense block concatenates, so features are preserved intact rather than summed. A block of L layers therefore has L(L+1)/2 direct connections instead of L.

Growth rate and feature reuse

Each layer contributes a small, fixed number of new feature maps, called the growth rate, often just a few dozen channels. Because every layer can access all features produced before it, the network does not need to relearn or copy information forward, so each layer can be narrow. This aggressive feature reuse lets DenseNet reach strong accuracy with notably fewer parameters than a comparable plain or residual network.

• Concatenation preserves earlier features rather than summing them
• A small growth rate keeps each layer narrow and parameter-efficient
• Strong implicit deep supervision, since early features reach the loss directly
• Transition layers with pooling and 1x1 convolution reduce size between blocks

Managing the concatenation

Because inputs concatenate, feature-map count grows within a block, so DenseNet is organized into several dense blocks separated by transition layers. A transition layer applies a 1x1 convolution to compress the channel count and a pooling operation to reduce spatial size, keeping the model's width and resolution manageable between blocks.

# within a dense block
features = [x]
for layer in layers:
    out = layer(torch.cat(features, dim=1))   # see all prior features
    features.append(out)
x = torch.cat(features, dim=1)

Comparison with residuals

Both DenseNet and residual networks create short paths from early layers to the output, easing gradient flow and enabling depth. The difference is additive versus concatenative combination: residuals are memory-light and now dominant, especially in transformers, while DenseNet's concatenation maximizes reuse and parameter efficiency at the cost of higher memory for the growing feature stack. DenseNet remains a strong choice where parameter count is the binding constraint.