Multimodal Fusion Strategies

Multimodal models combine signals from different modalities at different depths, trading simplicity against the richness of cross-modal interaction.

Where to combine

A multimodal network must decide where two or more streams of information meet. The classic taxonomy is early, late, and intermediate fusion, distinguished by the depth at which representations are merged. The choice affects how much cross-modal interaction the model can learn, how robust it is to a missing modality, and how much compute each stream requires.

Early fusion

Early fusion concatenates or interleaves the raw or lightly processed inputs and passes them through a single shared network. This lets the model learn fine-grained interactions from the start but assumes the modalities are aligned and available together, and it is sensitive to differences in scale and sampling rate between modalities.

Late fusion

Late fusion processes each modality with its own encoder and combines only the final representations or predictions, for example by averaging scores or concatenating embeddings before a small head. It is modular, tolerant of a missing modality, and easy to train, but it cannot model interactions that require the two streams to influence each other's internal features.

Intermediate and attention-based fusion

Intermediate fusion exchanges information at one or more middle layers. Cross-attention is the dominant mechanism: queries from one modality attend to keys and values from another, letting text condition on image regions or the reverse. This preserves separate encoders while enabling deep, learned interaction, and it scales naturally to more than two modalities.

• Early fusion: maximal interaction, brittle to misalignment and missing inputs
• Late fusion: modular and robust, limited interaction
• Cross-attention: flexible interaction with separate encoders
• Contrastive alignment as in CLIP is a form of late alignment in a shared space

Practical considerations

Modalities differ in dimensionality, noise, and how much each contributes to a task, so models often normalize and reweight streams and use modality dropout to avoid over-reliance on one input. Systems like CLIP show that even simple alignment in a shared space is powerful, while cross-attention fusion, drawing on the same attention machinery as transformers, supports richer grounding when the task demands it.