Transfer Learning and Pretraining
Reusing knowledge from large or related datasets to help models learn from scarce fusion data.
The scarce-data problem again
Deep models need lots of data, and fusion has little. Transfer learning addresses this by starting from a model trained on a larger or related dataset, then adapting it to the target task, so the target task needs far fewer examples than training from scratch.
Forms of transfer
- Fine-tuning a model pretrained on a data-rich source task
- Feature reuse: use a pretrained model's representations as inputs
- Multi-task learning: train jointly on related tasks
- Self-supervised pretraining on abundant unlabeled data
Self-supervised pretraining
Unlabeled diagnostic data are abundant even when labels are scarce. Self-supervised pretraining, learning to predict masked or future parts of signals, builds useful representations from this unlabeled data. A downstream task with few labels then fine-tunes from a strong starting point rather than random weights.
When transfer helps and when it hurts
Transfer helps when source and target share structure. When they differ too much, a pretrained model can transfer wrong assumptions, negative transfer, and underperform a model trained from scratch. Cross-machine transfer in fusion sits on this edge, and must be validated on target-machine data, not assumed.
Honest expectations
Transfer learning stretches scarce data but does not conjure information about regimes never observed. A model pretrained on existing devices and fine-tuned for a new design still cannot know what only that design's data could reveal. Its predictions in genuinely new regimes remain extrapolations, to be flagged as such and weighted accordingly in any decision.