Data Ingestion Patterns
Ingestion is how data enters a system; the main choices are push versus pull, batch versus continuous, and full versus incremental.
Getting data in
Ingestion is the first stage of any data system: moving data from where it is produced into where it will be processed and stored. The patterns chosen here shape latency, reliability, and load. Getting ingestion wrong causes data loss and duplication that no later stage can fully repair.
Push versus pull
- Push: the source sends data to the system as events occur; low latency, but the source must handle backpressure.
- Pull: the system fetches from the source on a schedule; simpler control, but adds polling delay.
Batch versus continuous
- Batch: collect a bounded chunk and ingest it periodically; efficient, higher latency.
- Continuous: ingest records as they arrive via a stream; low latency, more moving parts.
Full versus incremental
A full load re-ingests the entire source each time, simple but wasteful for large sources. An incremental load ingests only what changed since the last run, using a marker such as a timestamp or an increasing id (change data capture). Incremental loading scales but requires reliable tracking of what has already been consumed to avoid gaps or duplicates.
Exactly-once and idempotence
Networks fail mid-transfer, so ingestion must handle retries without creating duplicates. The durable approach makes writes idempotent, keyed so that re-ingesting the same record has no additional effect, and tracks committed positions so a restart resumes cleanly. See message brokers.
In a fusion program
Diagnostic acquisition typically buffers a pulse locally then pushes it to storage between shots, a burst-oriented pattern; machine telemetry may stream continuously; simulation outputs are ingested in batches as runs complete. Each source gets the pattern that matches its rate and criticality, all feeding the common archive.