Workflow Orchestration
Workflow orchestration coordinates multi-step data and model pipelines, running tasks in dependency order with scheduling, retries, and observability.
From scripts to orchestrated pipelines
A data pipeline is a sequence of steps: extract, clean, transform, train, evaluate, publish. Run as a single script, it is brittle: one failure loses all progress, there is no schedule, and no visibility into which step is slow. An orchestrator models the pipeline as a graph of tasks and manages their execution, so steps run in the correct order, failures are isolated, and the whole run is observable.
The dependency graph
Orchestrators represent a pipeline as a directed acyclic graph (DAG). Each node is a task; each edge is a dependency meaning "this task cannot start until that one finishes." Acyclicity guarantees the graph has a valid execution order and cannot deadlock on a circular wait. The orchestrator computes a topological order and runs independent tasks in parallel.
What the orchestrator provides
- Scheduling: run the DAG on a cron-like cadence or on an event
- Retries: re-run a failed task with backoff before failing the run
- Backfills: run past intervals to fill a gap in historical output
- Observability: per-task status, duration, and logs in one view
- Resource limits: cap concurrency so downstream systems are not overwhelmed
Idempotency and partial reruns
A well-designed task is idempotent: running it twice with the same inputs yields the same result and no duplicated side effects. Idempotency is what makes retries and backfills safe. When a run fails halfway, the orchestrator can rerun only the failed and downstream tasks, reusing the outputs of tasks that already succeeded, provided each writes deterministically to a partition it owns.
Static versus dynamic DAGs
Older tools define the DAG statically before the run. Newer tools allow the graph to be built dynamically at runtime, so the number of tasks can depend on the data, for example one task per file discovered. Dynamic graphs are more flexible but harder to visualize and reason about. See DAG scheduling and the medallion architecture.