Distributed Tracing & Correlation

Correlation and causation IDs thread a single shot's activity across every service so latency and causality can both be reconstructed.

One shot, many services

A single breeder shot touches diagnostics ingest, the twin, the copilot, the gating pipeline, and actuator adapters. To understand latency and causality across all of them, every event and span carries a shared correlation_id (the shot or campaign) and a causation_id (the specific event that triggered it). These are the same IDs the lineage bus uses, so tracing and audit share one identifier scheme.

Correlation vs causation

correlation_id = shot_00421          // groups everything for this shot
causation_id   = <parent event id>   // forms a causal tree, not just a group
// diagnostic -> twin_pred (caused_by diagnostic)
//            -> proposal  (caused_by twin_pred)
//            -> command   (caused_by proposal + envelope_ok + approval)

Latency budget across the gate

Tracing measures how long a candidate spends in each gating stage: reconcile, schema, rules, envelope, rate, approval. This matters because the gate sits between the twin's proposal and L1; if the human-approval stage dominates, that is visible and expected, while an unexpectedly slow envelope check is an alert. Tracing never sits on the L1 microsecond loop, which is off the event plane.

What traces reveal

• End-to-end path of a burner plug-field correction from measurement to actuation.
• Which stage rejected a breeder command and how long the pipeline took to say no.
• Consumer lag hotspots feeding backpressure decisions.

Deterministic reconstruction

Because IDs are recorded on the immutable log, a trace can be rebuilt exactly during replay, unlike sampled traces that are lost after the fact. Every causal edge is preserved, so 'why did this command fire' is answered from the causation tree rather than inferred from timestamps.