Heartbeat and Watchdog Supervision

A watchdog turns silence into safety: any controller that stops proving it is alive causes a trip, so a hung process fails safe instead of failing quiet.

Liveness as a safety signal

A controller that crashes and stops sending commands can look identical to a controller that is holding steady. The watchdog removes that ambiguity: every reflex and supervisory node must periodically prove liveness by petting a hardware watchdog. If the heartbeat stops arriving within the window, the watchdog assumes the worst and trips to the safe state.

def watchdog(last_pet_ns, now_ns, window_ns):
    # miss the window by any margin -> trip; silence is not safe
    if (now_ns - last_pet_ns) > window_ns:
        return 'TRIP'
    return 'ok'

def pet_deadline(loop_period_ns, misses_allowed=2):
    # allow a small number of missed ticks before declaring dead
    return loop_period_ns * (misses_allowed + 1)

Design rules

• The heartbeat must exercise the real work path, not a dedicated timer thread, so a hung control loop is detected.
• The watchdog is an independent device; the process it guards cannot disable it.
• The trip window is a small multiple of the loop period, not a generous timeout.
• A watchdog trip latches and requires a deliberate reset, so problems are investigated, not auto-cleared.

Windowed watchdogs also catch the opposite fault: a heartbeat arriving too early can indicate a runaway loop. A good watchdog therefore trips both on too-late and too-soon petting, bounding the loop period from both sides.

Watchdogs are layered like everything else: node-level watchdogs guard individual controllers, and a plant-level watchdog guards the aggregate heartbeat so that a coordinated failure of several nodes is also caught. Each watchdog is independent of the process it supervises and defaults to trip, so the failure of a watchdog itself does not leave a node unguarded. The result is that no controller can silently stop mattering; silence anywhere in the reflex tier resolves to a safe trip.

Watchdog supervision composes with the ML-independent failsafe: loss of heartbeat is one of the conditions that drives a hardwired trip. It also feeds the liveness input consumed by tier separation when the supervisory link goes silent.