Recovery and capacity

Workers are disposable, but lost work must not consume capacity forever. Sonar couples liveness- based recovery with bounded dispatch and durable result processing.

Worker liveness

Workers pulse Sonar periodically with their identity and routing tags. A worker is eligible for placement only while it is active and its latest pulse is recent. The same liveness view supports recovery; it is not the Inventory crawl refresh described by the recency phase.

Stale in-flight recovery

Every reconciliation cycle includes a global pass over old in-flight tasks:

• Published without a worker start for more than 10 minutes returns to Pending.
• Running for more than 10 minutes returns to Pending only when its worker is absent from the live registry.

The second rule deliberately uses worker liveness instead of a fixed command-duration timeout. A long-running command on a worker that still pulses is left alone. After backend restart, the age floor gives the in-memory registry time to repopulate before old running tasks are judged orphaned.

Redispatch creates fresh attempt credentials and clears result acceptance state from the abandoned attempt.

Spot interruption and shutdown

An AWS worker watches the instance metadata interruption endpoint. On a spot-revocation notice it stops accepting new jobs, notifies Sonar, and lets shutdown handling drain active work. Workers also stop consuming before normal timeout or process shutdown.

If a machine still disappears, stale-task recovery remains the correctness backstop.

Routing capacity

For each reconciliation tick, Sonar computes capacity from live physical workers:

• Tag capacity starts from a configurable multiple of workers advertising that tag, minus tasks already dispatched there.
• Global capacity counts each physical worker once, even when it advertises several tags, and subtracts all Published and Running tasks.
• Step capacity applies the step's optional maxConcurrent to its own in-flight tasks.

Pending tasks are water-filled across serviceable target tags in small batches. This avoids committing a multi-tag machine several times and gives independent ready steps a chance to share a worker pool. If no live worker advertises any target tag, tasks remain Pending and Sonar logs the starved step instead of publishing to nowhere.

Output backpressure

Result acceptance and output processing are separated. The callback durably enqueues finalization work; a fixed number of background consumers lease rows with expiration and renewal. This bounds simultaneous MinIO reads and database upserts without losing a burst when the backend restarts.

Failed finalization retries with exponential backoff up to a bounded attempt count. Work that continues to fail is marked poisoned and retained for inspection; it does not block all later work or let the scan claim a successful result.