Distributed workflow engine

The distributed workflow engine turns a reusable scan definition into structured Inventory updates. It plans a directed acyclic graph, queues ready tasks, matches them to live workers, collects raw output, and ingests declared results.

Responsibility

The engine owns:

• workflow, scan, step, dependency, and task execution models;
• readiness and fan-out/fan-in planning;
• RabbitMQ dispatch and worker routing;
• raw artifact transfer through MinIO;
• result parsing and output-table ingestion;
• reliability, consistency, recovery, and capacity behavior.

It does not own the meaning of programs, scopes, or assets. Those definitions belong to Inventory.

Execution flow

Workflow definition
        │ create scan
        ▼
Reconciler builds ready work
        │
        ▼
RabbitMQ task queue
        │
        ▼
Worker executes one command
        │
        ├────────▶ MinIO raw artifact
        │
        ▼
Result callback → parse → idempotent upsert → Inventory

The engine uses at-least-once delivery. It makes the effects of retries idempotent rather than assuming each message or callback arrives only once.

Read this subsystem

The engine's operating qualities are documented separately so their guarantees do not blur: