Operations

Run modes

workflow.svc                  # orchestrator — HTTP API + gRPC, the normal server
workflow.svc agent            # worker agent
workflow.svc local            # orchestrator + agent in one process (development)
workflow.svc version

An orchestrator additionally starts an embedded agent if the boot config contains a localagent: section. In local mode that section is mandatory.

Flags: -f/--config (default .workflow.yaml), --port, --dc, --cluster, --serviceinstanceid, --configurl, --configapikey, --servicediscoveryurl, --servicediscoverykey, --rootcacert, --svccert, --svckey, --jwtpublickey, --jwtprivatekey, --zerotrust.

Boot configuration

port: 8080
grpc: 9090                     # REQUIRED in orchestrator mode — boot aborts without it
serviceid: workflow-01
loadtenants: []                # tenants to warm at start

orchestrator:
  queue:
    maxdepth: 1000             # 0 or negative = unlimited
    drain_interval: 2s
    drain_batch_limit: 0       # 0 = unlimited per pass
  store:
    workitem: database         # database | memory
    affinity: database
    checkpoint: database

localagent:                    # optional on orchestrator, REQUIRED in local mode
  id: agent-01
  serverurl: https://workflow.internal:9090
  heartbeatduration: 15s
  grpcaddress: 10.0.0.5:9091
  httpaddress: 10.0.0.5:8081
  infrahost: 10.0.0.5
  tags: [gpu, build]
  models: []

Keep the three orchestrator.store.* values on database in production — that is what makes queue, affinity, and loop checkpoints survive an orchestrator restart. memory is for single-node development only.

Storage

Per tenant — workflow instances and node runs live in each tenant’s own schema.
Service schema — the work queue, affinity bindings, loop checkpoints, and the agent registry. The service holds its own reserved tenant entry in config; its dbschema decides where these land.

Schema is embedded and layered with the product’s workflow/extend/ plus per-tenant extension rows (augment-only — a tenant may add entities and fields, never shadow them). DDL goes through the plan/apply plane; apply requires superadmin.

Boot ordering

The orchestrator waits for its first meta callback before it will serve — no tenant engine exists until definitions arrive. In-memory stores are created first and then upgraded to database-backed and recovered once the datastore is confirmed ready. If recovery fails it is a warning, not a fatal: the service silently falls back to memory, which means a subsequent restart loses queued work. Watch the boot log for that.

Queue and capacity

When no agent is free or matching, work is queued rather than rejected, and a drain loop retries on a push signal plus a ticker. Callers see 202 on start. If the queue is at maxdepth, start returns 429 with Retry-After: 10.

Inspect with GET /workflow/queue (tenant) or GET /admin/queue (cross-tenant). Cancel a single queued item with DELETE /workflow/queue/:id.

Agent failure

When an agent dies the orchestrator clears its affinity bindings, finds the instances it was serving, and either:

resumes — remaining tasks are reassigned to healthy agents (the default), or
restarts — the whole instance runs again, when the run was started with restart-on-failure: true.

Choose restart only for idempotent workflows.

Retries

Retries are declared per node, not globally:

retry:
  max_attempts: 3
  backoff_base: 500ms
  backoff_max: 5s

Delay is backoff_base × multiplier^attempt, capped at backoff_max, with jitter. After the final attempt the node’s error: route is taken. Setting continueonerror: true at flow level keeps the run going past a failed task.

Failure modes

Symptom	Cause / action
`503` on start	No free agent. Retry after 5s, or scale the fleet
`429` on start	Queue at `maxdepth`. Retry after 10s; raise `maxdepth` or add agents
`400` on start mentioning selectors	No agent can satisfy the requested `tag`/`model`/`gpu`. Fix the selector or label an agent
Boot aborts: gRPC port missing	Orchestrator mode requires `grpc:` in the boot config
Runs vanish after an orchestrator restart	Stores fell back to memory — check the boot log for a store-recovery warning
Signal rejected	The signal name doesn’t match a declared Wait event, or a `required` field is missing
Instance stuck in waiting	Its Wait node has no matching signal and no `timeout` — add a timeout and an `error:` route
Map node runs serially	Parallelism is `min(items, max_concurrency, available workers)` — add workers

Shutdown

On SIGINT/SIGTERM: stop accepting work → drain agents (30s) → drain the orchestrator (30s) → stop gRPC → release agents and close stores → stop HTTP. In-flight nodes are given the drain window to finish.

Health

GET /health and GET /ready. Supervised deployments register the service with default state up and config .workflow.yaml.