Operations

Building, configuring, running, and tuning AI Flow (aiflow.svc).

Build

# from the AI Flow source directory
go build -o dist/aiflow.svc .

Or use the repo build task (build.yaml), which injects version/commit ldflags and produces dist/aiflow.svc:

kis build            # runs the "build" task in build.yaml

Check the version:

./dist/aiflow.svc version

Run tests with coverage (per build.yaml):

kis build test

Run

AI Flow is a cluster node built on lib-cluster/boot. A single binary can run as an orchestrator and/or an agent; both gRPC server credential sets are enabled by default.

./dist/aiflow.svc --config /etc/ai-flow/.ai-flow.yaml

The service gates its HTTP surface on the first tenant workflow load (the definition-load gate), so a fresh process won’t accept /workflow/request until at least one tenant’s definitions are loaded.
Agent nodes declare which tenants they serve:

./dist/aiflow.svc --tenants "acme:flow:prod:default,globex:flow:prod:default" --exclusive false

Configuration

Config is loaded via the chassis (bootconfig.Load("ai-flow"), default file .ai-flow.yaml). There is no config file in the repo — supply one at deploy time (or inject a config map for tests). Keys consumed:

Service / infra

Key	Purpose
`httpaddress`	HTTP listen address.
`grpcaddress`	gRPC listen address (orchestrator / agent server).
`infrahost`	Infra/monitoring address advertised by the agent.
`capabilities`	ML model proxies this node exposes (routing by `model`).
`tags`	Agent routing tags (matched by `selectors.tag`).

Vault & secrets

Key	Purpose
`vault.url`, `vault`	Vault connection. DB passwords and secrets resolve from here (literal fallback only in dev).

Model proxies

Key	Purpose
`mlproxies`	Model-server proxy definitions.
`forceproxyrestart`	Restart proxies on load (default `true`).

Orchestrator / queue

Key	Purpose
`orchestrator.queue.maxdepth`	Overflow work-queue depth. `0` or negative = unlimited.
`orchestrator.store.workitem`	Overflow-queue backend: `database` (default) or `memory`.
`orchestrator.store.affinity`	Affinity-binding backend: `database` (default) or `memory`.

Persistence backends. In an AI Flow deployment, execution instances and runs are persisted to PostgreSQL (the platform data layer), not in-memory — Use database stores in production so the overflow queue and affinity bindings survive restarts.

Hard-coded constants (not yet YAML-exposed)

These live in code (the agent fabric); changing them requires a build:

Constant	Value
Message chunk size	3.5 MiB
Circuit-breaker max failures / reset	5 / 30 s
Flow-control max in-flight	1000
Per-agent outbound queue	1000
Write/requeue timeout	5 s
Heartbeat jitter	±10%
gRPC keepalive (client)	`Time=30s`, `Timeout=10s`, `PermitWithoutStream`
gRPC keepalive (server)	+ `MinTime=5s`

Database & migrations

Each tenant maps to its own PostgreSQL schema (resolved via chassis config). Connections set app.tenant_id (and userid) session variables so row-level security applies.
The entity model (mlflow_*) is compiled into the binary from embedded entity YAML and built into a process-wide schema at boot. Malformed entity YAML is fatal at startup — catch it in CI.
Credential rotation is handled live: TenantConfigChanged drops the affected tenant’s cached engines/pools; the schema is untouched.

Workflow definitions (deploying flows)

Workflow definitions are not deployed with the binary. They’re delivered per tenant via the meta.svc product-config system under product.workflow.data (a list of YAML docs). On each meta refresh, AI Flow re-parses and rebuilds that tenant’s definitions and engine.

To ship a new flow: add/update its YAML in the tenant’s product.workflow.data, then trigger a meta refresh. Verify with GET /workflow/config (lists the tenant’s definitions/tasks).

Local definitions (development only)

A box with no meta server has no definition source at all, which means no definition provider and no execution engine for any tenant — every workflow route answers “not found”. For that case the service config can name a directory of workflow YAMLs:

workflows:
  localdir: workflows        # relative paths resolve against the bootstrap file
loadtenants:
  - default:ai-flow:dev:default

Every .yaml/.yml directly under localdir is parsed at boot and published to each tenant key in loadtenants — full four-part customer:product:env:tenant keys, because that is what the engine and every request-path lookup key by. Definitions go through the same refresh path as meta-delivered ones, so behaviour after load is identical.

Leave localdir unset in any deployment: it is a development stand-in, not a second supported delivery channel, and it does not re-read the directory after boot.

A worked example, dev-sample.yaml, uses only control-flow nodes (assign, succeed), which the engine runs in-process — a definition with ordinary task nodes still needs a connected agent to make progress.

Health & readiness

Readiness is effectively “first workflow load complete” — the HTTP surface comes up only once that gate opens.
Use GET /workflow/config (per tenant) as a functional check that definitions are loaded, and GET /agents to confirm agents are connected.
GET /workflow/queue shows overflow depth; a persistently non-empty queue means insufficient agent capacity for the offered load.

Tuning

The authoritative operator doc is the agent fabric’s tuning guide. Key knobs and signals:

Symptom	Likely cause	Lever
Growing overflow queue	Not enough agents / wrong tags	Add agents; check `selectors` match agent `tags`/`models`.
`selector.fail.count` rising with `NoMatchingAgent`	Flows request a tag/model no agent has	Fix the selector or deploy an agent with that capability.
Agents flapping unhealthy under load	Heartbeats starved	Already mitigated by `streamSendMu`; verify heartbeat interval vs. task duration.
`backpressure.count` climbing	Flow-control cap hit	More agents, or reduce fan-out `max_concurrency`.
Slow selection (`selector.duration`)	Large agent fleet / churn	Investigate heartbeat volume; consider agent sharding by tenant.

Heartbeat interval: shorter = faster failure detection, more traffic. Staleness threshold is 2× the interval.
max_concurrent_tasks per agent (0 = unlimited) bounds per-agent load and drives the overflow queue.
Fan-out: cap Map max_concurrency so a single flow can’t monopolize the fleet.

Observability

Orchestrator (kis.agent): connected, connect.count, disconnect.count, heartbeat.count, state_change.count, backpressure.count.
Selector (kis.selector): duration, success.count, fail.count.
Agent (OTel gauges): CPU/memory/GPU/disk usage, data.inflow/outflow.
the flow engine EventLog: append-only audit (AffinityBound, AgentFailover, SignalReceived, DataPipeRecord, …) for replay.
Instance logs: GET /workflow/logs, and engine GetInstanceLog / GetInstanceRunsLog / GetInstanceMLResponseLog.

All metric registries fall back to no-op instruments, so metric-registration failure never panics the hot path.

Graceful shutdown

OnShutdown calls the v2boot Shutdown, invalidating all cached engines and closing pools.
The orchestrator Drain(timeout) marks agents draining and broadcasts a drain control message; the flow engine performs a 3-phase drain that completes in-flight work before exit.

Security checklist

gRPC mTLS configured (both server and client certs present) — otherwise it falls back to insecure with a warning. Confirm certs load (cacert/servercert/serverkey [+ clientcert/clientkey]).
Vault reachable; no literal DB passwords in production config.
Per-tenant access rules configured for the granular permissions (workflow.start, workflow.resume, workflow.signal, workflow.cancel, workflow.status, workflow.logs, workflow.config, workflow.list, agents.list, queue.status, queue.cancel).
Superadmin routes (/admin/*) restricted appropriately.
RLS verified: cross-tenant reads return nothing.

Known issues

GET /workflow/request and GET /workflow/response read the id query param in their instanceid/conversationid filter branches — filter by filter= or id= until fixed.
Some code/comments still use the legacy names ai-ml / horus / the platform identity service; the service is ai-flow.

See the design spec for the full architecture and [orchestration](/blocks/ai/AI Flow/orchestration/) for the agent fabric internals.