Operations

Bootstrap

Build produces ai-eval.svc; run it with a config file (default .ai-eval.yaml). Ports and TLS are chassis-managed (TLS by default). The service persists definitions, runs, and baselines, and records an immutable verdict ledger.

Configuration

engine:
  default_concurrency: 8
  judge_defaults: { model: gateway:judge-default-v1, temperature: 0.0 }
registry:
  require_signatures: true          # only run signed definitions
harness:
  budgets:
    planner_max_cost_usd: 1.00      # agentic path cost ceilings
    probe_max_cost_usd: 5.00
gate:
  timeout: 10m
  timeout_verdict: fail             # fail closed
traffic:
  max_sample_rate: 0.10             # cap on production sampling
  retention: 30d
drift:
  detector: page_hinkley
  window: 7d
results:
  retention: 365d

Judge and classifier models are referenced as gateway:<model> and resolve through the AI gateway. Definitions, datasets, and rubrics are versioned, signed artifacts.

Baselines and regressions

Promote a good run to the baseline for an evaluation; later runs diff against it:

kis ai eval baseline assistant-quality --run <run-id>
kis ai eval diff assistant-quality --baseline <run-id>

The diff reports the overall-score delta, per-evaluator and per-metric deltas, and which specific cases regressed or were fixed — so a prompt or model change surfaces exactly what it moved.

Continuous and scheduled evaluation

Beyond on-demand and CI runs, evaluations run:

• Continuously against a sampled fraction of production traffic (capped by traffic.max_sample_rate), and
• On a schedule for periodic compliance runs.

Both feed the dashboards and the drift detectors, which flag when a subject’s behavior degrades over the configured window. Drift is detect-and-flag only: remediation (rolling back a version, retraining, adjusting a prompt) is always a developer decision.

Verdict ledger

Every verdict is written to a tamper-evident audit ledger — what was evaluated, by which definition version, against which dataset, with what result. This is the record CI, compliance, and post-incident review rely on.

Health

GET /admin/v1/health and stats. Observability includes per-run traces and metrics (runs and verdicts per evaluation, gate latency, pass-rate trends, judge cost, and drift flags).

Dependencies

The AI gateway (judge and classifier models), the data layer and a time-series store (run results + drift), an audit service (the verdict ledger), and — for gating — the publish flows of the prompts, bots, and skill blocks.