API & CLI

Every route is authenticated and CPET-scoped. Run results are the evaluation Run — per-item results, per-evaluator scores, aggregate metrics, the overall score, the verdict, and provenance.

HTTP API

Method	Path	Purpose
POST	/v1/evals	Register a Definition (validated, signed)
GET	/v1/evals · /v1/evals/:name	List / fetch definitions and versions
POST	/v1/evals/:name/runs	Trigger a run
POST	/v1/gate	Synchronous CI gate — {eval, subject_version} → verdict
POST	/v1/agentic	Turn an intent into an evaluation (async) → run id
GET	/v1/runs/:id · …/report · …/diff	Results, narrative report, baseline diff
POST	/v1/evals/:name/baseline	Promote a run to the baseline
POST	/v1/evals/:name/promote	Promote an agentic-produced definition into the registry
—	/v1/datasets · /v1/rubrics	Manage datasets and rubrics
GET	/v1/drift · /v1/drift/:eval	Drift status and flags

The CI gate

The gate is the enforcement point — a subject version is admitted only on a passing verdict:

POST /v1/gate
{ "eval": "assistant-quality", "subject_version": "v7" }
→ { "verdict": "pass" | "fail", "run_id": "…", "overall_score": 0.81 }

A gate that times out returns fail (fail-closed), so a slow or unavailable evaluation never lets a bad version through. The platform’s own publish flows — prompt promotions, skill publishing, bot deploys — call this gate.

CLI

kis ai eval validate -f eval.yaml            # validate a Definition
kis ai eval run <name> [--local] [-v k=v]    # run it (--local uses the embedded engine, no service)
kis ai eval gate <name> --subject-version v7 # CI gate; exit code reflects the verdict
kis ai eval generate "<intent>" [--run] [--promote]   # agentic evaluation
kis ai eval results <run-id>                 # scores + verdict
kis ai eval report <run-id>                  # narrative report
kis ai eval diff <name> --baseline <run-id>  # regression diff vs a baseline
kis ai eval baseline <name> --run <run-id>   # set the baseline
kis ai eval drift status [<name>]            # drift flags

--local runs the identical engine a production gate uses, so a result on your laptop, in CI, and in the service differ only in which backends are wired.

Agentic evaluation

Given an intent, the agentic path plans an evaluation, generates cases, runs them through the same engine, probes failures, and synthesizes a report:

kis ai eval generate "verify the support bot never shares another customer's order details" --run

The valuable artifact is the deterministic evaluation it produces — promote it (--promote) and it becomes a repeatable CI gate.

Running embedded

The evaluation engine can also run inside another service or a flow, with no evaluation service in the loop — compile a Definition and run it directly, injecting the subject runner and the model/embedder/classifier backends. This is how flows and other blocks self-evaluate.