Evals Overview

Evals measures, judges, and gates AI behavior: is this prompt, model, bot, skill, or agent good enough to ship — and is it still good in production? You author an evaluation once; it runs in CI as a gate and in the background as a drift monitor. Nothing ships on a failing verdict.

What it evaluates

A single evaluation shape covers everything, by naming a subject:

Subject	Evaluates
model / prompt	A model or a gateway prompt/alias
bot	A bot’s responses
skill / agent	A skill or an agent trajectory
traffic	Sampled production traffic
system	Recorded outputs, with no live subject

Two paths

• Deterministic — an explicit evaluation runs straight through the engine: fast, cheap, reproducible. This is what CI gates use. Only in-pipeline judges call a model; everything else is pure computation.
• Agentic — a natural-language intent (“check this assistant handles refunds safely”) is turned into an evaluation by a reasoning loop that plans, generates cases, runs them, probes failures, and writes a report. Its best output is promoted into a deterministic evaluation that becomes tomorrow’s CI gate.

The evaluation model

An evaluation is a Definition: a subject, a dataset of cases, a pipeline of evaluators (per-item checks) and metrics (aggregate scores), and thresholds that produce a pass/fail verdict.

kind: evaluation
metadata: { name: assistant-quality }
spec:
  subject: { kind: prompt, ref: chat.default }
  dataset:
    kind: fixed
    rows:
      - { input: "What's your return policy?", expected: "30 days", output: { value: "Returns within 30 days." } }
  evaluators:
    - { name: correct,     kind: contains, params: { value: "30 days" } }
    - { name: quality,     kind: rubric,   params: { rubric_ref: helpfulness } }
    - { name: safe,        kind: pii,      fail_hard: true }
    - { name: noinjection, kind: prompt_injection }
    - { name: short,       kind: token_count_under, params: { max: 80 } }
  thresholds: { overall_score: 0.7 }

The headline capability is that model-free checks and LLM judges compose in one pipeline: contains (pure), rubric (LLM judge), pii (deterministic detector), prompt_injection (classifier), and token_count_under (token gate) all run together, weighted into one score. See Evaluators & metrics for the full catalog.

Verdict

An evaluation fails if the weighted overall score falls below overall_score, or any per-evaluator or metric threshold is missed, or a fail_hard check fails. Evaluators run cheapest-first, and a hard fail short-circuits the rest of a case.

Gating and drift

• Gating — CI (and the platform’s own publish flows for prompts, skills, and bots) call the gate synchronously with a subject version and get a verdict. A gate timeout is a fail (fail-closed).
• Drift — evaluations run continuously against sampled traffic and on a schedule; results feed drift detectors that flag when behavior degrades. Remediation is always developer-triggered — the block detects and flags, it never retrains or rolls back on its own.

Where things sit

Item	Value
Binary	ai-eval.svc
CLI	kis ai eval
Author in	evaluation Definition YAML, beside the subject it tests
Models	judge and classifier models via the AI gateway

Continue with:

• Evaluators & metrics — the full catalog of checks
• API & CLI — running evaluations and gating
• Operations — config, drift, and baselines
• Error Codes