Evaluators & Metrics

An evaluation pipeline is built from two kinds of stage:

• Evaluators run per case and return a score (0–1 by default), optionally a hard pass/fail.
• Metrics run over the whole dataset and return one aggregate number.

Common evaluator params: field (a dot-path into the output), weight (contribution to the overall score), and fail_hard (a failure short-circuits and fails the case).

Model-free evaluators

Pure computation — no model, fully reproducible.

Comparison: equals, not_equals, approx_equals (tolerance, relative), contains, not_contains, matches (pattern), starts_with, ends_with, in_range (min, max), length_within, one_of / not_one_of (values), json_path_equals (path, value), exists, is_type (type).

Operational: latency_under (max_ms), cost_under (max).

Text overlap (graded): exact_match, rouge_1, rouge_2, rouge_l, meteor, token_f, char_f, bleu (max_n), edit_distance.

Extensible: conforms (validate against a schema / schema_ref), script (a custom code check).

AI evaluators

Model-requiring checks — the model, embedder, and classifier are supplied by the platform (via the AI gateway).

Judges (LLM-as-judge):

Kind	Params	Purpose
rubric	rubric_ref	Score against a named rubric
factuality		Judge factual accuracy against the expected answer
pairwise		Compare two candidate outputs
reference_semantic		Embedding cosine similarity to the expected answer

Safety:

Kind	Params	Purpose
pii		Detect PII (deterministic — email, SSN, phone, card)
prompt_injection		Detect prompt-injection attempts
jailbreak		Detect jailbreak attempts
refusal	threshold (0.5)	Detect refusals

Token/cost gates: token_count_under (max), token_cost_under (max, price_per_token).

A missing model backend produces a structured error captured on the case — the model-free evaluators in the same pipeline still run and score.

Aggregate metrics

Computed over the full result set — for classification, regression, and ranking subjects.

Classification: accuracy, balanced_accuracy, precision, recall, f1, specificity (average: macro | micro | weighted | binary), mcc, cohens_kappa, auc_roc, auc_pr, log_loss.

Regression: mae, mse, rmse, r2, mape, explained_variance.

Ranking / retrieval: ndcg, precision_at_k, recall_at_k (k), mrr, map, hit_rate.

Datasets

A dataset is fixed (inline rows, or a JSONL file by ref) or generated (produced by the agentic path). Each row is a case: an input, an expected value, optional metadata, and — for offline evaluation of recorded behavior — a pre-recorded output. A case with only an input requires a live subject, which the engine runs to produce the output before scoring.

Thresholds

thresholds:
  overall_score: 0.7            # weighted mean across evaluators
  per_evaluator: { quality: 0.6 }
  metrics: { accuracy: 0.9 }
  fail_on_any_hard_fail: true

The verdict is fail if any configured threshold is missed or a hard-fail occurred, otherwise pass.