Errors are structured { "code": "…", "message": "…" }.
| Code | Meaning |
|---|
definition_invalid | The evaluation Definition failed validation |
param_invalid | An evaluator or metric parameter is invalid |
evaluator_unknown | Unknown evaluator kind |
metric_unknown | Unknown metric kind |
dataset_unresolved | The dataset ref could not be loaded |
subject_runner_missing | The subject needs a live runner that wasn’t supplied |
label_mismatch | Predictions and labels don’t line up for a metric |
script_unavailable | A script evaluator was used with no script host |
conformer_missing | A conforms evaluator was used with no schema conformer |
evaluator_failed | An evaluator errored while running a case |
| Code | Meaning |
|---|
judge_failed | An LLM judge errored |
judge_model_missing | A judge kind was used with no model wired |
embedder_missing | reference_semantic was used with no embedder |
classifier_missing | A safety classifier was used with no classifier wired |
tokenizer_missing | A token gate was used with no tokenizer |
A missing backend fails only the affected evaluator — the rest of the pipeline still runs and scores.
| Code | Meaning |
|---|
eval_not_found | No evaluation by that name |
eval_unsigned | The definition isn’t signed and signatures are required |
subject_runner_unavailable | The subject can’t be run |
dataset_forbidden | Reference-based evaluators aren’t allowed on traffic rows |
baseline_missing | No baseline set for a diff |
promotion_invalid | An agentic definition failed promotion checks |
sample_rate_exceeded | Traffic sampling above the configured cap |
tenant_scope | The evaluation is outside the caller’s tenant scope |
gate_timeout | The CI gate timed out — treated as a fail (fail-closed) |
