Validating a rule set

The problem it solves

Loading a rule set checks its syntax. It does not check that findings.Add(a, b, c, d, e) resolves to a method taking five arguments.

That is discovered at execution, against a real document — and because a failed evaluation aborts the whole run, one unresolvable call costs every other rule’s output for that page. A rule set can be registered, deployed and running for months while a rule inside it has never once completed.

That is not hypothetical. Twenty-nine deployed, enabled validation rules called findings.Add with five positional strings against a method that took a structure the grammar could not build. Nothing looked, so nobody found out.

Validation is the thing that looks.

What it checks

Every call site in the rule text is resolved against the facts the engine has registered, before any document exists.

Diagnostic	Severity	Means
<fact> has no method <Method>	error	A misspelling or a method that does not exist. The message names the closest match
<fact>.<Method> takes N, called with M	error	Right name, wrong number of arguments
<fact>.<Method> is deprecated; use <replacement>	warning	Resolves and runs, but there is a better call
duplicate rule name, already defined in <file>	error	Two rules share a name across the files in one set

Error means the call will abort a page if it is ever reached. Warning means it works today.

What it deliberately does not check

Unknown receivers are skipped. A local variable, or a fact supplied per execution rather than at registration, cannot be resolved at load time — and flagging every one of them would bury the real findings under noise. Only facts the engine actually knows about are checked.

Arity is reported only when the checker is confident about the call shape. A check that cries wolf gets switched off, and then it protects nothing.

Note

This is a lint, not a type checker. It reads call sites out of the rule source rather than walking the parser’s syntax tree, which is version-coupled and undocumented. That trade is deliberate: a check that runs in milliseconds against every rule is worth more than a complete one that breaks whenever the engine’s internals move.

Why generated rules need it most

A person writing a rule can run it against a document and read the failure. A model generating one needs a verdict it can repair against, and it needs it fast, deterministic and without a document — running the whole pipeline to discover a typo is none of those.

Validation gives exactly that: a list of call sites that will not resolve, with the nearest correct name, in milliseconds. It is the feedback loop that makes generated rules correctable rather than merely plausible.

Running it

kis rules --rules ./rules --validate

No --input is needed and nothing is executed against a document — that is the point. The document namespace is registered anyway so document calls resolve, but no document is loaded.

Add --strict to make it a gate:

kis rules --rules ./rules --validate --strict

--strict exits non-zero when there is any error-level diagnostic or any file that failed to parse, which is what makes it usable as a pipeline step.

Reading the output

The output has three parts, in the order they matter.

Files that did not parse at all come first, because they are a stronger finding than anything validation itself can report: those rules are not in the set to be checked.

2 file(s) did not parse at all (excluded below):
  rules/totals.grl: got 3 error(s) in grl the script
  rules/dates.grl: got 1 error(s) in grl the script

Caution

A file that does not parse is excluded from the check that follows. A clean validation run does not mean every rule is sound — it means every rule that loaded is sound. Read this section first.

Errors, one at a time, with the snippet that caused them. Each names a call that will abort a page at runtime, so each is worth reading individually:

rules/classify.grl:42: error: findings.Add takes 5, called with 1
    findings.Add(Finding{RuleID: "x"});

Warnings, rolled up by kind. A single deprecated call is rarely interesting; the count of them is a migration checklist:

12 diagnostic(s): 1 error(s), 11 warning(s)

Warnings by kind:
      8  bbox.Get is deprecated
      3  FindIPhraseFuzzy is deprecated

Where to run it

Wherever a rule set is accepted — a pipeline step before delivery, or a check in whatever authors rules — so a set that cannot execute is rejected before it reaches a document rather than after.

--validate --strict is the whole gate. It needs no documents, no fixtures and no running service, so it costs a second and it is the difference between finding an unresolvable call in CI and finding it in production output.

See also

• Writing rules — structure, salience, the output envelope
• Functions — the callable surface a call is resolved against
• Errors — what an unresolved call looks like at execution
• Troubleshooting — symptom-first diagnosis