Validating a rule set

The problem it solves

The parser checks a rule set’s syntax. It does not know whether findings.Add(a, b, c) names a method that takes three arguments.

A call that cannot resolve is otherwise discovered when the rule runs. Because a failed evaluation aborts the whole run, one unresolvable call costs every other rule’s output for that input, and a rule that never matched in testing can still abort production traffic on its first match.

The load-time check resolves every call before anything runs, so an unresolvable call is reported when the rule set loads instead.

Where it runs

Where	What happens to a set with an error
rules.svc, on every load of a product’s rules/ folder	The load fails; the service keeps serving what it had, and GET /rulesets reports why
kis rules check	Loads a product’s rules/ folder as rules.svc does; exits non-zero naming each error
kis rules -r … --scope <name>	Loads the product’s folder the same way; the run does not start
kis rules -r … --validate	Lists every diagnostic; --strict exits non-zero
kis rules -r … (a plain run)	Not checked. The set runs, and fails at the first call that cannot resolve

What it checks

Every call on a binding the engine knows (in, out, findings, audit, strings, map, bbox, …) is resolved against that binding.

Diagnostic	Severity	Means
<binding> has no method <Method> (did you mean <Nearest>?)	error	A misspelling or a method that does not exist. The suggestion appears when a name is close
<binding>.<Method> takes N argument(s), called with M	error	Right name, wrong number of arguments
in.<Method>'s first argument must be a quoted path, e.g. "status"	error	A bare fact name where in.* or out.* expects the path as a string
<binding>.<Method> returns <type>, which a rule variable cannot hold	error	A local variable assigned a map or a list from a binding call; the next use of it aborts
<binding>.<Method> is deprecated; use <replacement>	warning	Resolves and runs, but there is a better call

Error means the call will abort an execution if it is ever reached. Warning means it works today.

A rule name used twice in one set is not a diagnostic: the file that reintroduces it fails to parse, and is reported with the parse failures below.

What it deliberately does not check

Not checked	Why it matters
Argument types	math.Min(10, 20) passes the check and aborts when it runs: write 10.0. A float default needs a decimal point: in.Double("x", 0.0)
A method called on a value a call returned	bbox.Doc().Top(10) checks bbox.Doc; a typo in .Top is found only when the rule runs
Unqualified functions	Retract, Complete, Now(), Max() and the rest are not checked; Repeat() (it is agenda.Repeat()) loads and fails when it runs
Unknown receivers	A local variable, or a fact supplied per execution, cannot be resolved at load time, and flagging every one would bury the real findings
Facts the caller will not send, dot access on a map fact	Both depend on the input, not on the rule text
A call whose arguments it cannot count	A call split across lines, for one. Arity is reported only when the checker is confident about the call shape

A check that cries wolf gets switched off, and then it protects nothing.

Note

This is a lint, not a type checker. It reads call sites out of the rule source rather than walking the parser’s syntax tree. That trade is deliberate: a check that runs in milliseconds against every rule is worth more than a complete one that breaks whenever the engine’s internals move.

Why generated rules need it most

A person writing a rule can run it against an input and read the failure, a model generating one needs a verdict it can repair against, and it needs it fast, deterministic and without an input: running the whole pipeline to discover a typo is none of those.

Validation gives exactly that: a list of call sites that will not resolve, with the nearest correct name, in milliseconds. It is the feedback loop that makes generated rules correctable rather than merely plausible.

Running it

For a product’s rules/ folder, the way rules.svc loads it:

kis rules check              # ./rules
kis rules check products/shop

rules: 1 folder scope(s), 0 static scope(s): they load
  claim-triage             folder, 1 rule file(s)

A folder that does not load names the scope and each error, and exits 1:

rules "claim-triage/**" do not validate:
rules/claim-triage/typo.grl: rule Typo, line 3: strings has no method ToUper (did you mean ToUpper?)

For a folder or file of rules outside a product:

kis rules -r ./rules --validate
kis rules -r ./rules --validate --strict

No input is needed and nothing is executed. The document binding is registered anyway, so bbox.* calls are checked too. Without --strict the command exits 0 whatever it found. --strict exits non-zero when there is any error-level diagnostic or any file that failed to parse, which is what makes it usable as a pipeline step.

Reading the --validate output

The output has three parts, in the order they matter.

1 file(s) did not parse at all (excluded below):
  /work/rules/dates.grl: got 1 error(s) in grl the script

CheckTotals:5: error: findings.Add takes 5 argument(s), called with 3
    findings.Add("CheckTotals", "ERROR", "totals");

5 diagnostic(s): 1 error(s), 4 warning(s)

Warnings by kind:
      3  in.GetStr
      1  in.MemSet

Files that did not parse at all come first, because they are a stronger finding than anything validation itself can report: those rules are not in the set to be checked.

Caution

A file that does not parse is excluded from the check that follows. A clean validation run does not mean every rule is sound. It means every rule that loaded is sound. Read this section first.

Errors, one at a time, with the line that caused them, as <rule>:<line>. Each names a call that will abort an execution when it is reached, so each is worth reading individually.

Warnings, rolled up by the call they are about. A single deprecated call is rarely interesting; the count of them is a migration checklist.

Where to run it

Wherever a rule set is accepted, a pipeline step before delivery, or a check in whatever authors rules, so a set that cannot execute is rejected before it reaches an input rather than after.

kis rules check for a product, or --validate --strict for a folder of rules, is the whole gate. It needs no inputs, no fixtures and no running service, so it costs a second and it is the difference between finding an unresolvable call in CI and finding it in production output.

See also

• Writing rules: structure, salience, the output envelope
• Functions: the callable surface a call is resolved against
• Errors: what an unresolved call looks like at execution
• Troubleshooting: symptom-first diagnosis