Troubleshooting

Every entry below is a symptom you can observe, what it means, a command that confirms it, and the fix. Failures in this block are quiet by design in several places — an empty result is far more common than an error — so read the “confirm” step before changing rules.

Symptom index

What you see	Section
got N error(s) in grl the script	Rule sets that will not load
ruleset <name> not found: specified knowledge base name and version not exist	Rule sets that will not load
ruleset <name> not found: expression ... is not on the clone table	Rule sets that will not load
A rule you can see in the file never fires	Rules that never fire
Output differs between identical runs	Rules that never fire
got non existent key <fact>	Executions that abort
error while executing rule <R>. got ... — a fault in a then	Executions that abort
evaluating expression in rule '<R>' the when raised an error. got ...	Executions that abort
this node identified as "<fact>" is not referencing to an object	Executions that abort
<fact>[string-><key>] is not an array nor map	Executions that abort
the GruleEngine successfully selected rule candidate ... after 5000 cycles	Executions that abort
context canceled / context deadline exceeded	Executions that abort
in.GetStr returns "" for a fact that is present	Accessors that return nothing
Error: 'rules' is not installed.	Command line
no files matched pattern / no prefixes resolved	Command line
recovered : runtime error: invalid memory address or nil pointer dereference	Command line
got non existent key bbox from POST /rule	The deployed service
400 rule engine not initialized from a ready instance	The deployed service
A 400 whose body is invalid tenant, not JSON	The deployed service

Rule sets that will not load

got N error(s) in grl the script

loading rules: loading rules from /srv/rules: /srv/rules/b.grl: got 1 error(s) in grl the script

Means — the rule text failed to compile. Two unrelated causes produce the identical message: a syntax error, or a rule name that is already registered in that rule set.

Confirm — the count is the only detail you get. The parser does record a line:column message per error, but neither the CLI nor rules.svc surfaces it. Bisect by loading one file at a time:

kis rules -r ./rules/a.grl -i ./in

Fix — rule names must be unique across every .grl file loaded in one run, because a directory is pooled into a single rule set and the knowledge base keys rules by name. Rename the collision, or fix the syntax. Struct literals are the most common syntax trap — see struct literals.

ruleset <name> not found: specified knowledge base name and version not exist

Means — no compiled rule set exists under that name at the engine’s default version.

Confirm — check the name you executed against the name you loaded. On the CLI the rule set name is rules for kis rules and --ruleset (default rules) for kis ocr page / kis ocr doc. Over HTTP it is the name field of the request body, which must match a rules[].name in the definitions the service loaded.

Fix — correct the name. If the name is right, the rules never loaded: see the deployed service.

Note

Rule sets load and execute at the engine’s default version, and that version is fixed. Neither kis rules nor rules.svc takes a version argument — both always pass the default — so a version mismatch is never the cause of this message. If the name is right, the rule set never loaded.

ruleset <name> not found: expression ... is not on the clone table

ruleset invoice not found: expression  "NEW" is not on the clone table - E(EA(A(C(string->"NEW"))))

Means — the rule set is corrupt. A rule whose name was already registered was re-added with a changed body. The knowledge base is add-only: it refuses the duplicate name, but the partial mutation leaves that rule set unexecutable for the life of the process.

Confirm — look for adding rule failed in the service log immediately before the first failing request; the service logs it at error level with the rule set name. Re-adding identical text produces the same load error but leaves the rule set working; only a changed body poisons it.

Fix — restart the process. There is no unload, no reload endpoint and no recovery in-band.

Caution

Rules cannot be updated in a running service. Rolling out a changed rule is a restart operation, not a hot reload — the service logs the rejected update at error level and keeps serving the poisoned rule set until it is restarted.

Rules that never fire

A downstream rule that reads out never fires

rule First salience 10 {
  when !out.Has("tier")
  then out.Set("tier", "gold"); Retract("First");
}
rule Second salience 5 {
  when out.GetStr("tier") == "gold"
  then out.Set("bonus", 10); Retract("Second");
}

Means — condition sub-expressions are memoised. Method calls such as out.Has, out.GetStr and in.MemGet are not detected as data changes, so Second’s cached condition is never re-evaluated and it never becomes runnable.

Confirm — run with --audit. Only First appears in the audit trail.

kis rules -r ./rules -i ./in --audit

Fix — signal the change in the writing rule’s then:

then out.Set("tier", "gold"); Changed("out"); Retract("First");

Forget("out.GetStr(\"tier\")") invalidates the single cached expression instead of the whole collector and works equally well.

The same rule set produces different output on different runs

Means — rules of equal salience fire in Go map-iteration order. Running the same three default-salience rules twelve times produced five distinct firing orders.

Confirm — run the same input twice with --audit and compare the order of the audit entries.

Fix — give every rule whose position matters a distinct salience. Exactly one rule fires per cycle, highest salience first, so distinct salience is the only ordering guarantee.

rule High "high" salience 100 { when true then out.Append("order", "high"); Retract("High"); }
rule Mid  "mid"  salience 50  { when true then out.Append("order", "mid");  Retract("Mid");  }
rule Low  "low"  salience 1   { when true then out.Append("order", "low");  Retract("Low");  }

A run reports success but writes nothing

Means — the rule output map was empty. kis rules and kis ocr page return early on an empty output, so --update leaves the source file untouched and nothing is printed except Processed N/M file(s) successfully (prefix(es) for kis ocr page). There is no “0 rules fired” diagnostic at the default log level.

Confirm — --audit prints the decision trail; an empty trail means no rule matched.

Fix — check the fact names your conditions reference. Input .json top-level keys become top-level fact identifiers, so a rule reads docid == "d1" directly, not through an accessor.

A transform appears to do nothing

Means — an unknown transform name resolves to nothing and returns silently, leaving the value untouched. A malformed ExtractRegex pattern behaves the same way. There is no error and no log line.

Confirm — emit the value straight after the transform and compare it with the input.

Fix — check the transform name against the function reference. Note that a transform only operates on values already staged with in.MemSet — it cannot seed itself from a fact.

Executions that abort and discard everything

Abort-on-failed-evaluation is unconditional, and every non-retracted rule is evaluated on every cycle. One failing rule ends the whole execution and no result is returned — every out.Set that already succeeded is discarded. There is no partial-result mode and no per-rule isolation.

got non existent key <fact>

A missing fact aborts the run from either scope, and the two scopes wrap the message differently. From when:

executing ruleset invoice: evaluating expression in rule 'CheckTotal' the when raised an error. got left hand expression error. got non existent key order

From then:

executing ruleset invoice: error while executing rule CheckTotal. got non existent key vendor

Means — a rule referenced a fact the caller did not supply. The rule does not have to be relevant to the request; if it is in the rule set and not retracted, its when is evaluated on every cycle.

Confirm — both forms name the rule and the fact. Compare that fact name against the keys of the request body or input file.

Fix — guard the fact in the same expression that dereferences it. && short-circuits, so the right-hand side is never evaluated when the guard is false:

rule CheckTotal salience 20 {
  when in.Has("order") && order["Total"] > 1000
  then out.Set("review", true); Retract("CheckTotal");
}

Note

in.Has is the one input accessor that works — it only tests for the key’s presence. The typed getters next to it do not; see accessors that return nothing.

got non existent key log

executing ruleset invoice: error while executing rule Notify. got non existent key log

Means — the log namespace is not bound. It comes from a plugin that no shipped surface enables, so the identifier exists in no execution.

Confirm — the message names the rule that called log.Info.

Caution

log.* is unavailable in rules.svc and on the CLI. A rule that calls it aborts the entire execution. Use audit.Log("RuleName", "message") for a decision trail instead — it is a plain string call and it works. The utility namespaces (util, num, strings, time, math, map, array) come from a different binding step and are always present.

this node identified as "<fact>" is not referencing to an object

executing ruleset invoice: evaluating expression in rule 'D' the when raised an error. got left hand expression error. got this node identified as "order" is not referencing to an object

Means — dot access was used on a map fact. Only structs and pointers to structs support dot access.

Confirm — the message names the fact. Any order.Total form against decoded JSON hits this.

Fix — use bracket access:

when order["Total"] > 1000

<fact>[string-><key>] is not an array nor map

Means — two-level indexing into decoded JSON. The intermediate value is interface-typed, so it cannot be indexed. Assigning it to a rule variable first does not help.

Confirm — doc["header"] alone resolves; doc["header"]["total"] fails.

Fix — stage the intermediate through an accessor whose Go return type is concrete, then index that. out.GetMap returns map[string]any, so the second index succeeds:

rule ReadTotal salience 20 {
  when true
  then
    out.Set("header", doc["header"]);
    out.Set("total",  out.GetMap("header")["total"]);
    Retract("ReadTotal");
}

in.MemSet / in.MemGet does not work as a substitute — MemGet returns any, so indexing its result panics. The alternative is to flatten the facts before you send them, so that every value a rule indexes sits at the top level.

after 5000 cycles

executing ruleset loop: the GruleEngine successfully selected rule candidate for execution after
5000 cycles, this could possibly caused by rule entry(s) that keep added into execution pool but
when executed it does not change any data in context...

Means — a rule re-qualified every cycle until the cap tripped. The whole execution fails and all output is lost.

Confirm — the rule at fault is one whose then does not falsify its own when. A guard that looks self-limiting still loops, because the condition is cached and never re-evaluated as false:

rule SetOnce { when !out.Has("x") then out.Set("x", 1); }   // burns all 5000 cycles

Fix — call Retract("RuleName") in every then, or Complete() to end the execution outright. Treat Retract as mandatory.

Caution

The cycle cap is fixed at 5000 in every shipped binary and cannot be raised by configuration. A rule set that hits it must be fixed, not tuned. The cap is also not a timeout: the counter only advances when a rule fires, so a rule blocked inside a helper runs indefinitely regardless of it.

A rule that writes a finding never loads

loading rules: loading rules from /srv/rules: /srv/rules/checks.grl: got 3 error(s) in grl the script

Means — a struct literal in a rule body. The rule language has no struct-literal syntax, so Finding{...} produces a burst of parse errors. Only the count reaches you; the underlying mismatched input '{' message is not surfaced.

Confirm — delete the findings.Add(...) line and reload. If the count drops to zero, that was it.

Caution

findings.Add(Finding{...}) does not parse and no constructor for it exists. There is no way for a rule to record a finding, so the severity model is unreachable from rule code. Use audit.Log / audit.LogWithData for rule-authored diagnostics. Any example showing the Finding{...} form does not compile.

context canceled / context deadline exceeded

executing ruleset invoice: context deadline exceeded

Means — the caller’s context ended. A cancelled context gives context canceled; an expired deadline gives context deadline exceeded. This is the only wall-clock bound on an execution — the engine checks the context once per cycle and once per rule evaluation, so it stops at the next rule boundary, not mid-rule.

Fix — over HTTP the deadline comes from the X-Api-Timeout header; see the deployed service.

Accessors that return nothing

in.GetStr returns "" for a fact that exists

rule Probe {
  when true
  then
    out.Set("direct",    doctype);
    out.Set("in_has",    in.Has("doctype"));
    out.Set("in_getstr", in.GetStr("doctype"));
    Retract("Probe");
}

With the input {"doctype": "invoice", "pages": 3} this returns {"direct":"invoice","in_has":true,"in_getstr":""}.

Means — the typed accessors receive a reflection wrapper rather than the value, so every internal type assertion fails and each accessor returns its zero value. in.Has is unaffected, which makes the failure look like missing data rather than a broken accessor.

Confirm — run the probe above. in_has is true while in_getstr is empty.

Caution

in.GetStr, in.GetInt, in.GetFloat, in.GetBool and in.GetMap always return the zero value. Reference facts directly by identifier (doctype == "invoice") instead. in.Mem* accessors are unaffected — they read an in-process map — but in.Transform inherits the defect for any key not already staged with in.MemSet.

A value staged in a named memory slot is missing from the result

Means — no memory slot reaches the result on its own; only out is returned. in.GetMemAsMap() is the bulk read-back, and it snapshots the default slot only, so a value written with in.Mem("other").Set(...) is readable within the same execution via in.Mem("other").Get(...) and invisible everywhere else.

Fix — copy the value out explicitly with out.Set(...) before the execution ends.

out.Append replaced a value instead of appending to it

Means — Append concatenates only when the key currently holds a string. If it holds a number, map or bool, the existing value is overwritten.

Fix — do not mix out.Set of a non-string with out.Append on the same key. The separator is also read at append time, so changing it mid-run yields mixed separators within one value.

Command line

Error: 'rules' is not installed.

Error: 'rules' is not installed.
Install with: kvm install <plugin binary>

Means — kis rules, kis ocr, kis bbox and kis hocr2bbox are proxy commands. The kis binary carries only the stubs; the implementation is a separately installed plugin executable.

Confirm — the lookup order is the directory holding the kis binary, then ~/.kisai/bin, then every entry of $KIS_PLUGIN_PATH, then $PATH. First executable match wins.

echo $KIS_PLUGIN_PATH
ls ~/.kisai/bin

Fix — install the plugin, or point $KIS_PLUGIN_PATH at it. See the CLI reference.

Note

Global kis flags must precede the subcommand. Flag parsing is disabled on the proxy commands, so everything after the command name is forwarded to the plugin verbatim.

no files matched pattern <dir>/<glob>

Means — kis rules found nothing to process. This is a hard error, not a no-op.

Fix — check -i and -p. Input discovery is a single non-recursive glob of <-i>/<-p>; -R/--recursive affects rule-file discovery only.

no prefixes resolved (no positional args matched and no files matched pattern ...)

Means — kis ocr page / kis ocr doc resolved zero <prefix>.json + <prefix>.bbox pairs.

Confirm — list the directory. A .json whose .bbox sidecar is missing is dropped silently when it arrives through a directory glob or a file path; only a bare prefix argument errors loudly. Files matching *-hist-audit.json are always skipped.

ls pages/*.json pages/*.bbox

Fix — produce the missing sidecars with kis hocr2bbox, or correct the path.

recovered : runtime error: invalid memory address or nil pointer dereference

executing rules on page1.json: error while executing rule B. got rule engine execute panic on rule B ! recovered : runtime error: invalid memory address or nil pointer dereference

Means — a rule called a document accessor with a document name that is not loaded. There is no “no such document” message; the lookup returns nothing and the method call on it panics.

Confirm — compare the name in the rule against the injected name:

Command	Document name	Override
kis rules -b <file>	bbox	write -b <name>=<file>
kis rules --with-bbox	bbox	--with-bbox-name
kis ocr page	page	--bbox-name
kis ocr doc	doc	--bbox-name

Fix — align the two. A rule written for kis ocr page needs --with-bbox-name page to run under kis rules --with-bbox, and the same rule needs its document sent under the key page when you run it through rules.svc.

*-hist-audit.json files keep appearing, and counts keep growing

Means — the sidecar is written next to every processed input whenever a run produced any history or audit entries. --history and --audit only control whether the same data is also printed. The writer reads the existing file and appends, so repeated runs accumulate duplicates.

Confirm — run the same command twice and count the entries.

Fix — redirect them out of the data directory with --history-audit <dir>, and delete stale ones between runs. kis ocr always skips *-hist-audit.json on input; kis rules does not, so its default *.json pattern re-processes the sidecars it wrote.

--ruleset had no effect

Means — kis rules accepts --ruleset, shows a default of rules in help, and ignores it. The name is fixed. kis ocr page and kis ocr doc honour theirs.

A numeric comparison against a --vars value never matches

Means — -v/--vars values are always injected as strings. -v threshold=10 gives a rule the string "10".

Fix — put numeric facts in a --facts JSON file instead, where the JSON type is preserved.

got non existent key bbox from kis rules

Means — the bounding-box plugin is registered only when you pass -b/--bbox or --with-bbox. Without one of them the bbox name is never bound and the first rule that touches it aborts the whole run. kis ocr page and kis ocr doc always register it.

Fix — add --with-bbox (sidecar next to each input) or -b [name=]<file>.

A batch run exited 1 but some files were rewritten

Means — batch runs are best-effort. Every input is attempted, each failure prints ERROR: <path>: <err>, and the command exits 1 after reporting Processed N/M. With --update the successful files are already written.

Confirm — kis rules prints those ERROR: lines to stdout and finishes with Processed N/M file(s) successfully; kis ocr page prints them to stderr and finishes with Processed N/M prefix(es) successfully. Redirect both streams.

The deployed service

got non existent key bbox over HTTP

{"error":{"code":"","context":null,"message":"executing ruleset invoice: error while executing rule ExtractTotal. got non existent key bbox","meta":null}}

Means — the request carried facts but no document. bbox is bound from the reserved bbox key of the request body; with no bbox object the name resolves to nothing and the first rule that touches it aborts the whole execution with a 400. You do not get a partial result. This is the HTTP equivalent of running kis rules without -b / --with-bbox.

Confirm — the message names the failing rule. Check that the body carries a bbox object, and that the document name inside it is the name the rule passes to bbox.Get / bbox.Has.

Fix — send the document alongside the facts, under the name the rule uses. The value is the same word-box JSON that kis hocr2bbox writes to a .bbox file:

{
  "name": "invoice",
  "doctype": "invoice",
  "bbox": {
    "page": {
      "pages": [
        {"number": 1, "width": 612, "height": 792,
         "words": [{"text": "Total", "box": {"x0": 72, "y0": 100, "x1": 150, "y1": 115},
                    "confidence": 0.98, "page": 1}]}
      ]
    }
  }
}

The bindings a rule then calls — fields, forms, checkboxes, tables, repeaters, matching and confidence — are the ones the CLI binds, under the same names and with the same behaviour. When the rule set calls ExtractForm, switch bbox to the two-key form — __docs for the documents, __schemas for the schemas. See the document rules bridge.

Note

Document input is read inline. The service does not resolve file paths, so a string value under bbox fails to load; send the parsed OCR JSON. A document that will not parse or analyse fails the request with 400 and a plugin bbox prepare: detail, before any rule is evaluated.

The response has no audit trail

Means — the request did not ask for one. The trail is opt-in: without "audit": true on the body the 200 response is the out map alone.

Fix — set "audit": true on the request. The entries come back under a reserved audit key, the same {"rule_id", "message"} records kis rules --audit prints.

curl -sS -X POST https://rules.example.com/rule -H 'Content-Type: application/json' -d '{"name":"invoice-checks","total":1200,"audit":true}'

If an out key called audit disappears from the response, that is the collision — the trail is serialised last and overwrites it. Rename the output key.

The response has no findings, no rule-match count and no timing

Means — findings are always empty, because no rule can record one. The per-key change history, the rules-matched count and the execution duration are computed by the engine and never serialised.

Fix — write everything a caller needs into out. A rule set that detected a critical problem still returns HTTP 200 with whatever output was set; there is no business-failure signal on the wire. To tell “matched but wrote nothing” apart from “nothing matched”, request the trail and count its executed entries.

400 rule engine not initialized from an instance reporting ready

Means — no rule definitions have loaded in this process. Readiness and health are flipped true immediately after routes are registered, before any rule pull completes and without the engine as a dependency, so a load balancer will route traffic to an instance that cannot serve anything.

Confirm — /ready returns ready:true while every /rule call returns this 400.

curl -s http://localhost:8080/ready

Fix — check the service log for repeated rules service configuration load failed lines. The per-tenant load retries on a one-second loop forever and never degrades readiness, so a single unreachable source stalls rule loading indefinitely. See service operations.

A 400 whose body is invalid tenant, not JSON

Means — the tenant middleware rejected the request before the handler ran. Middleware failures are text/plain; only handler failures use the JSON envelope.

Confirm — the four tenancy headers are mandatory on /rule:

curl -X POST http://localhost:8080/rule -H 'Content-Type: application/json' -H 'Authorization: Bearer <token>' -H 'X-Customer: acme' -H 'X-Product: billing' -H 'X-Env: prod' -H 'X-Tenant: default' -d '{"name":"invoice","doctype":"invoice"}'

Fix — send all four, and provision the tenant. X-Tenant carries the tenant segment only — the middleware joins the four headers into the lookup key itself. The same message is emitted for missing headers and for an unknown tenant.

Caution

401 Unauthorized, 400 invalid tenant, 400 config client not initialised, 405 Method Not Allowed and 500 - Internal error are plain text, and 404 has an empty body. A client that parses every response as JSON breaks on the most common failures.

Every error is a 400 with an empty code

{"error":{"code":"","context":null,"message":"executing ruleset invoice: error while executing rule R1. got non existent key vendor","meta":null}}

Means — exactly one error in the service carries a code: rules-json-request-parse-failed. Unknown rule set, poisoned rule set, evaluation fault, cycle-cap abort, missing name and uninitialised engine are all HTTP 400 with code: "".

Fix — match on message. A malformed request is not distinguishable from an unwarmed service by status or code. Response bodies carry no trace or span identifier, so the server log line is the only correlation handle. The full table is on the error reference.

json request parse failed due to {{err}}

Means — the body was not decodable JSON. The placeholder is never substituted; the real decoder message is in meta.error.

Confirm — the body must be a JSON object. An array, a bare string or a number fails the decode.

{"error":{"code":"rules-json-request-parse-failed","context":{},"message":"json request parse failed due to {{err}}","meta":{"error":"invalid character 'b' looking for beginning of object key string"}}}

name of rule to execute not given

Means — the decoded object had no usable name field. name selects the rule set, and the whole body — including name — is handed to the engine as facts, apart from the reserved bbox key, which becomes the document input instead.

A long request never times out

Means — the timeout middleware runs in header-driven mode. Without an X-Api-Timeout header it disables itself, and a 0ms value disables it too. There is no request-body size limit and no rate limiting on this service.

Fix — send a deadline:

curl -X POST http://localhost:8080/rule -H 'X-Api-Timeout: 5s' ...

Note — when the deadline fires the client gets 408 context-middleware-timed-out in plain text. The handler goroutine sees the cancelled request context and unwinds at the engine’s next rule boundary, so anything already written to out is discarded. The server write timeout (default 15s, timeouts.write) bounds the response, not the execution.

One tenant’s request executed another tenant’s rules

Means — a single engine instance is shared by every tenant of the process, keyed only by rule set name. Two tenants publishing the same rule set name collide: the first wins, the second is logged as a failed add, and nothing in the request path checks that the named rule set belongs to the calling tenant.

Fix — namespace rule set names per tenant, or run separate deployments.

Memory grows across a batch of requests

Means — every execution clones the whole knowledge base. The engine builds a fresh instance of the named rule set per call so concurrent requests cannot share mutable evaluation state, so allocation scales with rule-set size times request rate. Nothing is pooled or reused between calls.

Confirm — /health reports Go memory statistics in MiB and is the only built-in memory signal:

curl -s http://localhost:8080/health

Fix — keep rule sets small and split unrelated rules into separate named sets rather than one large one. The service applies no request-body size limit and no rate limiting, so an upstream gateway is the only place to bound concurrency.

Note

Document work adds a second cost on either surface: the analysis pipeline runs in full for every execution that carries a document and caches nothing across them, holding every page’s words, merged blocks, lines, tables and a spatial index with no page, size or time cap. Over HTTP that is one analysis per request, on top of the knowledge-base clone. On the CLI, kis ocr page --workers N holds N fully analysed documents at once, and kis ocr doc merges the entire corpus into one document, so peak memory is the whole corpus.

Continue with

• Running the service — boot sequence, probes, rule delivery
• Configuration — every key and its default
• Error reference — the complete code and message table
• Rule language — what compiles and what does not
• Patterns — rule shapes that avoid the traps above