`vocab` — named lists and tables

vocab holds the domain words a ruleset needs — the section headings that end a scan, the watermarks to strip, the state-code table, the phrases that mean “this page is a duplicate” — in data beside the rules rather than in the rules themselves.

The reason to use it is maintenance. A list of forty stop phrases repeated across twelve rules is forty phrases you will one day update in eleven places. In a vocabulary file it is one edit, and a non-programmer can make it.

The library ships the mechanism only. The content is your domain vocabulary, and never lives in the platform.

Where vocabulary comes from

Three sources, in increasing precedence:

Source	Scope	Set by
Vocabulary files beside the .grl corpus	Every execution	The ruleset author
Per-execution plugin data	One execution	The caller
vocab.Define / vocab.Append in a rule	One execution	A rule

A file is named vocab.yaml, vocab.yml, vocab.json, or ends in .vocab.yaml / .vocab.yml / .vocab.json. Directories load parents first, so a nested file overrides the one above it, and load order is precedence order.

# vocab.yaml, beside the rules
lists:
  section_stops:
    - General Provisions
    - Signatures
    - Exhibits
  watermarks: "DRAFT|COPY|SPECIMEN"     # the pipe form
tables:
  state_abbrev:
    California: CA
    New York: NY

The pipe character is reserved as the phrase separator everywhere. A list entry containing | is split, so the sequence form and the pipe form mean exactly the same thing. That is what lets vocab.List hand a list straight to any parameter that takes pipe-delimited phrases.

Reading a list

vocab.List(names...) → string

Returns the named list or lists pipe-joined — the format that pipe-delimited rule parameters take directly.

Parameter	Type	Meaning
names	string, variadic	One or more list names

Several names union in argument order, case-insensitively de-duplicated, first occurrence winning. An unknown name contributes nothing rather than erroring, so a list you have not defined yet degrades to “no phrases” instead of failing the run.

rule ExtractAgreementDate "read the date, stopping at any known section heading" {
    when
        bbox.Doc().Has("Agreement Date")
    then
        out.Set("agreementDate",
            bbox.Doc().Below("Agreement Date").Date());
        audit.LogWithData("ExtractAgreementDate", "date read",
            map.Of("stops", vocab.List("section_stops", "exhibit_stops")));
}

Passing two names is the composition point: a base list shared across rulesets, plus a document-specific one, without either file knowing about the other.

vocab.Items(name) → list of string

Returns the named list as a list, for rules that iterate rather than pass it along.

Parameter	Type	Meaning
name	string	The list name

vocab.Has(name) → bool

Reports whether a list with this name is defined — even if it is empty.

Parameter	Type	Meaning
name	string	The list name

The “even if empty” clause is the useful half: it separates “this ruleset does not configure watermarks” from “this ruleset configures watermarks as none”, which are different intentions.

rule WarnNoStopList "the ruleset expected a stop list and none is configured" salience 90 {
    when
        !vocab.Has("section_stops")
    then
        log.Warn("no section_stops vocabulary defined; scans will run to end of document");
}

vocab.Count(name) → int64

Returns the number of phrases in the named list.

Parameter	Type	Meaning
name	string	The list name

vocab.Names() → list of string

Returns every defined list name, sorted. Diagnostic — log it when a lookup returns nothing and you want to know what the ruleset actually loaded.

Testing text against a list

The two predicates differ in exactly one way, and choosing wrongly is the most common mistake here.

vocab.Contains(name, text) → bool

Reports whether text contains any phrase in the named list, case-insensitively — a substring test.

Parameter	Type	Meaning
name	string	The list name
text	string	The text to search

rule DraftDocument "a watermark phrase appears anywhere on the page" salience 50 {
    when
        vocab.Contains("watermarks", bbox.Doc().AllText().Str())
    then
        findings.Warn("DraftDocument", "provenance", "Document carries a draft watermark");
        out.Set("isDraft", true);
}

vocab.Matches(name, text) → bool

Reports whether text, trimmed, is one of the phrases — exact membership, not substring.

Parameter	Type	Meaning
name	string	The list name
text	string	The text to test

rule KnownSectionHeading "this line is exactly one of the section headings" {
    when
        vocab.Matches("known_sections", bbox.Doc().FirstLine().Str())
    then
        out.Set("sectionStart", bbox.Doc().FirstLine().Str());
}

Use Contains when scanning a body of text for a marker; use Matches when classifying a value that should be one of a known set. A heading check written with Contains will match a line that merely mentions the heading.

Transforming a value

vocab.Strip(name, value) → string

Removes every occurrence of every phrase in the named list from value, case-insensitively, then collapses runs of whitespace.

Parameter	Type	Meaning
name	string	The list name
value	string	The text to clean

Phrases are applied longest first, so a phrase that extends another is removed whole rather than being left as a fragment. With a list of COPY and COPY — NOT FOR EXECUTION, the longer phrase goes first and you are not left with the trailing — NOT FOR EXECUTION.

rule CleanBorrowerName "strip watermark text bled into the name field" {
    when
        bbox.Doc().Right("Borrower Name").Found()
    then
        out.Set("borrowerName",
            vocab.Strip("watermarks", bbox.Doc().Right("Borrower Name").Str()));
}

This is the function that pays for the whole namespace on scanned documents, where a diagonal watermark routinely lands inside an extracted field.

Lookup tables

vocab.Lookup(table, key) → string

Returns the value for key in the named table — exact match first, then case-insensitive — or the empty string when the table or the key is absent.

Parameter	Type	Meaning
table	string	The table name
key	string	The key to look up

rule NormaliseState "store the two-letter state code" {
    when
        bbox.Doc().Right("Property State").Found()
    then
        out.Set("stateCode",
            vocab.Lookup("state_abbrev", bbox.Doc().Right("Property State").Str()));
}

vocab.LookupOr(table, key, fallback) → string

vocab.Lookup with a fallback for the absent case.

Parameter	Type	Meaning
table	string	The table name
key	string	The key to look up
fallback	string	Returned when the table or key is absent

The case-insensitive pass scans keys in sorted order, so a fold collision — two keys differing only in case — resolves the same way on every run. Deterministic, if arbitrary; do not rely on which of the two wins, but do rely on it not changing between runs.

rule StateCodeWithPassthrough "unknown states keep their original text" {
    when
        bbox.Doc().Right("Property State").Found()
    then
        out.Set("stateCode",
            vocab.LookupOr("state_abbrev",
                bbox.Doc().Right("Property State").Str(),
                bbox.Doc().Right("Property State").Str()));
}

Defining vocabulary from a rule

Both of these write a per-execution overlay. The base list loaded from file is copied, never mutated, so a definition made while processing one document never leaks into the next. That isolation is the reason these are safe to use at all.

vocab.Define(name, phrases) → bool

Sets a named list from a pipe-delimited string, replacing any existing definition for this execution only.

Parameter	Type	Meaning
name	string	The list name
phrases	string	Pipe-delimited phrases

rule NarrowStopsForShortForm "this form type ends at a different heading" salience 80 {
    when
        bbox.Doc().Top(15).Has("Short Form Disclosure")
    then
        vocab.Define("section_stops", "Signatures|Acknowledgement");
        Retract("NarrowStopsForShortForm");
}

Note the salience and the Retract: a rule that redefines vocabulary should run before the rules that read it, and exactly once.

vocab.Append(name, phrases) → bool

Extends a named list, creating it if needed, with pipe-delimited phrases.

Parameter	Type	Meaning
name	string	The list name
phrases	string	Pipe-delimited phrases to add

Per-execution like Define, and the underlying base list is copied rather than mutated.

rule AddLenderSpecificStops "this lender adds two headings of its own" salience 80 {
    when
        bbox.Doc().Has("First National")
    then
        vocab.Append("section_stops", "Lender Certification|Branch Use Only");
        Retract("AddLenderSpecificStops");
}

Prefer Append to Define when you are adding to a shared list — Define replaces it, and a rule that replaces a list another rule depends on is a rule whose ordering now matters.

Choosing where a phrase lives

The phrase is	Put it in
Stable across every document this ruleset sees	A vocabulary file beside the rules
Supplied per request by the calling service	Per-execution plugin data
Derived from what this document turned out to be	vocab.Define / vocab.Append in a rule
Used exactly once, in one rule	A literal in that rule

The last row is worth stating: not everything belongs in a vocabulary. A single phrase used in a single rule is clearer inline, and moving it to a file makes the rule harder to read for no benefit.

See also

• The Document API — the reads these lists feed
• Built-in functions — the full callable surface
• Writing rules — salience and retraction, used above