Matching and confidence

OCR text is wrong in predictable ways, so every text lookup in the document library is a distance calculation and every extracted value carries a score. This page covers the distance functions and how to threshold them, the phrase / snippet / table query APIs, and the confidence model that turns spatial signals into a number you can gate on.

Note

Both surfaces reach everything on this page. The kis CLI matches over local .bbox files — kis bbox phrase inspects them directly — and rules.svc matches over a document sent with the execute-rule request. Thresholds, scores and returned shapes are identical on both. The rule-language spellings of these calls are on Rules over documents.

Distance functions

Function	Signature	Cost model
Distance	Distance(a, b string) int	Levenshtein over runes; insert/delete/substitute cost 1
DamerauDistance	DamerauDistance(a, b string) int	Levenshtein plus adjacent transposition at cost 1
OCRAwareDistance	OCRAwareDistance(a, b string) float64	Levenshtein with fractional substitution cost for known OCR confusions
Similarity	Similarity(a, b string) float64	1 - Distance/max(len(a), len(b)) in runes
OCRAwareSimilarity	OCRAwareSimilarity(a, b string) float64	Same ratio over OCRAwareDistance

All five operate on runes, so a multi-byte character counts as one edit — Distance("café", "cafe") is 1. Identical strings and two empty strings both give similarity 1.0.

The similarity denominator is the longer string. A short query against a long block scores low even when the query is fully contained in it, so similarity is a poor containment test.

DamerauDistance is implemented and tested, but no shipping matcher calls it — no kis bbox subcommand and no call a rule can make reaches it. A transposition is never discounted on either surface; it costs two edits like any other pair of substitutions.

The OCR confusion table

OCRAwareDistance discounts 17 substitution pairs — 33 directed table entries, because every pair except %→9 is registered in both directions:

Cost	Pairs
0.3	l↔1, O↔0, o↔0, I↔1, I↔l
0.4	S↔5, s↔5, B↔8, $↔S, space↔_, ,↔.
0.5	G↔6, Z↔2, c↔e, n↔h, u↔v
0.6	%→9 (one direction only)

Insertions and deletions always cost 1.0. The effect is that SCHEDULE B-l sits 0.3 from SCHEDULE B-1 while a genuine one-character error sits 1.0 away, so a tight budget can accept OCR noise and still reject real mismatches.

The table is a compiled-in package variable. There is no config key, environment variable or flag that extends or overrides it.

Caution

Three shipping call sites use OCRAwareDistance: region-scoped phrase matching (QueryResult.Containing*PhraseFuzzy), the vocabulary spell corrector, and fuzzy anchor lookup in schema-driven form extraction. Every document-level fuzzy phrase API — FindPhraseFuzzy, FindIPhraseFuzzy, HasPhraseFuzzy, the FuzzyTextNear* family, FieldNear* and the merged-word fallback — uses plain Levenshtein, where an l/1 confusion costs a full edit. Some in-source comments claim otherwise; the code does not.

The form-extraction site truncates the float to an integer before comparing it against the anchor’s MaxDistance, so there every fractional discount collapses to 0 and a single OCR confusion is free.

Caution

Case-insensitive matching silently disables most of the table. Several pairs exist only in uppercase — I↔1, I↔l, B↔8, G↔6, Z↔2, $↔S — so any path that lowercases first (fuzzy.Match and fuzzy.BestMatch with the default CaseSensitive: false, and ContainingIPhraseFuzzy) charges them the full 1.0. OCRAwareDistance("I", "1") is 0.30; ("i", "1") is 1.00. Measured consequence: SCHEDULE B-I ITEM I OF I against SCHEDULE B-1 ITEM 1 OF 1 at per-word 1 / total 1 matches through ContainingPhraseFuzzy (3 × 0.3) and fails through ContainingIPhraseFuzzy (3 × 1.0).

Match configuration

type MatchConfig struct {
    MaxDistance   int
    CaseSensitive bool
    OCRAware      bool
    MinSimilarity float64
}

DefaultConfig() returns MaxDistance: 2, CaseSensitive: false, OCRAware: true, MinSimilarity: 0.8. Match(target, candidate string, cfg MatchConfig) bool and BestMatch(target string, candidates []string, cfg MatchConfig) (int, float64) consume it; BestMatch returns the winning index and its similarity, or -1 plus the best score seen when nothing clears MinSimilarity.

Caution

MaxDistance is ignored whenever OCRAware is true, and BestMatch never reads it at all — both gate on MinSimilarity only. Since DefaultConfig sets OCRAware: true, its MaxDistance: 2 is dead — MinSimilarity alone decides the match. The values above are compiled in: no config key, flag or request field changes them on either surface. The budgets you can set are the per-word and total distances the phrase APIs take, below.

Choosing thresholds

Fuzzy phrase APIs take two integer budgets: maxDistancePerWord, applied to each word independently, and maxTotalDistance, applied to the accumulated distance across the phrase.

Budget	Effect
per-word 0	Exact words only. Rejects a 0.3 OCR confusion as hard as a real edit
per-word 1	One substitution per word — the working default for scanned text
per-word 2	Accepts short-word collisions; expect false positives on 3–4 letter anchors
total	Caps how much error the whole phrase may accumulate; set to roughly the word count

Caution

In region-scoped matching maxDistancePerWord is an int64 but OCR-aware costs are fractional, and the comparison is dist > float64(maxDistPerWord). A per-word budget of 0 therefore rejects a 0.3 confusion outright — SCHEDULE B-I against SCHEDULE B-1 does not match at per-word 0 even with total 5. The fractional discount is only reachable through the total budget. Set per-word to 1 and tune the total.

NormalizeForMatch(s string) string lowercases, trims and collapses whitespace runs to one space. GenerateOCRVariants(s string) []string returns s plus every single-character substitution drawn from the confusion table; it backs Vocabulary.AddOCRConfusions. Variants come back in nondeterministic order.

Phrase queries

Exact

FindPhrase(phrase string, linesBefore, linesAfter int64) []PhraseMatch
FindIPhrase(phrase string, linesBefore, linesAfter int64) []PhraseMatch

FindIPhrase is the case-insensitive form. A PhraseMatch carries Phrase, Page, Box (the union of the matched blocks), Blocks, LineIndex, MatchedLine, LinesBefore and LinesAfter.

Fuzzy

FindPhraseFuzzy(phrase string, maxDistancePerWord, maxTotalDistance, linesBefore, linesAfter int64) []FuzzyPhraseMatch
FindIPhraseFuzzy(phrase string, maxDistancePerWord, maxTotalDistance, linesBefore, linesAfter int64) []FuzzyPhraseMatch

FuzzyPhraseMatch embeds PhraseMatch and adds TotalDistance int and MatchedWords []string.

The algorithm finds candidate blocks for the first word within the per-word budget, then walks the remaining words on the same page, ranking candidates by editDistance*100 + spatialDistance and taking the minimum. A candidate qualifies spatially if it shares the OCR line id, is horizontally aligned within a 0.3 baseline tolerance, or belongs to the same reconstructed line.

Caution

The walk is single-page and at most one line deep: a candidate may be at most one reconstructed line below the current word and at most one line below the phrase start. A phrase wrapping across two line breaks, or across a page boundary, never matches regardless of budget.

When the walk yields nothing and the phrase has more than one word, a merged-word fallback strips all spaces from the phrase and compares it against each whole block with plain Distance. This recovers phrases the OCR engine welded together (At Closing → AtClosing).

Caution

The merged-word fallback compares against every block in the document with no spatial constraint, and emits every block within maxTotalDistance. At generous budgets it returns matches on unrelated pages. Separately, word candidacy records distance 0 for any block whose text merely contains the search word, so a short anchor word matches large unrelated blocks at zero cost and skews the ranking toward big blocks.

Convenience family

All of these match case-insensitively. The fuzzy forms derive the total budget automatically as maxDistancePerWord × wordCount — you cannot set it separately here.

Function	Signature
HasPhrase	(phrase string) bool
HasPhraseFuzzy	(phrase string, maxDistancePerWord int64) bool
HasPhraseOnPage	(phrase string, page int64) bool
HasPhraseOnPageFuzzy	(phrase string, page, maxDistancePerWord int64) bool
HasPhraseInBox	(phrase string, box geometry.Box) bool
HasPhraseInBoxFuzzy	(phrase string, box geometry.Box, maxDistancePerWord int64) bool
HasAnyPhrase	(phrases ...string) bool
HasAnyPhraseFuzzy	(maxDistancePerWord int64, phrases ...string) bool
HasAllPhrases	(phrases ...string) bool
HasAllPhrasesFuzzy	(maxDistancePerWord int64, phrases ...string) bool
HasAnyPhraseInBox	(box geometry.Box, phrases ...string) bool
HasAnyPhraseOnPage	(page int64, phrases ...string) bool
CountPhrase	(phrase string) int64
CountPhraseFuzzy	(phrase string, maxDistancePerWord int64) int64
CountPhraseOnPage	(phrase string, page int64) int64
PhraseNear	(a, b string, maxPixels float64) bool
PhraseBetween	(startAnchor, endAnchor string) string

Region-scoped matching

Scoping the search to a region first is the only phrase-query path that uses the OCR-aware distance.

ContainingPhraseFuzzy(phrase string, maxDistPerWord, maxTotalDist int64) QueryResult
ContainingIPhraseFuzzy(phrase string, maxDistPerWord, maxTotalDist int64) QueryResult
ContainingAnyPhraseFuzzy(phrases string, maxDistPerWord, maxTotalDist int64) QueryResult
ContainingAnyIPhraseFuzzy(phrases string, maxDistPerWord, maxTotalDist int64) QueryResult

The Any forms split phrases on | and return the first phrase that matches. Words from the region’s blocks are flattened, quotes and dashes are normalised, then the phrase is slid over the word stream using OCRAwareDistance. These match at per-word 1 / total 2:

SCHEDULE B-l   ->  SCHEDULE B-1
SCHEDULE B-I   ->  SCHEDULE B-1
LOAN O12345    ->  LOAN 012345
5CHEDULE A     ->  SCHEDULE A
EXHIBIT “A”    ->  EXHIBIT "A"
ANTI–COERCION  ->  ANTI-COERCION

Written as a rule condition:

bbox.Get("page").InRegionPct(0.0, 0.0, 100.0, 20.0).ContainingAnyIPhraseFuzzy("ANTI-COERCION INSURANCE|Anti-Coercion Statement|STATEMENT OF ANTI-COERCION", 2, 4).Count() > 0

Caution

Numeric literals in a rule must match the Go parameter type exactly — method dispatch is by exact type and a mismatch fails to find the method. InRegionPct takes float64, so write 0.0 and 100.0, never 0 and 100. The distance budgets are int64, so write 2 and 4, never 2.0. The same split applies across this page: PhraseNear’s maxPixels and FieldNear*’s expand / maxExtent are float64; every maxDistancePerWord, maxTotalDistance and page is int64.

From the CLI

kis bbox phrase -p "Invoice Number" doc.bbox
kis bbox phrase -p "total amount due" -B 5 -A 2 doc.bbox
kis bbox phrase -p "Inviice Numbr" --fuzzy doc.bbox
kis bbox phrase -p "Invoice Number" --fuzzy --max-dist-word 1 --max-dist-total 2 doc.bbox
kis bbox phrase -i page1.bbox -i page2.bbox -p "signature required" -f json

Flag	Default	Meaning
-p, --phrase	—	Phrase to search for (required)
-i, --input	—	Input file; repeatable, also accepted positionally
-B, --before	3	Context lines before the match
-A, --after	3	Context lines after the match
--fuzzy	false	Enable fuzzy matching
--case-insensitive	false	Case-insensitive comparison
--max-dist-word	2	Per-word budget (fuzzy mode)
--max-dist-total	3	Total budget (fuzzy mode)
-f, --format	text	text or json
-o, --output	stdout	Output file

Caution

No match is not an error. kis bbox phrase prints No matches found for phrase: "<phrase>" — with (fuzzy mode) appended under --fuzzy — and exits 0. kis bbox table behaves the same way, with No table found near anchor: <anchor>. Scripts must inspect the output — use -f json and check count — never the exit status.

From the service

rules.svc runs the same queries from a rule, over a document sent inline on the execute-rule request. Documents travel under the reserved bbox key, one entry per document name:

{
  "name": "schedule-b-check",
  "bbox": {
    "page": {"pages": [{"number": 1, "width": 612, "height": 792, "words": []}]}
  }
}

rule ScheduleBPresent "schedule B heading, OCR-tolerant" salience 100 {
    when
        bbox.Has("page") &&
        bbox.Get("page").InRegionPct(0.0, 0.0, 100.0, 20.0).ContainingPhraseFuzzy("SCHEDULE B-1", 1, 2).Count() > 0
    then
        out.Set("schedule_b", true);
        Retract("ScheduleBPresent");
}

The budgets carry the same meaning as the CLI flags — --max-dist-word is maxDistancePerWord, --max-dist-total is maxTotalDistance — and match the same strings. The service runs rules, so what a match yields reaches the caller through the out collector rather than a printed report: write the matched text to out when you need to see it. The full request and response shapes are on The rules service.

Snippets

ExtractSnippet(startPhrase, endPhrase string, caseInsensitive, includeEnd bool) *Snippet
ExtractSnippetFuzzy(startPhrase, endPhrase string, caseInsensitive, includeEnd bool, maxDist int64) *Snippet

Snippet is {StartPhrase, EndPhrase, StartPage, EndPage, Lines []string}. The start line is included; the end line is excluded unless includeEnd. Extraction spans page boundaries. The result is nil only when the start phrase is not found.

kis bbox snippet -s "Section 5:" -e "Section 6:" doc.bbox
kis bbox snippet -s "BEGIN" -e "END" --include-end -f json doc.bbox
kis bbox snippet -s "Sectlon 5:" -e "Sectlon 6:" --fuzzy 2 doc.bbox
kis bbox snippets --snip "section8|Section 8:|Section 9:" doc.bbox
kis bbox snippets --snip "terms|TERMS AND CONDITIONS|SIGNATURE" --snip "privacy|PRIVACY|END" *.bbox

--fuzzy takes an integer per-word distance, 0 meaning exact. kis bbox snippets always emits JSON — an object keyed by snippet name, with a missing snippet present as an empty string rather than absent — and has no --format flag.

Caution

Three sharp edges in snippet extraction:

• The first start match anchors the snippet. A start phrase appearing in a table of contents wins over the real section, and there is no way to select a different occurrence.
• If no end match follows the start, EndPage becomes the page count and extraction runs to the end of the document.
• The end-phrase walk has a raw strings.Contains fallback, so a snippet can stop at a line the matcher never matched — and in fuzzy mode that fallback is still an exact substring test.

The fuzzy total budget is derived, not settable: maxDist × max(startWordCount, endWordCount), applied to both phrases. A short end phrase inherits a budget sized by a long start phrase.

Table queries

Detection, defaults and the row/column model are on Tables. The lookup APIs are:

Function	Signature	Behaviour
TableNear	(anchorText string) *table.Table	Delegates to TableNearPhrase when the anchor contains a space, otherwise a single-block substring search
TableNearPhrase	(phrase string) *table.Table	Case-sensitive FindPhrase, falling back to a block contains search
TableBelow / TableAbove	(anchor string) *TableResult	Nearest table in that direction
TableBetween	(start, end string) *TableResult	The table bounded by two anchors
TablesBetween	(start, end string) []*TableResult	Every table bounded by two anchors
TableBelowFuzzy / TableAboveFuzzy	(anchor string, maxDistPerWord, maxTotalDist int64) *TableResult	Fuzzy anchor forms
TableBetweenFuzzy	(start, end string, maxDistPerWord, maxTotalDist int64) *TableResult	Fuzzy bounded form

The two TableNear* calls hand back the raw detected table; everything else returns a TableResult wrapper with row/cell accessors. SafeTableBelow, SafeTableAbove, SafeTableBetween and their Fuzzy variants wrap that result so its accessors return empty values instead of panicking on a missing table. SafeTablesBetween is not a wrapper at all — it calls TablesBetween directly and returns a plain (possibly nil) slice, which is safe to range over but carries no nil-safe accessors.

Caution

TableNear is not page-scoped. Once the anchor is located it returns the table with the smallest box-to-box distance across the whole merged document with no page or maximum-distance constraint — if the anchor’s page has no table you silently get one from another page. It is also inconsistent about case: multi-word anchors go through a case-sensitive phrase search, single-word anchors through a substring block search.

kis bbox tables page1.bbox page2.bbox
kis bbox tables *.bbox -f csv
kis bbox table -a "Invoice Items" doc.bbox
kis bbox table -a "Price List" -H "Item" -H "Price" -H "Qty" doc.bbox

kis bbox table requires --anchor. -H/--header validates expected column headers case-insensitively after trimming and is the one subcommand that exits non-zero on a miss: Header validation failed: missing headers: <list> (found: <list>).

Table confidence is not field confidence

A detected Table carries Confidence float64 — (consistencyScore + sizeScore) / 2, where consistency is 1 / (1 + variance(cellCountsPerRow) × 0.1) and size is 0.5, 0.8 at ≥ 3 rows and ≥ 2 columns, or 1.0 at ≥ 5 rows and ≥ 3 columns — plus a full TableScore. Both are geometry-only: they say nothing about whether the cell text is right, they are not on the same scale as field-extraction confidence, and the CLI prints neither.

The confidence model

Every confidence number in the library is the same shape:

type Score struct {
    Value   float64  // 0.0 - 1.0
    Factors []Factor
}

type Factor struct {
    Name   string
    Value  float64
    Weight float64
    Detail string
}

FromFactors(factors ...Factor) Score computes the weight-normalised mean and clamps it to [0,1]; a factor with weight ≤ 0 is coerced to 1.0. New(value float64) Score wraps a bare number. The Detail strings are what make a value explainable in a reviewer interface.

Built-in factors

Constructor	Name	Weight	Value
OCRFactor(conf)	ocr	1.0	OCR engine confidence
ValidationFactor(ok, name)	validation	1.0	1.0 or 0.0
PatternMatchFactor(score, pattern)	pattern_match	0.9	Supplied score
AlignmentFactor(score)	alignment	0.8	Supplied score
ConsistencyFactor(ok, detail)	consistency	0.8	1.0 or 0.0
FuzzyMatchFactor(sim, dist)	fuzzy_match	0.7	Supplied similarity
ProximityFactor(dist, maxDist)	proximity	0.6	1 - distance/maxDistance
CountFactor(found, expected)	count	0.5	found/expected

These weights are the system’s compiled-in policy about what it trusts. They are not configuration.

How a field score is produced

ScoreFieldExtraction(ExtractionSignals) (confidence.Score, Provenance) is the single place a label-to-value extraction is scored. It builds two sub-scores and combines them:

Label score — the label block’s OCR confidence (weight 1.0); a label_format factor (weight 0.5) worth 0.9 when the text ends with :, is ALL CAPS with 40 letters or fewer, or starts with a capital, and 0.6 otherwise; and, when the label matched fuzzily, a fuzzy_match factor of 1 - editDistance/len(requestedLabel).

Value score — the value block’s OCR confidence (weight 1.0); an alignment factor; a proximity factor measured against the search budget; a positional prior for the relation; and a pattern factor when the value matched an expected type.

Overall — WeightedAverage([label, value], [0.4, 0.6]). The value is trusted more than the label.

Relation	Positional prior (weight 0.6)	Alignment axis
inline	0.95	Flat 1.0
right_of	0.85	Vertical centre offset against half the label height
below	0.75	Left-edge offset against half the wider box
left_of	0.70	Vertical centre offset against half the label height
near	0.70	Vertical centre offset against half the label height
above	0.65	Left-edge offset against half the wider box
anything else	0.60	Factor omitted

Alignment converts an offset to a score with FromAlignment(offset, tolerance) = 1 - |offset|/tolerance, reaching 0 at or beyond the tolerance. Two related helpers exist: FromEditDistance(editDist, maxLen int) float64 and FromOverlap(overlapArea, box1Area, box2Area) (intersection over union).

Provenance

type Provenance struct {
    Method            string  // "field.right_of", "field.below", "field.chain", "schema.checkbox", ...
    Anchor            string
    MatchedText       string
    Page              int64
    Distance          float64 // px between label box and value box
    LabelEditDistance float64
    LabelBox          geometry.Box
    ValueBox          geometry.Box
    Factors           []confidence.Factor
}

Without it a wrong value and a right value look identical to a reviewer. String() renders one line; FieldResult.Explain() renders the value plus every contributing factor.

Getting a scored value

The FuzzyTextNear* family returns a bare string and discards every trust signal. The confidence-carrying counterparts return a query.FieldResult:

FieldNear(label string, dir Direction, maxDistPerWord, maxTotalDist int64, expand float64) query.FieldResult
FieldNearCapped(label string, dir Direction, maxDistPerWord, maxTotalDist int64, expand, maxExtent float64) query.FieldResult
FieldNearChain(chain string, valueDir Direction, maxDistPerWord, maxTotalDist int64, expand float64) query.FieldResult
FieldNearChainCapped(chain string, valueDir Direction, maxDistPerWord, maxTotalDist int64, expand, maxExtent float64) query.FieldResult

Anchor matches are tried best-edit-distance-first and the first match that actually yields a value wins, so a bare mention of the label cannot shadow the real labelled occurrence. When the label is not found you still get a FieldResult — empty, Found false, carrying Method and Anchor in provenance, and therefore still flagged for review.

amt  := doc.FieldNearCapped("Loan Amount",   document.DirRight, 1, 2, 0.0, 200.0) // value OCR 0.98
rate := doc.FieldNearCapped("Interest Rate", document.DirRight, 1, 2, 0.0, 200.0) // value OCR 0.50
// amt.Confidence() >= 0.85, and rate.Confidence() < amt.Confidence()
// amt.Provenance().Method == "field.right_of"

const threshold = 0.90
amt.NeedsReview(threshold)  // false
rate.NeedsReview(threshold) // true
doc.FieldNearCapped("Nonexistent Label", document.DirRight, 1, 2, 0.0, 200.0).NeedsReview(threshold) // true

A rule spells the same call bbox.Get("page").FieldNearCapped("Loan Amount", bbox.Dir("right"), 1, 2, 0.0, 200.0), under the CLI and on the service alike — the direction factory has to be inline and every literal has to match its parameter type. See Rules over documents.

Chain extraction retags provenance to field.chain. Intermediate hop edit distances are not surfaced, so the label factor reflects only the final anchor:

term := doc.FieldNearChainCapped("Loan Information<below>|Loan Term", document.DirRight, 1, 2, 0.0, 200.0)
// term.AsText() == "30 years"; term.Provenance().Method == "field.chain"

Accessor	Returns
Confidence()	float64 in [0,1]
ConfidenceScore()	Full confidence.Score with factors
ConfidencePct()	Confidence() * 100
NeedsReview(threshold)	True if not found or below threshold
IsConfident(threshold)	Found and at or above threshold
Provenance()	The Provenance record
Explain()	Multi-line value, confidence and per-factor breakdown

Value blocks are trimmed before scoring. A right-of read keeps the contiguous same-line run and breaks at the first gap wider than 3.5× the median word gap; a below read keeps only the line nearest the label, then gap-trims left to right. Surviving blocks are unioned into one block whose confidence is the minimum OCR confidence across its words — one bad word lowers trust — and anything after a column-separator artifact is cut. A scored value is therefore often shorter than the string the equivalent FuzzyTextNear* call returns; the bare-string path deliberately keeps its wider capture.

When no explicit cap is given, the search extent is max(labelWidth × 2.5, labelHeight × 2). That same number is the denominator of the proximity factor, so capping the search also changes the score.

Aggregating and shaping scores

Group	Functions
Aggregate	Average, WeightedAverage(scores, weights), Min, Max, Geometric, Harmonic
Shape	Penalize(p) → v*(1-p), PenalizeIf, Boost(b) → v + (1-v)*b, BoostIf, AtLeast(t), Cap(max), Floor(min)
Classify	IsHigh (≥ 0.8), IsMedium (≥ 0.5 and < 0.8), IsLow (< 0.5), Level(), Percent()

Level() returns VeryHigh at ≥ 0.9, High at ≥ 0.8, Medium at ≥ 0.6, Low at ≥ 0.4, else VeryLow.

Caution

IsHigh/IsMedium/IsLow and Level() use different, incompatible bands. A score of 0.55 is both IsMedium() == true and Level() == LevelLow. Pick one taxonomy and do not assume the other agrees.

Geometric and Harmonic return Score{Value: 0} as soon as any input is ≤ 0, discarding all factors — combining a not-found field with good ones zeroes the aggregate. Average and WeightedAverage do not clamp. FromFactors writes back into the caller’s factor slice (factors[i].Weight = w) and keeps that backing array on the Score, so a shared slice gets its weights rewritten.

confidence.Report ({Overall, Components map[string]Score, Issues, Suggestions}) collects a document-level rollup with Add, AddIssue, AddSuggestion, Compute() and Summary(). Compute() is a plain unweighted average over an unordered map.

Thresholding: calibrate, do not guess

A raw score of 0.8 does not mean “80% likely correct”. It is a weighted average of heuristic factors, and its relationship to actual correctness depends on your documents, your OCR engine and your schema. The library says so in its own package documentation, and NeedsReview’s contract is explicit that the threshold should come from calibration.

Caution

Nothing in the shipping extraction path calls Calibration.Apply. FieldResult.Confidence() is always the raw score. Treat it as an ordering, not a probability, until you have fitted it against labelled outcomes — and do not adopt a round number like 0.8 as a default review threshold.

The calibration package turns labelled outcomes into a defensible operating point:

Function	Signature	Purpose
Fit	Fit(samples []Sample) *Calibration	Isotonic regression (Pool-Adjacent-Violators) fitting raw score → empirical probability
Apply	(c *Calibration) Apply(raw float64) float64	Piecewise-linear interpolation between knots; holds the end value outside the fitted range
SampleCount	(c *Calibration) SampleCount() int64	Size of the fit
Reliability	Reliability(samples []Sample, bins int) []ReliabilityBin	Equal-width bands over [0,1] (10 when bins <= 0), empty bands omitted
ECE	ECE(samples []Sample, bins int) float64	Sample-weighted mean absolute gap between mean predicted and empirical; 0 is perfect
PrecisionCoverageCurve	PrecisionCoverageCurve(samples []Sample, steps int) []ThresholdStat	Sweeps the threshold 0..1 (20 steps by default, returns steps+1 points)
ThresholdForPrecision	ThresholdForPrecision(samples []Sample, targetPrecision float64, minAccepted int64) (ThresholdStat, bool)	Lowest threshold meeting the precision floor — that is, maximum coverage
CurveReport	CurveReport(curve []ThresholdStat) string	Renders the curve as a text table

A Sample is {Predicted float64; Correct bool}. A ThresholdStat is {Threshold, Coverage, Precision, Accepted, AcceptedWrong, ReviewedCount} — AcceptedWrong is the count that matters, the values auto-accepted but actually wrong.

Two properties of the fit are worth knowing when you read its numbers. Isotonic regression repairs a non-monotone raw scorer: if raw 0.50 is 75% correct and raw 0.70 is 25% correct, both pool to the same calibrated value. And an empty sample set produces no calibration at all, where an absent calibration is the identity map, still clamped to [0,1] — 0.37 stays 0.37, 1.5 becomes 1.0, −1 becomes 0.0.

With nothing accepted, precision is defined as 1.0 — vacuously precise, nothing accepted, nothing wrong. Always read Accepted alongside Precision, and pass a minAccepted floor to ThresholdForPrecision so the estimate is not based on a handful of points.

The measurement is: run ExtractFormScored over a sample — from either surface — label the results against ground truth, and take the lowest threshold that meets your precision target with enough accepted values behind it. ExtractFormScored(name string) map[string]query.FieldResult returns every declared property, with unfound ones at Found: false, confidence 0 and NeedsReview true, so you can measure coverage and not just hits. The number you settle on is what you pass to NeedsReview.

Caution

Calibration is not reachable. The reliability table, ECE, precision/coverage curve and operating-point search have no CLI subcommand, no flag and no request field — neither kis nor rules.svc exposes them. The same is true of the field-evaluation harness. Gate on the raw confidence a query returns, and do your calibration analysis outside the block.

Special-region detection

Signature, handwriting and barcode detection runs inside the analysis pipeline. DefaultConfig enables it and the CLI always uses DefaultConfig, so every kis bbox invocation pays for it even though no subcommand prints a region — and there is no flag to switch it off.

Detection runs during analysis, before any rule sees the document, so the regions are already attached by the time a rule asks for them.

A Region carries Type, Box, Page, Confidence (a full Score), NearLabel, OCRText, OCRConfidence, AspectRatio, WordDensity and IsIsolated. Types render as Signature, Handwriting, Barcode1D, QRCode, Barcode2D, Unknown. Filters: FilterByConfidence(regions, min), FilterByType(regions, type), GetSignatures, GetHandwriting, GetBarcodes. On the document: HasSignatures(), HasHandwriting(), HasBarcodes(), GetSignaturesOnPage(page), GetHandwritingOnPage(page), GetBarcodesOnPage(page), GetSpecialRegionsByConfidence(min), GetHighConfidenceSignatures(), SignatureCount(), BarcodeCount().

All three detectors are inference over OCR geometry. None of them looks at the page image.

Signatures

Two strategies. Near one of the signature labels listed below, the detector searches right and below for words whose OCR confidence is under the threshold or text more than 30% non-alphanumeric; failing that it looks for a signature line (five or more underscores, or a leading X plus three or more). Separately it emits isolated low-confidence words in the bottom third of a page with fewer than three blocks within 50px, at confidence ≥ 0.4.

The anchor list is fixed. Nine anchors ship — signature, sign here, sign:, signed:, authorized signature, x_, x:, sign below, your signature — matched case-insensitively as substrings, and neither surface adds to it: there is no config key, flag or request field that registers a domain anchor. A label like Borrower Signature is still found, because it contains signature; a label that shares no substring with the nine is not an anchor at all.

Caution

The signature ocr_confidence factor is 1.0 - word.Confidence — inverted. The worse the OCR, the more confident the detector is. Combined with a position factor of y / pageHeight, any low-quality OCR near the bottom of a page reads as a probable signature. The field-extraction model reads the same OCR number the right way up, under a different factor name (ocr), so the two Score values are not comparable even though both are called confidence.

A second code path, used when building a region from an isolated candidate word, carries its own shorter keyword list — signature, sign, signed, x_, x: — so NearLabel there is filled from five keywords, not the nine above. GetHighConfidenceSignatures hard-codes a threshold of 0.7 with no parameter; use GetSpecialRegionsByConfidence or special.FilterByConfidence for any other bar.

Handwriting

Four factors over OCR geometry, emitted at score ≥ 0.5 and merged with same-type neighbours within 50px: ocr_range (weight 1.0, peaking at OCR confidence 0.5), height_variance (weight 0.8, coefficient of variation of word heights), spacing_variance (weight 0.6, added only when the CV of inter-word gaps is ≥ 0.2) and baseline_variance (weight 0.7, added only when the CV of word bottoms is ≥ 0.02). It is a wobble detector over OCR output, not a handwriting classifier.

Barcodes

Caution

Barcode detection is not usable. Nothing is ever decoded — OCRText on a barcode region is OCR garbage, never a payload — and the candidate generator unconditionally proposes the four 150×150 page corners whenever a corner holds two or fewer words. A 150×150 box has aspect ratio 1.0, which classifies as a QR code, and the remaining factors push it past the 0.4 emission floor. Measured: a single-page document containing one word yields four QRCode regions at confidence 0.912 with empty OCRText. HasBarcodes() is therefore effectively always true and BarcodeCount() effectively always at least 4.

OCR cleanup and the matcher

Curly quotes, ligatures, homoglyphs and fullwidth characters all burn edit-distance budget. Cleanup happens at conversion time only:

kis hocr2bbox -i scan.hocr -o scan.bbox --clean
kis hocr2bbox -i ./pages -o ./bbox --clean-form -w 8

Caution

The kis bbox query subcommands never clean text at read time — they call the plain loader. The one exception is region phrase matching, which normalises quotes and dashes internally before comparing. If you need clean text for matching, clean it during conversion.

Full operation list, presets and the lossy cases are on OCR ingestion.

Things that are not wired up

Capability	State
Vocabulary spell correction	Fully implemented, but document.Config.SpellCheck is never read and no spell checker is constructed during analysis, so nothing runs it on either surface
Correction.Reason	Documented as known_error / fuzzy_match / pattern; always set to vocabulary_match
Calibration and field evaluation	No CLI subcommand and no request field reaches them
Barcode / QR detection	Emits phantom regions, decodes nothing

Vocabulary.Correct also iterates an unordered map and keeps the strictly-smaller distance, so two vocabulary terms tied at the same OCR-aware distance resolve nondeterministically between runs.

A note on page numbers

kis bbox loads each input file, appends its pages in argument order and renumbers pages sequentially from 1, rewriting each word’s page number to match. Reported page numbers are positions in the argument list, not the page numbers stored inside the files, so shell glob order determines them.

Note

kis bbox, kis ocr, kis rules and kis hocr2bbox are proxy commands that shell out to a separately installed plugin. Without it you get Error: 'bbox' is not installed. on stderr and a non-zero exit. See Command line.

Continue with

• Extraction — anchors, schema hints and the field APIs
• Tables — detection, headers and master-detail
• OCR ingestion — conversion and text cleanup
• Rules over documents — calling these from a rule
• Command line — the full kis bbox surface