Batch-process JSON files with a script

Goal

Every JSON file in a directory passed through one script, with the results written where you choose.

Why bother

A loop in a shell script works until it does not — and the ways it fails are all quiet.

Without this	With this
A shell loop that stops at the first bad file, or does not	Every file processed, failures reported
Output overwrites input, so a re-run processes its own results	Separate directories, by default
Result shape differs from what the consumer expects	`--key` nests it where the consumer looks
No record of what was processed	A count and a per-file line

Step 1 — the script

The entry point receives the parsed file:

function main(doc) {
  return {
    id: doc.id,
    total: (doc.lines || []).reduce((s, l) => s + l.qty * l.price, 0)
  };
}

Find the files, then fan out over them:

name: totals
tasks:
  - name: find
    glob:
      pattern: "./orders/order-*.json"
      setvar: files
    next:
      go: each

  - name: each
    map:
      items_path: files
      task: one
      max_concurrency: 4
    next:
      go: done

  - name: one
    shell: "kis script run totals.js --debug < {{_item}}"

  - name: done
    succeed: true

kis flow -f totals.yaml -w 4

glob: collects the paths, map: runs the node once per path with {{_item}} bound to it, and max_concurrency bounds the parallelism. Give the run at least as many workers as the concurrency — with the default single worker, max_concurrency: 4 still runs one at a time.

kis script rules totals.js --input ./orders --output ./totals

Every *.json under ./orders is processed and the result written to ./totals.

Step 3 — narrow what is matched

kis script rules totals.js -i ./orders -o ./totals --pattern 'order-*.json'

Files not matching are skipped silently, which is convenient and worth remembering when the count is lower than you expected.

Step 4 — nest the result

Replacing the document with the result is often not what a downstream consumer wants:

kis script rules totals.js -i ./orders -o ./totals --key computed

The result lands under computed rather than as the whole document, so the original content is preserved alongside it.

Step 5 — supply extra inputs

Facts and documents can be passed alongside each file:

kis script rules classify.js -i ./pages -o ./out \
  --facts config=./rates.json \
  --bbox page=./page-1.bbox.json

Which to use

The job	Use
A one-off reshape of a directory you have in front of you	Script — `kis script rules` is one command
Thousands of files, where a failure partway matters	Flow — the run names which items completed
It runs on a schedule, or someone else runs it	Flow
The files are large enough that memory matters	A data pipeline — it streams

The finished command

kis script rules totals.js \
  --input ./orders \
  --output ./totals \
  --pattern 'order-*.json' \
  --key computed \
  --func main

Verify

Count in and out — they should match:

ls ./orders/order-*.json | wc -l
ls ./totals/*.json | wc -l

A shortfall means files failed or the pattern excluded them, and the run output distinguishes the two.