End-to-End Testing

How the Intake test suite is structured, how to run it, and the manual end-to-end smoke test for the durable write path. Companion to the forms guide (behaviour) and the design spec.

The two tiers

Tier	Needs a C toolchain?	What it covers
Unit	no (pure Go)	the security-critical logic — schema identifier validation, INSERT/CREATE compilation, value coercion/validation, additive-vs-destructive migration diff, and the rate limiter (burst, global shed, map eviction). Fast; runs in CI on every change.
Smoke (end-to-end)	yes (CGO/DuckDB)	the real HTTP flows and the durable write path: start the binary, drive every request/hardening path with curl, and confirm rows landed in the JSONL WAL and the DuckDB file. Manual today; the recipe below is copy-pasteable.

The unit tests live in the packages they cover and carry no build tag — a plain go test ./... compiles and runs them. There is no external service dependency: no Postgres, no network.

Prerequisites

• Go 1.26+.
• A C toolchain (clang/gcc) is required to compile the module, because sink/duckdb links DuckDB via CGO (github.com/duckdb/duckdb-go/v2). This affects go test ./... too, since it builds every package. The schema and ratelimit unit tests are pure Go, but the go test invocation still compiles the whole module.
• No database, no gateway, nothing else to stand up.

Running the tests

# everything
go test ./...

# just the fast, security-critical cores
go test ./schema/ ./ratelimit/

# verbose
go test -v ./schema/

# via the fleet-uniform build.yaml (writes coverage.txt)
#   <task-runner> build.yaml test

Expected:

ok  edge-intake/schema
ok  edge-intake/ratelimit
?   edge-intake/...        [no test files]

Always run go vet ./... alongside — it must be clean.

Test layout

schema/schema_test.go       identifier validation, INSERT compilation, coercion,
                            strict/honeypot/capture, migration diff, utf-8/control chars,
                            notify block parse + validation
ratelimit/ratelimit_test.go per-IP burst→429, global ceiling→shed, map eviction bound
notify/notify_test.go       async dispatch to a httptest webhook, retry-then-succeed,
                            nil-dispatcher no-op, email/Teams render shape

The notify tests stand up an httptest server as the Teams webhook and assert the dispatcher posts a MessageCard, retries a failing endpoint, and counts delivery — no real network, fully deterministic.

Other packages (config, logging, sink/*, httpapi, cmd) are exercised by the smoke test. When you add automated tests for them, keep the file next to the code (sink/jsonl/jsonl_test.go, …).

What the unit tests cover

schema — the injection-safety and validation core:

• bad identifiers rejected at compile (Bad-Name, 1table, DROP TABLE, reserved system-field collisions);
• InsertSQL is fully parameterised and column-ordered — the placeholder count equals the column count, so no user value can reach SQL text;
• coercion happy-path (string→int64, truthy bool) and the fact that all violations for a submission are collected together (bad email + out-of-range int + bad enum + missing required in one response);
• strict mode rejects unknown fields but never flags the honeypot;
• additional_properties: capture folds extras into a trailing _extra column;
• Reconcile returns create / additive-add / destructive-error for the three table states;
• invalid UTF-8 and control characters are rejected.

ratelimit — burst then LimitedPerIP, distinct IPs get distinct buckets, the global ceiling sheds independently of per-IP, and the IP map never exceeds MaxIPs (eviction bound).

Security regression coverage

The point of this service is hardening, so treat these as regression anchors — if you change the schema or sink layer, these must stay green:

• No value in SQL text — TestInsertSQLIsParameterisedAndOrdered asserts placeholder-count == column-count. The smoke test additionally sends a '); DROP TABLE …;-- payload and confirms the table survives with the string stored as inert data.
• Strict rejection — unknown fields (TestStrictRejectsUnknownButHoneypotIgnored).
• Encoding — control chars / invalid UTF-8 (TestControlCharsAndUTF8Rejected).
• No silent destructive migration — TestReconcileCreateAdditiveDestructive.
• Rate-limit trip & bounded map — the ratelimit tests.

End-to-end smoke test

Runs the real binary with a scratch data directory and high rate limits, drives every path, then shuts down cleanly. Nothing here touches your real data dirs.

cd edge-intake
go build -tags netgo -o dist/edge-intake .

SCRATCH=$(mktemp -d)
./dist/edge-intake \
  --http-bind 127.0.0.1:18081 \
  --storage-backend duckdb --wal-jsonl \
  --duckdb-path "$SCRATCH/intake.duckdb" \
  --jsonl-dir  "$SCRATCH/jsonl" \
  --forms-dir  examples/forms \
  --log-output stdout --log-format text \
  --ratelimit-per-ip 1000 --ratelimit-global 5000 &
SVR=$!; sleep 2
B=http://127.0.0.1:18081
IP='-H X-Kis-Client-IP: 203.0.113.7'

sc() { curl -s -o /dev/null -w "%{http_code}" "$@"; }

echo "valid urlencoded : $(sc $IP -d [email protected] -d team_size=25 -d use_case=banking -d consent=true $B/f/demo_request)  (want 200)"
echo "valid json       : $(sc $IP -H 'Content-Type: application/json' -d '{"email":"[email protected]","consent":true}' $B/f/demo_request)  (want 200)"
echo "bad email        : $(sc $IP -H 'Content-Type: application/json' -d '{"email":"nope","consent":true}' $B/f/demo_request)  (want 400)"
echo "unknown form     : $(sc $IP -d [email protected] $B/f/nope)  (want 404)"
echo "honeypot         : $(sc $IP -d [email protected] -d consent=true -d website_url=http://spam $B/f/demo_request)  (want 200, discarded)"
echo "bad content-type : $(sc $IP -H 'Content-Type: text/xml' --data '<x/>' $B/f/demo_request)  (want 415)"
echo "unknown field    : $(sc $IP -H 'Content-Type: application/json' -d '{"email":"[email protected]","consent":true,"evil":"x"}' $B/f/demo_request)  (want 400)"
echo "too large        : $(sc $IP --data "message=$(head -c 70000 /dev/zero | tr '\0' a)" $B/f/demo_request)  (want 413)"
echo "sql-injection val: $(sc $IP -H 'Content-Type: application/json' -d '{"email":"[email protected]","consent":true,"full_name":"x'\''); DROP TABLE demo_request;--"}' $B/f/demo_request)  (want 200, stored literal)"
echo "health           : $(sc $B/health)   ready: $(sc $B/ready)   metrics: $(sc $B/metrics)"

# rate-limit trip (dedicated low limits, fresh IP)
kill -TERM $SVR; wait $SVR 2>/dev/null

To watch the 429 / 503 paths, run a second instance with tiny limits (--ratelimit-per-ip 2 --ratelimit-per-ip-burst 3) and fire ~8 rapid requests from one X-Kis-Client-IP: the first ~3 return 200, the rest 429.

To watch graceful shutdown, send SIGTERM and confirm the log ends with shutdown signal received, draining → Intake stopped (with the counter snapshot) and the process exits cleanly.

Migration checks: point a second boot at the same DuckDB file with a form that adds a field (expect an applied additive migration log), then with a form that retypes an existing field (expect a fatal destructive diff on … startup error).

Verifying the DuckDB file

DuckDB is single-writer, so stop the server first (release the lock), then query the file. A throwaway program using the same driver:

// go run . <path-to.duckdb>
package main

import (
  "database/sql"
  "fmt"
  "os"

  _ "github.com/duckdb/duckdb-go/v2"
)

func main() {
  db, _ := sql.Open("duckdb", os.Args[1])
  defer db.Close()

  var n int
  db.QueryRow("SELECT count(*) FROM demo_request").Scan(&n)
  fmt.Println("rows:", n)

  // the injection payload must be stored as an inert literal; both tables intact
  rows, _ := db.Query("SELECT email, coalesce(full_name,'') FROM demo_request ORDER BY id")
  defer rows.Close()
  for rows.Next() {
    var e, f string
    rows.Scan(&e, &f)
    fmt.Printf("  %-18s full_name=%q\n", e, f)
  }
}

The JSONL WAL is safe to read while the server runs (that’s the point of the WAL) — cat "$SCRATCH"/jsonl/*.jsonl shows one JSON object per accepted submission, honeypot hits excluded.

Patterns you must follow

• Keep unit tests pure and tag-free. The security-critical logic (schema, ratelimit) has no external dependency — keep it that way so CI stays fast and deterministic.
• No payloads in test assertions leak to logs. Mirror production: assert on outcomes/field-names, never echo submitted values.
• New sink backend or migration change ⇒ extend the smoke test and add a Reconcile-style unit test; the destructive-diff guard must never regress.
• Use a scratch dir (mktemp -d) for any test that writes a DuckDB/JSONL file; never point tests at a real data directory.
• Run go vet ./... and go test ./... before committing; both must be clean.