End-to-End Testing

Audience: kis.ai engineers writing tests, plus DevSecOps running acceptance checks against a deployed instance. Covers unit, integration, and end-to-end layers, with recipes per package and a curl smoke sequence for post-deploy verification.

0. State of the world (be honest)

There are NO automated tests in this repo today. The chassis migration deleted the v0.1 tests (which targeted the hand-rolled YAML parser and mTLS-principal flow); replacements have not been written. Tracked in the design pending list under “Tests.”

This document is therefore both an authoring guide (“here’s how to write the missing test for X”) and a manual-acceptance guide (“here’s how to verify a deployment by hand”). Treat the recipes in §3 as the green-field starting point — copy them into <pkg>/<pkg>_test.go to seed the suite.

1. Layers

Layer	Scope	Tools	When to write
Unit	Single package, no I/O beyond t.TempDir(). Fake clock for time-driven code.	Go testing, t.TempDir(), stub VaultFactory.	Always — should be ≥ 80% of test code by volume.
Integration	Cross-package, real filesystem + real lib-audit + stub vault. Server-layer tests inject kconst.Tenant into ctx; no HTTP.	Go testing, httptest.NewRecorder where HTTP is in scope.	For any change to the server / logmgr / workers interplay.
End-to-end	Real binary, real vault, real JWT, curl against /submit etc.	audit-svc.bin, vault dev mode (or compose), Bruno or curl.	Pre-release acceptance + deployment smoke.

2. Build, vet, race, coverage

go build ./...                       # must always pass
go vet   ./...                       # must always pass
go test  ./...                       # the suite (once written)
go test -race ./...                  # before merging concurrency changes
metis run build.yaml test            # what CI runs — emits coverage.txt

The test task in build.yaml writes a combined coverage.txt (per-package coverage merged) targeting the audit.svc module (-coverpkg). CI parses that.

3. Per-package test recipes

3.1 config/

Test surface: FromOrderedMap, MatchLog, MatchGlob.

Build the input with smap.Of or smap.FromMSI — no YAML parsing in tests (svcboot has already done that in production):

package config

import (
    "testing"
    "git.kis.ai/kis.ai/lib/supermap/smap"
)

func TestFromOrderedMap_Minimal(t *testing.T) {
    om := smap.Of("audit", map[string]any{
        "base_dir": "/tmp/audit",
        "signer": map[string]any{
            "vault_path_template": "audit/{tenant}/key",
        },
        "routing": []any{
            map[string]any{"log_id_pattern": "**", "durability": "strict"},
        },
    })

    cfg, err := FromOrderedMap(om)
    if err != nil { t.Fatal(err) }
    if cfg.Logs.BaseDir != "/tmp/audit" { t.Fatalf("base_dir: %q", cfg.Logs.BaseDir) }
    if cfg.Signer.KeyIDTemplate != "{tenant}-key" {
        t.Fatalf("key_id_template default: %q", cfg.Signer.KeyIDTemplate)
    }
}

func TestFromOrderedMap_RejectsBadDurability(t *testing.T) {
    om := smap.Of("audit", map[string]any{
        "base_dir": "/x",
        "signer":   map[string]any{"vault_path_template": "x"},
        "routing":  []any{map[string]any{"durability": "wat"}},
    })
    if _, err := FromOrderedMap(om); err == nil {
        t.Fatal("expected error on durability=wat")
    }
}

func TestMatchGlob(t *testing.T) {
    cases := []struct{ pat, in string; want bool }{
        {"**", "anything", true},
        {"prod/*", "prod/auth", true},
        {"prod/*", "dev/auth", false},
        {"", "anything", true},
    }
    for _, c := range cases {
        if got := MatchGlob(c.pat, c.in); got != c.want {
            t.Errorf("MatchGlob(%q, %q) = %v, want %v", c.pat, c.in, got, c.want)
        }
    }
}

3.2 signer/

Test surface: VaultFactory.For cache hit, miss, parse errors, Invalidate.

Stub the vault client behind the zvault.VaultClient interface:

package signer

import (
    "context"
    "crypto/ed25519"
    "crypto/x509"
    "encoding/pem"
    "testing"

    "git.kis.ai/kis.ai/lib/zero/kiserr"
    zvault "git.kis.ai/kis.ai/lib/vault"
)

// stubVault implements zvault.VaultClient, returning a fixed PEM per path.
type stubVault struct {
    secrets map[string]string
    calls   map[string]int
}

func (s *stubVault) GetSecret(ctx context.Context, name string) (string, *kiserr.KisErr) {
    s.calls[name]++
    if v, ok := s.secrets[name]; ok { return v, nil }
    return "", kiserr.NewError(ctx, "NOT_FOUND", errNotFound)
}
// ... satisfy the rest of zvault.VaultClient with no-ops ...

func TestVaultFactory_CachesPerTenant(t *testing.T) {
    pem := generatePKCS8EdPEM(t)
    sv := &stubVault{
        secrets: map[string]string{"audit/cust-acme/key": pem},
        calls:   map[string]int{},
    }
    f := NewVaultFactory(sv, "audit/{tenant}/key", "{tenant}-key")

    s1, err := f.For(context.Background(), "cust-acme")
    if err != nil { t.Fatal(err) }
    s2, err := f.For(context.Background(), "cust-acme")
    if err != nil { t.Fatal(err) }
    if s1 != s2 { t.Fatal("expected same cached signer") }
    if sv.calls["audit/cust-acme/key"] != 1 {
        t.Fatalf("expected 1 vault call, got %d", sv.calls["audit/cust-acme/key"])
    }
}

// Helper: generate a PKCS#8-encoded Ed25519 PEM for tests.
func generatePKCS8EdPEM(t *testing.T) string {
    _, priv, _ := ed25519.GenerateKey(nil)
    der, _ := x509.MarshalPKCS8PrivateKey(priv)
    return string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der}))
}

3.3 logmgr/

Test surface: Get (open/create), validation, SnapshotForTenant, Close. Real filesystem via t.TempDir(); real *audit.Log (no need to stub lib-audit at this layer).

Provide a stub VaultFactory that returns a hand-built *auditsigner.Local — see the helper above. Recommended: add signer.NewVaultFactoryFromSigner(s types.Signer) *VaultFactory as a test helper so you can wire a static signer without the vault plumbing.

func TestManager_GetCreatesOnFirstCall(t *testing.T) {
    dir := t.TempDir()
    cfg := minimalConfig(dir)              // helper: build *config.Config
    sig := newTestSignerFactory(t)         // returns *VaultFactory backed by Local
    log := loggertest.New(t)               // helper: returns a no-op logger.Logger

    mgr := New(cfg, sig, log)
    defer mgr.Close()

    mg, err := mgr.Get(context.Background(), "cust-acme", "auth")
    if err != nil { t.Fatal(err) }
    if mg.TreeSize() != 0 { t.Fatalf("new log tree_size = %d, want 0", mg.TreeSize()) }
    if _, statErr := os.Stat(mg.Root() + "/log.json"); statErr != nil {
        t.Fatalf("log.json not created: %v", statErr)
    }
}

func TestManager_RejectsTraversalLogID(t *testing.T) {
    mgr := New(minimalConfig(t.TempDir()), nil, loggertest.New(t))
    _, err := mgr.Get(context.Background(), "cust-acme", "../escape")
    if err == nil { t.Fatal("expected traversal rejection") }
}

3.4 workers/

Test surface: dueNow / recordSuccess / recordFailure / backoffFor math, plus the per-tick logic in sealTick / syncTick against a real manager.

The clock is time.Now() everywhere — for deterministic tests, inject time.Time values directly through the public tick methods rather than waiting on a ticker:

func TestPool_BackoffDoublesOnFailure(t *testing.T) {
    p := New(minimalCfg(), stubMgr(), loggertest.New(t))
    now := time.Now()

    p.recordFailure("k1", now)
    if got := p.failureCount("k1"); got != 1 { t.Fatalf("failures = %d", got) }
    if p.dueNow("k1", now.Add(20*time.Second), 0) {
        t.Fatal("should be in backoff window (30s)")
    }

    p.recordFailure("k1", now)
    // now backoff is 30s × 2 = 60s
    if p.dueNow("k1", now.Add(45*time.Second), 0) {
        t.Fatal("should still be in backoff window (60s)")
    }
    if !p.dueNow("k1", now.Add(70*time.Second), 0) {
        t.Fatal("should be out of backoff after 60s")
    }
}

func TestBackoffFor_CapsAtMax(t *testing.T) {
    if got := backoffFor(100); got != backoffMax {
        t.Fatalf("backoffFor(100) = %v, want %v", got, backoffMax)
    }
}

For panic-recovery tests, drive supervise directly with a func(ctx, time.Time) that panics, then assert the loop survives and the success log is emitted on the next tick.

3.5 server/

Test surface: Submit, Head, InclusionProof, Read, Seal, ListLogs. Inject kconst.Tenant directly into ctx — no HTTP, no middleware:

func TestServer_SubmitRoundTrip(t *testing.T) {
    mgr := newRealManager(t)                  // helper from logmgr tests
    srv := New(minimalCfg(), mgr, loggertest.New(t))

    ctx := context.WithValue(context.Background(), kconst.Tenant, "cust-acme")
    resp, err := srv.Submit(ctx, &svcapi.SubmitRequest{
        Submission: svcapi.Submission{
            LogID:   "auth",
            Payload: []byte(`{"event":"x"}`),
        },
    })
    if err != nil { t.Fatal(err) }
    if resp.Receipt.Sequence != 0 { t.Fatalf("seq = %d", resp.Receipt.Sequence) }
}

func TestServer_RejectsMissingTenant(t *testing.T) {
    srv := New(minimalCfg(), newRealManager(t), loggertest.New(t))
    _, err := srv.Submit(context.Background(), validReq())
    // No kconst.Tenant in ctx → ErrUnauthenticated
    if !errors.Is(err, svcapi.ErrUnauthenticated("")) { // implement Is on ServiceError
        t.Fatalf("err = %v, want unauthenticated", err)
    }
}

3.6 httpapi/

Lightest layer — mostly JSON encode/decode plumbing. Drive with httptest.NewRequest + httptest.NewRecorder, pass through httpapi.RegisterRoutes registered on a fresh httprouter.Router:

func TestHTTPAPI_Submit_BadJSON(t *testing.T) {
    router := httprouter.New()
    RegisterRoutes(router, newTestServer(t), "audit")

    req := httptest.NewRequest("POST", "/submit", strings.NewReader("not json"))
    req.Header.Set("Content-Type", "application/json")
    // Skip JWT here — we're testing the inner handler, not auth.
    // For full coverage, also test through the SecureWithJWTToken wrapper.
    w := httptest.NewRecorder()
    router.ServeHTTP(w, req)

    if w.Code != http.StatusBadRequest {
        t.Fatalf("status = %d, want 400", w.Code)
    }
}

Full-chain HTTP tests (with SecureWithJWTToken in the loop) need either a stub JWT keypair or hitting the real chassis — defer those to end-to-end (§5).

4. Integration test recipe

A single test that boots the whole graph against t.TempDir() + a stub vault. No HTTP needed.

func TestEndToEnd_SubmitVerifyRead(t *testing.T) {
    dir := t.TempDir()
    log := loggertest.New(t)

    cfg := mustParse(t, fmt.Sprintf(`
audit:
  base_dir: %q
  signer:
    vault_path_template: audit/{tenant}/key
  routing:
    - log_id_pattern: "**"
      durability: strict
      seal_interval: 1s
`, dir))

    sig := signer.NewVaultFactoryFromSigner(testEd25519Signer(t))  // test helper
    mgr := logmgr.New(cfg, sig, log)
    defer mgr.Close()
    srv := server.New(cfg, mgr, log)

    ctx := context.WithValue(context.Background(), kconst.Tenant, "t1")

    // Append three records.
    for i := 0; i < 3; i++ {
        _, err := srv.Submit(ctx, &svcapi.SubmitRequest{
            Submission: svcapi.Submission{
                LogID:   "auth",
                Payload: []byte(fmt.Sprintf(`{"i":%d}`, i)),
            },
        })
        if err != nil { t.Fatal(err) }
    }

    // Seal and verify inclusion of record 1.
    head, err := srv.Seal(ctx, &svcapi.SealRequest{LogID: "auth"})
    if err != nil { t.Fatal(err) }
    if head.Head.TreeSize != 3 { t.Fatalf("tree_size = %d", head.Head.TreeSize) }

    proof, err := srv.InclusionProof(ctx, &svcapi.InclusionProofRequest{
        LogID: "auth", Sequence: 1, TreeSize: 3,
    })
    if err != nil { t.Fatal(err) }
    if proof.LeafIndex != 1 { t.Fatalf("leaf_index = %d", proof.LeafIndex) }

    // (Optional) call lib-audit's verifier directly on the on-disk tree
    // to confirm chain integrity end-to-end.
}

5. End-to-end (deployed binary)

The audit.svc OpenAPI spec is the contract. Import it into Bruno / Postman / Insomnia and every endpoint becomes a saved request with schemas + examples. Bruno’s CLI runs the same collection in CI:

npm i -g @usebruno/cli
bru run path/to/audit-collection --env dev

5.1 Manual smoke (curl)

Run from any host that can reach $AUDIT_URL and has a JWT issued by IAM for an identity in the tenant under test.

export AUDIT_URL=https://audit.dev.kis.ai
export JWT=eyJhbGciOi...
export TENANT=cust-acme

# 1. Readiness
curl -fsSL $AUDIT_URL/ready                                  # → ok

# 2. Submit
curl -fsSX POST $AUDIT_URL/submit \
  -H "Authorization: Bearer $JWT" -H "X-Tenant: $TENANT" \
  -H "Content-Type: application/json" \
  -d '{"submission":{"log_id":"smoke","payload":"aGVsbG8="}}' | jq

# 3. Force a seal so we have a signed head
curl -fsSX POST $AUDIT_URL/seal \
  -H "Authorization: Bearer $JWT" -H "X-Tenant: $TENANT" \
  -H "Content-Type: application/json" \
  -d '{"log_id":"smoke"}' | jq

# 4. Read it back
curl -fsSX POST $AUDIT_URL/read \
  -H "Authorization: Bearer $JWT" -H "X-Tenant: $TENANT" \
  -H "Content-Type: application/json" \
  -d '{"log_id":"smoke","from_seq":0,"to_seq":0,"with_body":true}'

Failure signals:

Symptom	Likely cause
401 invalid attempt detected by user	JWT tenant claim ≠ X-Tenant.
401 missing tenant in request	X-Tenant header absent.
400 invalid tenant	X-Tenant value not registered in the chassis config service.
500 vault signer: get secret ...	Per-tenant key not provisioned in vault. See the operations guide §4.
404 logmgr: no routing rule matches log_id	The log_id matched no rule. Add one or use a catch-all **.

5.2 Docker-compose for full local stack

A reference compose layout (not yet shipped with the repo — adapt from config.svc’s testing setup):

docker-compose.test.yaml

services:
  vault:
    image: hashicorp/vault:latest
    cap_add: [IPC_LOCK]
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: dev-token
      VAULT_DEV_LISTEN_ADDRESS: 0.0.0.0:8200
    ports: ["8200:8200"]

  config:
    image: kisai/config.svc:latest      # chassis config service
    # ...

  audit:
    build: .                            # builds audit-svc.bin
    depends_on: [vault, config]
    volumes:
      - audit-data:/var/lib/audit/logs
      - ./.audit-svc.yaml:/etc/audit/audit-svc.yaml
    ports: ["8443:8443"]

volumes:
  audit-data:

After docker compose up:

1. vault kv put audit/test-tenant/ed25519-private-key [email protected]
2. Mint a JWT for test-tenant against IAM.
3. Run the curl smoke from §5.1 against https://localhost:8443.

6. Client-side verification recipe

For consumers writing their own verification (auditors, compliance pipelines), the recipe in the HTTP API reference is canonical:

1. POST /inclusion-proof for (sequence, tree_size).
2. POST /head and confirm tree_size ≥ what you used in step 1; keep the root_hash + signature + key_id.
3. Reconstruct the leaf hash from the original payload using the canonical leaf encoding in lib-audit/segment.
4. Walk the proof hashes upward; compare the computed root to step 2.
5. Verify the head’s signature against the tenant’s Ed25519 public key (out of band today — see the pending GET /pubkey item).

A reference implementation can use lib-audit’s verify.Verify if you have access to the on-disk tree, or write a slim verifier against the HTTP responses if you only have wire access.

7. CI integration

The chassis CI driver runs metis run build.yaml test, which runs go test -coverprofile=... per package and merges into coverage.txt. Once tests are written, the test task will start producing real numbers.

There is no CI workflow file in this repo yet — the platform CI discovers build.yaml automatically.