End-to-End Testing
Audience: customer developers and kis.ai DevSecOps. Walks through every deployment + consumption combination (except etcd, which has its own infra-heavy story in the distributed deployment guide).
For each combination, the doc gives you:
- The bootstrap YAML.
- What to expect on boot.
- A curl smoke sequence.
- The equivalent OpenAPI operation (see the config service OpenAPI spec — importable into Bruno, Postman, Insomnia, Swagger UI, or any OpenAPI-aware client).
- The write-and-watch-SSE round-trip you’d use in your acceptance test.
- The mode-specific failure signals to look for.
The single canonical API reference is the HTTP API. This doc focuses on which mode to use; the API reference covers how each endpoint behaves; the config service OpenAPI spec is the machine-readable contract.
0. Prerequisites
Section titled “0. Prerequisites”| Tool | Used for | Install |
|---|---|---|
config.svc | The binary under test. | Standard install (see the single-node deployment guide). |
curl | Command-line smoke tests. | Pre-installed on Linux/macOS. |
jq | Pretty-print + extract from JSON responses. | brew install jq / apt install jq. |
| Bruno (or Postman / Insomnia) | OSS REST client. Imports the config service OpenAPI spec directly — every endpoint becomes a saved request with schemas + examples. | https://www.usebruno.com — desktop app, or npm i -g @usebruno/cli for headless runs in CI. |
| Swagger UI (optional) | Renders the config service OpenAPI spec as interactive docs in a browser. | docker run -p 8080:8080 -e SWAGGER_JSON=/spec/openapi.yaml -v $PWD:/spec swaggerapi/swagger-ui |
git | Only for the git-source combinations (Part A, sections A.4 / A.5). | Pre-installed on most dev machines. |
1. Terminology
Section titled “1. Terminology”| Term | Meaning in this doc |
|---|---|
| Server mode | How the config.svc binary itself reads its source-of-truth: filesystem (fs), git, postgres. Configured under hives.*.sources in the bootstrap. |
| Client mode | How a consumer service reads config via lib-chassis: remote (SSE against config.svc), local-file, local-dir, or bootstrap-inline. Configured via top-level config_url: or config: keys in the consumer’s bootstrap. |
| Boot config | The YAML file passed to config.svc -f <path> server. Holds chassis identity + source declarations + auth. |
| Hive | A namespaced, schemafied config tree. The platform ships two: cluster (dc/cluster/service/version) and tenant (customer/product/env/tenant/service). |
| Snapshot | A resolved, validated value at a specific (hive, path). Versioned per hive; pushed over SSE on change. |
2. The OpenAPI spec
Section titled “2. The OpenAPI spec”The config service OpenAPI spec is the machine-readable contract for the entire HTTP surface — health probes, hive CRUD, SSE streams, discovery, introspection. It groups operations by tag:
| Tag | Operations |
|---|---|
Health | /health, /ready, /version (unauthenticated) |
Hives | List, read, write (PUT/PATCH), delete, schema, example, tree |
ConfigStream | /stream/config (SSE), /stream/config/drift |
Discovery | Register, heartbeat, deregister, list services, list instances |
DiscoveryStream | /stream/discovery (SSE) |
Introspection | /sources, /subscribers |
Import into Bruno:
- Bruno → Import → OpenAPI → select the config service OpenAPI spec file.
- Edit the Local environment (auto-created from the
servers:block) to set your API key. - Every operation appears as a saved request with parameters, example bodies, and response schemas pre-filled.
Gotcha: configure auth at the collection level. OpenAPI’s
security: [{ApiKeyAuth: []}] tells Bruno each request needs an
API key, but the spec doesn’t carry the secret value — so the
Auth tab on each request is wired to “API Key” type with an
empty value field. If you then add X-Api-Key manually under
the Headers tab, Bruno sends two X-Api-Key headers (empty
one wins → 401 “unauthenticated”). Fix once at the collection
level:
- Right-click config service collection → Settings → Auth.
- Type: API Key. Key:
X-Api-Key. Value:{{key}}. Placement: Header. - Add a
keyvariable to each environment with the actual plain-text key value. - On each request’s Auth tab, set type to Inherit.
Now switching environments (Local / staging / prod) swaps the key automatically and no per-request configuration is needed.
Import into Postman / Insomnia: same flow — both import OpenAPI 3.x natively.
View as interactive docs (Swagger UI):
docker run --rm -p 8080:8080 \ -e SWAGGER_JSON=/spec/openapi.yaml \ -v $PWD:/spec \ swaggerapi/swagger-ui# → http://localhost:8080Generate client code: point any OpenAPI generator at
openapi.yaml — openapi-generator-cli generate -i openapi.yaml -g python (or go, typescript-fetch, java, …) produces a typed client library you can drop into your application.
Headless contract testing in CI: use Schemathesis or Dredd to validate that a running instance conforms to the spec:
schemathesis run --base-url http://localhost:7123 \ --header "X-Api-Key: $KEY" \ openapi.yaml3. Universal smoke sequence
Section titled “3. Universal smoke sequence”Independent of which combination you’re testing, run this first to prove the binary is reachable + authenticating:
export BASE="http://localhost:7123"export KEY="devkey-please-change-me"
curl -s "$BASE/health" # → {"status":"ok"}curl -s "$BASE/ready" # → {"status":"ready"}curl -s "$BASE/version" | jq # → JSON build infocurl -s -H "X-Api-Key: $KEY" "$BASE/hives" | jq # → {"hives":["cluster","tenant"]}In Bruno: run the 00-health/* folder. Three green ticks → server is
up and accepting your key. If any fail, do NOT proceed; see
Troubleshooting.
Part A — Server mode (the config.svc binary’s source backends)
Section titled “Part A — Server mode (the config.svc binary’s source backends)”The five source backends (excluding etcd) you’d configure under
hives.*.sources in the boot YAML. Each section is self-contained:
boot YAML → setup → smoke → write+watch round-trip.
A.1 — fs source: single-file values, simplest layout
Section titled “A.1 — fs source: single-file values, simplest layout”The filesystem source watches a directory tree. Each level of the
hive path is a directory; the actual value document lives in
config.yaml inside the leaf directory.
Boot YAML (config.yaml):
service: configversion: 0.0.0-devsid: dev-local
host: ""port: "7123"zerotrust: false
storage: mode: bbolt bbolt: path: ./cache
hives: cluster: sources: - { name: local, backend: fs, path: ./cluster, owns: ["**"], priority: 0, writable: true } tenant: sources: - { name: local, backend: fs, path: ./tenant, owns: ["**"], priority: 0, writable: true }
auth: apikey_file: ./apikeys.yamlSetup:
mkdir -p workdir && cd workdirmkdir -p cluster tenant cache
# Generate the API key — stored plain-text in apikeys.yaml,# compared constant-time on every request.KEY=$(openssl rand -hex 32)cat > apikeys.yaml <<EOFkeys: - key: $KEY subject_type: user subject_id: dev attrs: role: platform_adminEOFecho "API key: $KEY"
# Write the bootstrap (from above) to ./config.yaml
config.svc -f config.yaml serverSmoke:
curl -s -H "X-Api-Key: $KEY" "$BASE/hives" | jq# → {"hives":["cluster","tenant"]}
curl -s -H "X-Api-Key: $KEY" \ "$BASE/hives/tenant/" | jq# → root snapshot, version 1, empty valueWrite + watch round-trip:
# Terminal 1 — subscribe to SSEcurl -N -H "X-Api-Key: $KEY" "$BASE/stream/config?hives=tenant"
# Terminal 2 — write a tenant configmkdir -p tenant/acme/forge/prod/acme-prod/forge-metacat > tenant/acme/forge/prod/acme-prod/forge-meta/config.yaml <<'EOF'log_level: debugfeatures: fast_path: trueEOFWithin ~250 ms, terminal 1 emits:
event: snapshotid: 2data: {"hive":"tenant","path":["acme","forge","prod","acme-prod","forge-meta"],"value":{...},"version":2,...}Bruno equivalent:
00-health/*to confirm reachable.10-hives/read-tenant.bruparameterised on the path.10-hives/write-tenant.bruto PUT a fresh value (writes go through the REST API → the writable fs source → the sameconfig.yamlfile appears on disk).20-sse/stream-config.bruto subscribe.
Mode-specific failure signals:
| Symptom | Likely cause |
|---|---|
| Edited a file, no SSE event arrived | Filename isn’t exactly config.yaml. The fs source filters on that exact name. |
| Direct edit via Vim/VSCode misses events on macOS | Editor uses atomic-rename-on-save; fsnotify on Darwin sometimes loses the watch. Workaround: touch the file or write via REST. |
| Reconciler logs “validation failed” | YAML body doesn’t conform to the hive’s CUE schema. curl /hives/<hive>/_meta/schema for the schema. |
A.2 — fs source: directory tree mirroring the hive shape
Section titled “A.2 — fs source: directory tree mirroring the hive shape”Same as A.1 but you author multiple node levels with intermediate defaults that leaves inherit. Directory tree:
tenant/ config.yaml # tenant-hive root defaults acme/ config.yaml # customer-level defaults forge/ config.yaml # product-level defaults prod/ config.yaml # env-level defaults acme-prod/ config.yaml # tenant-level defaults forge-meta/ config.yaml # full leaf forge-cli/ config.yamlBoot YAML is identical to A.1.
Setup:
mkdir -p tenant/acme/forge/prod/acme-prod/forge-meta
cat > tenant/config.yaml <<'EOF'log_level: infoEOF
cat > tenant/acme/config.yaml <<'EOF'features: legacy_dashboard: falseEOF
cat > tenant/acme/forge/prod/config.yaml <<'EOF'jaeger_endpoint: prod-jaeger:14268EOF
cat > tenant/acme/forge/prod/acme-prod/forge-meta/config.yaml <<'EOF'log_level: debug # overrides root "info"EOFVerify resolution (this is the interesting part):
curl -s -H "X-Api-Key: $KEY" \ "$BASE/hives/tenant/acme/forge/prod/acme-prod/forge-meta" | jq '.value'Expect:
{ "log_level": "debug", # leaf wins over root "features": { "legacy_dashboard": false },# inherited from acme "jaeger_endpoint": "prod-jaeger:14268" # inherited from prod}The provenance field on the snapshot shows which directory each
key came from.
Bruno: 10-hives/read-tenant.bru — set the path variable to
acme/forge/prod/acme-prod/forge-meta and inspect value +
provenance in the response.
A.3 — fs source: cluster + tenant mounted under one root
Section titled “A.3 — fs source: cluster + tenant mounted under one root”Some teams prefer one source root rather than two:
hives: cluster: sources: - { name: local, backend: fs, path: ./hives/cluster, owns: ["**"], priority: 0, writable: true } tenant: sources: - { name: local, backend: fs, path: ./hives/tenant, owns: ["**"], priority: 0, writable: true }Two paths, same parent. Functionally identical to A.1/A.2; the difference is purely operational (one git repo for both vs. two).
A.4 — git source: local repo, no remote
Section titled “A.4 — git source: local repo, no remote”git source can open a local checkout directly — useful for testing
without a remote git server.
Setup:
mkdir -p sources/cluster-defaults && cd sources/cluster-defaultsgit init -qecho "log_level: info" > config.yamlgit add config.yaml && git -c user.email=ci@local commit -qm initialcd -Boot YAML (only the hives: section differs):
hives: cluster: sources: - name: local-git backend: git url: ./sources/cluster-defaults # local path, not a URL local_open: true # treat as a local working tree branch: main owns: ["**"] priority: 0 writable: false # writes via REST go to a higher-priority writable source below - name: overlay backend: fs path: ./overlay owns: ["**"] priority: 20 writable: trueSmoke: identical to A.1.
Write round-trip: REST writes land in the overlay fs source
(higher priority + writable). Git source is read-only here; updates
appear by committing to the local repo and waiting for the
reconciler’s poll cadence (default 60s).
Mode-specific failure signals:
| Symptom | Cause |
|---|---|
Boot fails with git: not a repository | The path isn’t a git repo — run git init there first. |
| Changes committed to the repo don’t appear via API | Poll cadence is 60s. Reconciler logs git source local-git: polled, no change on each tick; commit, wait, re-poll. |
A.5 — git source: remote repo (HTTPS / SSH)
Section titled “A.5 — git source: remote repo (HTTPS / SSH)”Production-style. Two flavors:
HTTPS with PAT:
sources: - name: platform-cluster-defaults backend: git url: https://git.example.com/platform/cluster-defaults.git branch: main auth_username: x-access-token auth_token: ${env:GIT_TOKEN} owns: ["**"] priority: 0 writable: falseExport GIT_TOKEN before starting config.svc. The reconciler does
a fresh clone into a temp directory on first boot; subsequent polls
do git pull.
SSH key:
sources: - name: platform-cluster-defaults backend: git branch: main auth_ssh_key_file: /etc/config/git-ssh-key owns: ["**"] priority: 0 writable: falseTesting locally without a real remote:
Spin up a local bare repo to act as the remote:
mkdir -p /tmp/git-remote/cluster-defaults.git && cd /tmp/git-remote/cluster-defaults.gitgit init --bare -qcd -
git clone /tmp/git-remote/cluster-defaults.git /tmp/git-workecho "log_level: info" > /tmp/git-work/config.yamlgit -C /tmp/git-work add . && git -C /tmp/git-work -c user.email=ci@local commit -qm initialgit -C /tmp/git-work push -qPoint your boot YAML at file:///tmp/git-remote/cluster-defaults.git
(go-git supports file:// URLs as if they were remote). No auth
needed.
Write round-trip via git: push to the remote, wait for the reconciler’s poll. To shortcut the wait, configure a webhook (see the single-node deployment guide §7.5) and POST to it from your git provider on push.
A.6 — postgres source
Section titled “A.6 — postgres source”Each hive snapshot row in a single table.
Setup (assuming PG running locally):
createdb config_testpsql config_test < <(config.svc generate pgsource) # ships a template DDLpsql config_test -c " INSERT INTO infinity_config_overrides (hive, path, value) VALUES ('tenant', 'acme/forge/prod/acme-prod/forge-meta', '{\"log_level\":\"debug\"}'::jsonb);"Boot YAML:
hives: tenant: sources: - name: pg-overrides backend: postgres dsn: postgres://localhost:5432/config_test?sslmode=disable table: infinity_config_overrides owns: ["**"] priority: 0 writable: trueWrite round-trip:
REST writes go through PG INSERT … ON CONFLICT DO UPDATE. The
reconciler picks them up via PG LISTEN/NOTIFY. SSE events fire
within milliseconds.
You can also write directly with psql — the LISTEN/NOTIFY path
catches it the same way.
Mode-specific failure signals:
| Symptom | Cause |
|---|---|
Boot fails with pgsource: connect: … | DSN wrong, PG not running, or sslmode setting mismatched. |
| Direct SQL UPDATE doesn’t fire SSE | The trigger that emits NOTIFY is missing — config.svc generate pgsource includes it; if you customized the DDL, ensure it’s still there. |
A.7 — mixed sources with priorities
Section titled “A.7 — mixed sources with priorities”The realistic production layout: a baseline source (low priority, read-only) + an overlay (high priority, writable).
hives: cluster: sources: - name: platform-defaults backend: git url: file:///tmp/git-remote/cluster-defaults.git branch: main owns: ["**"] priority: 0 # baseline writable: false - name: runtime-overlay backend: fs path: ./overlay owns: ["**"] priority: 100 # overrides baseline writable: true # REST writes land hereResolution rule: for each key in the merged tree, the highest-
priority source contributing that key wins. provenance shows which
source produced which key — useful in debug.
Verify with curl:
# Write via REST → lands in overlay (priority 100, writable)curl -s -X PUT \ -H "X-Api-Key: $KEY" \ -H "Content-Type: application/json" \ -d '{"feature_x": true}' \ "$BASE/hives/cluster/us-west/prod-01/myservice/2.0.0"
# Read backcurl -s -H "X-Api-Key: $KEY" \ "$BASE/hives/cluster/us-west/prod-01/myservice/2.0.0" | jq '.provenance'# → {"feature_x": "runtime-overlay@<fs-version>",# "log_level": "platform-defaults@<git-sha>"}The two sources are stitched: feature_x from the overlay,
log_level from the git baseline.
Part B — Client mode (chassis consumer side)
Section titled “Part B — Client mode (chassis consumer side)”How a kis.ai service that uses lib-chassis discovers and reads
config. Customer developers building products on top of the binary
typically use the REST API directly (Part A is what you test); kis.ai
internal services use the chassis client, which has four modes.
These are all configured in the consumer service’s own bootstrap
YAML, not in config.svc’s bootstrap. The config.svc binary
itself may run in any of the Part A modes regardless of which client
mode a consumer picks.
B.1 — bootstrap-inline (no external source)
Section titled “B.1 — bootstrap-inline (no external source)”cluster: and tenants: blocks live directly in the consumer
service’s bootstrap. No external file, no config.svc involved.
Useful for: ultra-minimal dev, air-gapped one-off binaries.
Consumer’s bootstrap YAML:
service: myserviceversion: 1.0.0sid: dev-1
host: ""port: "8080"zerotrust: false
# No config_url, no config: → bootstrap-inline mode.
cluster: kafka_brokers: ["k0:9092", "k1:9092"] log_level: info
tenants: - key: acme:forge:prod:acme-prod name: acme-prod active: true overrides: feature_x: trueWhat to verify (from inside the consumer service — kis.ai
engineers; customer devs skip this):
cc, _ := container.ResolveTyped[config.Client]("myservice-config-v2")brokers, _ := cc.GetClusterConfig(ctx)// brokers.Get("kafka_brokers").StringSlice() == ["k0:9092","k1:9092"]B.2 — local-file mode
Section titled “B.2 — local-file mode”Cluster + tenants are in a single external YAML, referenced via
config:. Useful when you want to edit one file without restarting
the consumer.
Consumer’s bootstrap:
service: myserviceversion: 1.0.0sid: dev-1
config: ./cluster.yaml # single file, chassis stats it./cluster.yaml (same schema as the inline blocks above):
cluster: kafka_brokers: ["k0:9092"]
tenants: - key: acme:forge:prod:acme-prod active: true overrides: feature_x: trueLive updates: edit cluster.yaml, save. chassis fsnotify
re-parses; change callbacks fire. No restart.
B.3 — local-dir mode
Section titled “B.3 — local-dir mode”config: points at a directory; chassis recursively walks every
*.yaml/*.yml, merges, watches.
Layout:
config/ cluster.yaml # cluster: block tenants/ acme-prod.yaml # one tenant per file globex-prod.yamlPer-file shape (same schema everywhere):
tenants: - key: acme:forge:prod:acme-prod active: true overrides: feature_x: trueConsumer’s bootstrap:
config: ./configMerge: cluster: blocks deep-merged in lex-sorted file path
order (later wins); tenants: arrays concatenated with duplicate-
key hard-error.
B.4 — remote mode (chassis → running config.svc)
Section titled “B.4 — remote mode (chassis → running config.svc)”The consumer connects to config.svc via SSE.
Consumer’s bootstrap:
service: myserviceversion: 1.0.0sid: dev-1
config_url: http://localhost:7123config_apikey: ${env:MYSERVICE_CONFIG_APIKEY}The chassis client lazy-inits — even if config.svc is down or
starting concurrently, the consumer boots successfully. Reads serve
from in-memory cache → bbolt warm-load → optional LocalConfig
fallback → empty. The SSE consumer reconnects with exponential
backoff and refreshes the cache when config.svc returns.
This is the production mode for all kis.ai services.
Part C — Combination matrix
Section titled “Part C — Combination matrix”Every server mode (Part A) is compatible with every client mode (Part B). The matrix below shows the practical combinations and what each one is for.
| Server (A) | Client (B) | Use case |
|---|---|---|
| A.1 fs single-file | B.4 remote | Dev — edit files, see clients update over SSE. |
| A.1 fs single-file | B.1 bootstrap-inline | Standalone test — no config.svc needed; consumer reads only its own bootstrap. |
| A.2 fs directory tree | B.4 remote | Most common dev workflow. |
| A.4 git (local) | B.4 remote | Pre-flight test of a git-backed deployment without a real remote. |
| A.5 git (remote) | B.4 remote | Production-equivalent. |
| A.6 postgres | B.4 remote | When the source-of-truth is a relational DB. |
| A.7 mixed (git + fs) | B.4 remote | Production overlay pattern. |
| — | B.2 local-file | Consumer tests without any config.svc instance running. |
| — | B.3 local-dir | Consumer tests with per-tenant files committed alongside the consumer’s code. |
The client doesn’t know or care which server mode is running — the SSE wire protocol is identical. That’s the point of the abstraction.
Part D — End-to-end acceptance workflow
Section titled “Part D — End-to-end acceptance workflow”A complete green-path test against any Part A server mode. The operationIds below match those in the config service OpenAPI spec; in Bruno / Postman, each is a saved request after importing the spec.
| Step | Operation (path + method) | Expected |
|---|---|---|
| 1 | GET /version | 200, JSON build info. |
| 2 | GET /ready | 200, {"status":"ready"}. |
| 3 | GET /hives | 200, {"hives":["cluster","tenant"]}. |
| 4 | GET /hives/tenant/_meta/example | 200, example value document. |
| 5 | PUT /hives/tenant/{path} (any leaf, to a writable source) | 200, response version + ETag = the source version (e.g. "1"). |
| 6 | GET /hives/tenant/{path} | 200, value matches what you wrote. (Its ETag is the per-hive snapshot version — a different counter; use the write version for If-Match.) |
| 7 | PATCH /hives/tenant/{path} (with If-Match: "{step-5 version}") | 200, merged value, new version (e.g. "2"). A stale If-Match → 409 with the current version. |
| 8 | GET /stream/config?hives=tenant (left open in another shell) | event: snapshot for steps 5 and 7. |
| 9 | DELETE /hives/tenant/{path} (with If-Match: "{step-7 version}") | 200, {"deleted":true}. |
| 10 | GET /services | 200, {"services":["config", …]} (config self-registered). |
| 11 | GET /services/config/instances | 200, one instance with tags.sid + addr. |
| 12 | GET /sources | 200, per-hive source list with priorities. |
Contract-test the whole thing in CI:
schemathesis run --base-url http://localhost:7123 \ --header "X-Api-Key: $KEY" \ --checks all \ openapi.yamlOr for a happy-path smoke without the full schemathesis property explosion, use Dredd:
dredd openapi.yaml http://localhost:7123 \ --header "X-Api-Key: $KEY"Part E — Troubleshooting matrix
Section titled “Part E — Troubleshooting matrix”| Symptom | Mode | Cause + fix |
|---|---|---|
Connection refused on curl | All | config.svc not running. Check journalctl -u config.svc or stderr of the foreground process. |
401 unauthenticated on every authed call | All | Wrong header (X-Api-Key:), or the key’s SHA-256 hash doesn’t match the entry in apikeys.yaml. Regenerate. |
403 forbidden on a specific path | All | API key valid but the role’s expr policy denies this action. Check schemas/access/<hive>.expr semantics. |
SSE shows only :connected then closes | All | (Fixed in current versions) — clear write deadline on the SSE writer; reproduce only on very old builds. |
| Snapshot version stuck at 1 | A.1 / A.2 | Edited file isn’t named exactly config.yaml. The fs source filters on that name. |
| No SSE event after file save (macOS) | A.1–A.3 | Editor uses atomic-rename-on-save; fsnotify Darwin can lose the watch. Workaround: write via REST. |
| Git changes don’t propagate | A.4 / A.5 | Default poll is 60s. Either wait, or configure a webhook. |
git: not a repository on boot | A.4 | local_open: true requires the path to be a git work tree. git init it. |
pq: connect failed on boot | A.6 | DSN wrong, PG not running, or sslmode setting mismatched. |
| Direct SQL UPDATE doesn’t fire SSE | A.6 | The trigger emitting NOTIFY is missing — re-apply config.svc generate pgsource. |
| Consumer reads stale data | B.4 | Check the consumer’s SSE connection on /subscribers. If it’s not listed, the consumer hasn’t connected. |
| Consumer “tenant not found” but tenant clearly exists | B.2 / B.3 / B.4 | CPET in ctx is incomplete. chassis TenantMiddleware extracts from headers; ensure your test request includes them. |
| Bootstrap-inline tenants ignored | B.1 | Tenant key: must be the full CPET (4 colon-segments). 1–3 segments are prefix layers, not addressable tenants. |
Where to go next
Section titled “Where to go next”- HTTP API — every endpoint, every payload.
- Quickstart — the 5-minute developer onramp.
- single-node deployment, distributed deployment — production deployment SOPs.
- chassis integration — Go-library integration for kis.ai engineers.