Skip to content
Talk to our solutions team

End-to-End Testing

Audience: customer developers and kis.ai DevSecOps. Walks through every deployment + consumption combination (except etcd, which has its own infra-heavy story in the distributed deployment guide).

For each combination, the doc gives you:

  • The bootstrap YAML.
  • What to expect on boot.
  • A curl smoke sequence.
  • The equivalent OpenAPI operation (see the config service OpenAPI spec — importable into Bruno, Postman, Insomnia, Swagger UI, or any OpenAPI-aware client).
  • The write-and-watch-SSE round-trip you’d use in your acceptance test.
  • The mode-specific failure signals to look for.

The single canonical API reference is the HTTP API. This doc focuses on which mode to use; the API reference covers how each endpoint behaves; the config service OpenAPI spec is the machine-readable contract.

ToolUsed forInstall
config.svcThe binary under test.Standard install (see the single-node deployment guide).
curlCommand-line smoke tests.Pre-installed on Linux/macOS.
jqPretty-print + extract from JSON responses.brew install jq / apt install jq.
Bruno (or Postman / Insomnia)OSS REST client. Imports the config service OpenAPI spec directly — every endpoint becomes a saved request with schemas + examples.https://www.usebruno.com — desktop app, or npm i -g @usebruno/cli for headless runs in CI.
Swagger UI (optional)Renders the config service OpenAPI spec as interactive docs in a browser.docker run -p 8080:8080 -e SWAGGER_JSON=/spec/openapi.yaml -v $PWD:/spec swaggerapi/swagger-ui
gitOnly for the git-source combinations (Part A, sections A.4 / A.5).Pre-installed on most dev machines.
TermMeaning in this doc
Server modeHow the config.svc binary itself reads its source-of-truth: filesystem (fs), git, postgres. Configured under hives.*.sources in the bootstrap.
Client modeHow a consumer service reads config via lib-chassis: remote (SSE against config.svc), local-file, local-dir, or bootstrap-inline. Configured via top-level config_url: or config: keys in the consumer’s bootstrap.
Boot configThe YAML file passed to config.svc -f <path> server. Holds chassis identity + source declarations + auth.
HiveA namespaced, schemafied config tree. The platform ships two: cluster (dc/cluster/service/version) and tenant (customer/product/env/tenant/service).
SnapshotA resolved, validated value at a specific (hive, path). Versioned per hive; pushed over SSE on change.

The config service OpenAPI spec is the machine-readable contract for the entire HTTP surface — health probes, hive CRUD, SSE streams, discovery, introspection. It groups operations by tag:

TagOperations
Health/health, /ready, /version (unauthenticated)
HivesList, read, write (PUT/PATCH), delete, schema, example, tree
ConfigStream/stream/config (SSE), /stream/config/drift
DiscoveryRegister, heartbeat, deregister, list services, list instances
DiscoveryStream/stream/discovery (SSE)
Introspection/sources, /subscribers

Import into Bruno:

  1. Bruno → ImportOpenAPI → select the config service OpenAPI spec file.
  2. Edit the Local environment (auto-created from the servers: block) to set your API key.
  3. Every operation appears as a saved request with parameters, example bodies, and response schemas pre-filled.

Gotcha: configure auth at the collection level. OpenAPI’s security: [{ApiKeyAuth: []}] tells Bruno each request needs an API key, but the spec doesn’t carry the secret value — so the Auth tab on each request is wired to “API Key” type with an empty value field. If you then add X-Api-Key manually under the Headers tab, Bruno sends two X-Api-Key headers (empty one wins → 401 “unauthenticated”). Fix once at the collection level:

  1. Right-click config service collection → SettingsAuth.
  2. Type: API Key. Key: X-Api-Key. Value: {{key}}. Placement: Header.
  3. Add a key variable to each environment with the actual plain-text key value.
  4. On each request’s Auth tab, set type to Inherit.

Now switching environments (Local / staging / prod) swaps the key automatically and no per-request configuration is needed.

Import into Postman / Insomnia: same flow — both import OpenAPI 3.x natively.

View as interactive docs (Swagger UI):

Terminal window
docker run --rm -p 8080:8080 \
-e SWAGGER_JSON=/spec/openapi.yaml \
-v $PWD:/spec \
swaggerapi/swagger-ui
# → http://localhost:8080

Generate client code: point any OpenAPI generator at openapi.yamlopenapi-generator-cli generate -i openapi.yaml -g python (or go, typescript-fetch, java, …) produces a typed client library you can drop into your application.

Headless contract testing in CI: use Schemathesis or Dredd to validate that a running instance conforms to the spec:

Terminal window
schemathesis run --base-url http://localhost:7123 \
--header "X-Api-Key: $KEY" \
openapi.yaml

Independent of which combination you’re testing, run this first to prove the binary is reachable + authenticating:

Terminal window
export BASE="http://localhost:7123"
export KEY="devkey-please-change-me"
curl -s "$BASE/health" # → {"status":"ok"}
curl -s "$BASE/ready" # → {"status":"ready"}
curl -s "$BASE/version" | jq # → JSON build info
curl -s -H "X-Api-Key: $KEY" "$BASE/hives" | jq # → {"hives":["cluster","tenant"]}

In Bruno: run the 00-health/* folder. Three green ticks → server is up and accepting your key. If any fail, do NOT proceed; see Troubleshooting.

Part A — Server mode (the config.svc binary’s source backends)

Section titled “Part A — Server mode (the config.svc binary’s source backends)”

The five source backends (excluding etcd) you’d configure under hives.*.sources in the boot YAML. Each section is self-contained: boot YAML → setup → smoke → write+watch round-trip.

A.1 — fs source: single-file values, simplest layout

Section titled “A.1 — fs source: single-file values, simplest layout”

The filesystem source watches a directory tree. Each level of the hive path is a directory; the actual value document lives in config.yaml inside the leaf directory.

Boot YAML (config.yaml):

service: config
version: 0.0.0-dev
sid: dev-local
host: ""
port: "7123"
zerotrust: false
storage:
mode: bbolt
bbolt:
path: ./cache
hives:
cluster:
sources:
- { name: local, backend: fs, path: ./cluster,
owns: ["**"], priority: 0, writable: true }
tenant:
sources:
- { name: local, backend: fs, path: ./tenant,
owns: ["**"], priority: 0, writable: true }
auth:
apikey_file: ./apikeys.yaml

Setup:

Terminal window
mkdir -p workdir && cd workdir
mkdir -p cluster tenant cache
# Generate the API key — stored plain-text in apikeys.yaml,
# compared constant-time on every request.
KEY=$(openssl rand -hex 32)
cat > apikeys.yaml <<EOF
keys:
- key: $KEY
subject_type: user
subject_id: dev
attrs:
role: platform_admin
EOF
echo "API key: $KEY"
# Write the bootstrap (from above) to ./config.yaml
config.svc -f config.yaml server

Smoke:

Terminal window
curl -s -H "X-Api-Key: $KEY" "$BASE/hives" | jq
# → {"hives":["cluster","tenant"]}
curl -s -H "X-Api-Key: $KEY" \
"$BASE/hives/tenant/" | jq
# → root snapshot, version 1, empty value

Write + watch round-trip:

Terminal window
# Terminal 1 — subscribe to SSE
curl -N -H "X-Api-Key: $KEY" "$BASE/stream/config?hives=tenant"
# Terminal 2 — write a tenant config
mkdir -p tenant/acme/forge/prod/acme-prod/forge-meta
cat > tenant/acme/forge/prod/acme-prod/forge-meta/config.yaml <<'EOF'
log_level: debug
features:
fast_path: true
EOF

Within ~250 ms, terminal 1 emits:

event: snapshot
id: 2
data: {"hive":"tenant","path":["acme","forge","prod","acme-prod","forge-meta"],"value":{...},"version":2,...}

Bruno equivalent:

  • 00-health/* to confirm reachable.
  • 10-hives/read-tenant.bru parameterised on the path.
  • 10-hives/write-tenant.bru to PUT a fresh value (writes go through the REST API → the writable fs source → the same config.yaml file appears on disk).
  • 20-sse/stream-config.bru to subscribe.

Mode-specific failure signals:

SymptomLikely cause
Edited a file, no SSE event arrivedFilename isn’t exactly config.yaml. The fs source filters on that exact name.
Direct edit via Vim/VSCode misses events on macOSEditor uses atomic-rename-on-save; fsnotify on Darwin sometimes loses the watch. Workaround: touch the file or write via REST.
Reconciler logs “validation failed”YAML body doesn’t conform to the hive’s CUE schema. curl /hives/<hive>/_meta/schema for the schema.

A.2 — fs source: directory tree mirroring the hive shape

Section titled “A.2 — fs source: directory tree mirroring the hive shape”

Same as A.1 but you author multiple node levels with intermediate defaults that leaves inherit. Directory tree:

tenant/
config.yaml # tenant-hive root defaults
acme/
config.yaml # customer-level defaults
forge/
config.yaml # product-level defaults
prod/
config.yaml # env-level defaults
acme-prod/
config.yaml # tenant-level defaults
forge-meta/
config.yaml # full leaf
forge-cli/
config.yaml

Boot YAML is identical to A.1.

Setup:

Terminal window
mkdir -p tenant/acme/forge/prod/acme-prod/forge-meta
cat > tenant/config.yaml <<'EOF'
log_level: info
EOF
cat > tenant/acme/config.yaml <<'EOF'
features:
legacy_dashboard: false
EOF
cat > tenant/acme/forge/prod/config.yaml <<'EOF'
jaeger_endpoint: prod-jaeger:14268
EOF
cat > tenant/acme/forge/prod/acme-prod/forge-meta/config.yaml <<'EOF'
log_level: debug # overrides root "info"
EOF

Verify resolution (this is the interesting part):

Terminal window
curl -s -H "X-Api-Key: $KEY" \
"$BASE/hives/tenant/acme/forge/prod/acme-prod/forge-meta" | jq '.value'

Expect:

{
"log_level": "debug", # leaf wins over root
"features": { "legacy_dashboard": false },# inherited from acme
"jaeger_endpoint": "prod-jaeger:14268" # inherited from prod
}

The provenance field on the snapshot shows which directory each key came from.

Bruno: 10-hives/read-tenant.bru — set the path variable to acme/forge/prod/acme-prod/forge-meta and inspect value + provenance in the response.

A.3 — fs source: cluster + tenant mounted under one root

Section titled “A.3 — fs source: cluster + tenant mounted under one root”

Some teams prefer one source root rather than two:

hives:
cluster:
sources:
- { name: local, backend: fs, path: ./hives/cluster,
owns: ["**"], priority: 0, writable: true }
tenant:
sources:
- { name: local, backend: fs, path: ./hives/tenant,
owns: ["**"], priority: 0, writable: true }

Two paths, same parent. Functionally identical to A.1/A.2; the difference is purely operational (one git repo for both vs. two).

git source can open a local checkout directly — useful for testing without a remote git server.

Setup:

Terminal window
mkdir -p sources/cluster-defaults && cd sources/cluster-defaults
git init -q
echo "log_level: info" > config.yaml
git add config.yaml && git -c user.email=ci@local commit -qm initial
cd -

Boot YAML (only the hives: section differs):

hives:
cluster:
sources:
- name: local-git
backend: git
url: ./sources/cluster-defaults # local path, not a URL
local_open: true # treat as a local working tree
branch: main
owns: ["**"]
priority: 0
writable: false # writes via REST go to a higher-priority writable source below
- name: overlay
backend: fs
path: ./overlay
owns: ["**"]
priority: 20
writable: true

Smoke: identical to A.1.

Write round-trip: REST writes land in the overlay fs source (higher priority + writable). Git source is read-only here; updates appear by committing to the local repo and waiting for the reconciler’s poll cadence (default 60s).

Mode-specific failure signals:

SymptomCause
Boot fails with git: not a repositoryThe path isn’t a git repo — run git init there first.
Changes committed to the repo don’t appear via APIPoll cadence is 60s. Reconciler logs git source local-git: polled, no change on each tick; commit, wait, re-poll.

A.5 — git source: remote repo (HTTPS / SSH)

Section titled “A.5 — git source: remote repo (HTTPS / SSH)”

Production-style. Two flavors:

HTTPS with PAT:

sources:
- name: platform-cluster-defaults
backend: git
url: https://git.example.com/platform/cluster-defaults.git
branch: main
auth_username: x-access-token
auth_token: ${env:GIT_TOKEN}
owns: ["**"]
priority: 0
writable: false

Export GIT_TOKEN before starting config.svc. The reconciler does a fresh clone into a temp directory on first boot; subsequent polls do git pull.

SSH key:

sources:
- name: platform-cluster-defaults
backend: git
url: [email protected]:platform/cluster-defaults.git
branch: main
auth_ssh_key_file: /etc/config/git-ssh-key
owns: ["**"]
priority: 0
writable: false

Testing locally without a real remote:

Spin up a local bare repo to act as the remote:

Terminal window
mkdir -p /tmp/git-remote/cluster-defaults.git && cd /tmp/git-remote/cluster-defaults.git
git init --bare -q
cd -
git clone /tmp/git-remote/cluster-defaults.git /tmp/git-work
echo "log_level: info" > /tmp/git-work/config.yaml
git -C /tmp/git-work add . && git -C /tmp/git-work -c user.email=ci@local commit -qm initial
git -C /tmp/git-work push -q

Point your boot YAML at file:///tmp/git-remote/cluster-defaults.git (go-git supports file:// URLs as if they were remote). No auth needed.

Write round-trip via git: push to the remote, wait for the reconciler’s poll. To shortcut the wait, configure a webhook (see the single-node deployment guide §7.5) and POST to it from your git provider on push.

Each hive snapshot row in a single table.

Setup (assuming PG running locally):

Terminal window
createdb config_test
psql config_test < <(config.svc generate pgsource) # ships a template DDL
psql config_test -c "
INSERT INTO infinity_config_overrides (hive, path, value)
VALUES ('tenant', 'acme/forge/prod/acme-prod/forge-meta',
'{\"log_level\":\"debug\"}'::jsonb);
"

Boot YAML:

hives:
tenant:
sources:
- name: pg-overrides
backend: postgres
dsn: postgres://localhost:5432/config_test?sslmode=disable
table: infinity_config_overrides
owns: ["**"]
priority: 0
writable: true

Write round-trip:

REST writes go through PG INSERT … ON CONFLICT DO UPDATE. The reconciler picks them up via PG LISTEN/NOTIFY. SSE events fire within milliseconds.

You can also write directly with psql — the LISTEN/NOTIFY path catches it the same way.

Mode-specific failure signals:

SymptomCause
Boot fails with pgsource: connect: …DSN wrong, PG not running, or sslmode setting mismatched.
Direct SQL UPDATE doesn’t fire SSEThe trigger that emits NOTIFY is missing — config.svc generate pgsource includes it; if you customized the DDL, ensure it’s still there.

The realistic production layout: a baseline source (low priority, read-only) + an overlay (high priority, writable).

hives:
cluster:
sources:
- name: platform-defaults
backend: git
url: file:///tmp/git-remote/cluster-defaults.git
branch: main
owns: ["**"]
priority: 0 # baseline
writable: false
- name: runtime-overlay
backend: fs
path: ./overlay
owns: ["**"]
priority: 100 # overrides baseline
writable: true # REST writes land here

Resolution rule: for each key in the merged tree, the highest- priority source contributing that key wins. provenance shows which source produced which key — useful in debug.

Verify with curl:

Terminal window
# Write via REST → lands in overlay (priority 100, writable)
curl -s -X PUT \
-H "X-Api-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"feature_x": true}' \
"$BASE/hives/cluster/us-west/prod-01/myservice/2.0.0"
# Read back
curl -s -H "X-Api-Key: $KEY" \
"$BASE/hives/cluster/us-west/prod-01/myservice/2.0.0" | jq '.provenance'
# → {"feature_x": "runtime-overlay@<fs-version>",
# "log_level": "platform-defaults@<git-sha>"}

The two sources are stitched: feature_x from the overlay, log_level from the git baseline.

Part B — Client mode (chassis consumer side)

Section titled “Part B — Client mode (chassis consumer side)”

How a kis.ai service that uses lib-chassis discovers and reads config. Customer developers building products on top of the binary typically use the REST API directly (Part A is what you test); kis.ai internal services use the chassis client, which has four modes.

These are all configured in the consumer service’s own bootstrap YAML, not in config.svc’s bootstrap. The config.svc binary itself may run in any of the Part A modes regardless of which client mode a consumer picks.

B.1 — bootstrap-inline (no external source)

Section titled “B.1 — bootstrap-inline (no external source)”

cluster: and tenants: blocks live directly in the consumer service’s bootstrap. No external file, no config.svc involved. Useful for: ultra-minimal dev, air-gapped one-off binaries.

Consumer’s bootstrap YAML:

service: myservice
version: 1.0.0
sid: dev-1
host: ""
port: "8080"
zerotrust: false
# No config_url, no config: → bootstrap-inline mode.
cluster:
kafka_brokers: ["k0:9092", "k1:9092"]
log_level: info
tenants:
- key: acme:forge:prod:acme-prod
name: acme-prod
active: true
overrides:
feature_x: true

What to verify (from inside the consumer service — kis.ai engineers; customer devs skip this):

cc, _ := container.ResolveTyped[config.Client]("myservice-config-v2")
brokers, _ := cc.GetClusterConfig(ctx)
// brokers.Get("kafka_brokers").StringSlice() == ["k0:9092","k1:9092"]

Cluster + tenants are in a single external YAML, referenced via config:. Useful when you want to edit one file without restarting the consumer.

Consumer’s bootstrap:

service: myservice
version: 1.0.0
sid: dev-1
config: ./cluster.yaml # single file, chassis stats it

./cluster.yaml (same schema as the inline blocks above):

cluster:
kafka_brokers: ["k0:9092"]
tenants:
- key: acme:forge:prod:acme-prod
active: true
overrides:
feature_x: true

Live updates: edit cluster.yaml, save. chassis fsnotify re-parses; change callbacks fire. No restart.

config: points at a directory; chassis recursively walks every *.yaml/*.yml, merges, watches.

Layout:

config/
cluster.yaml # cluster: block
tenants/
acme-prod.yaml # one tenant per file
globex-prod.yaml

Per-file shape (same schema everywhere):

config/tenants/acme-prod.yaml
tenants:
- key: acme:forge:prod:acme-prod
active: true
overrides:
feature_x: true

Consumer’s bootstrap:

config: ./config

Merge: cluster: blocks deep-merged in lex-sorted file path order (later wins); tenants: arrays concatenated with duplicate- key hard-error.

B.4 — remote mode (chassis → running config.svc)

Section titled “B.4 — remote mode (chassis → running config.svc)”

The consumer connects to config.svc via SSE.

Consumer’s bootstrap:

service: myservice
version: 1.0.0
sid: dev-1
config_url: http://localhost:7123
config_apikey: ${env:MYSERVICE_CONFIG_APIKEY}

The chassis client lazy-inits — even if config.svc is down or starting concurrently, the consumer boots successfully. Reads serve from in-memory cache → bbolt warm-load → optional LocalConfig fallback → empty. The SSE consumer reconnects with exponential backoff and refreshes the cache when config.svc returns.

This is the production mode for all kis.ai services.

Every server mode (Part A) is compatible with every client mode (Part B). The matrix below shows the practical combinations and what each one is for.

Server (A)Client (B)Use case
A.1 fs single-fileB.4 remoteDev — edit files, see clients update over SSE.
A.1 fs single-fileB.1 bootstrap-inlineStandalone test — no config.svc needed; consumer reads only its own bootstrap.
A.2 fs directory treeB.4 remoteMost common dev workflow.
A.4 git (local)B.4 remotePre-flight test of a git-backed deployment without a real remote.
A.5 git (remote)B.4 remoteProduction-equivalent.
A.6 postgresB.4 remoteWhen the source-of-truth is a relational DB.
A.7 mixed (git + fs)B.4 remoteProduction overlay pattern.
B.2 local-fileConsumer tests without any config.svc instance running.
B.3 local-dirConsumer tests with per-tenant files committed alongside the consumer’s code.

The client doesn’t know or care which server mode is running — the SSE wire protocol is identical. That’s the point of the abstraction.

A complete green-path test against any Part A server mode. The operationIds below match those in the config service OpenAPI spec; in Bruno / Postman, each is a saved request after importing the spec.

StepOperation (path + method)Expected
1GET /version200, JSON build info.
2GET /ready200, {"status":"ready"}.
3GET /hives200, {"hives":["cluster","tenant"]}.
4GET /hives/tenant/_meta/example200, example value document.
5PUT /hives/tenant/{path} (any leaf, to a writable source)200, response version + ETag = the source version (e.g. "1").
6GET /hives/tenant/{path}200, value matches what you wrote. (Its ETag is the per-hive snapshot version — a different counter; use the write version for If-Match.)
7PATCH /hives/tenant/{path} (with If-Match: "{step-5 version}")200, merged value, new version (e.g. "2"). A stale If-Match409 with the current version.
8GET /stream/config?hives=tenant (left open in another shell)event: snapshot for steps 5 and 7.
9DELETE /hives/tenant/{path} (with If-Match: "{step-7 version}")200, {"deleted":true}.
10GET /services200, {"services":["config", …]} (config self-registered).
11GET /services/config/instances200, one instance with tags.sid + addr.
12GET /sources200, per-hive source list with priorities.

Contract-test the whole thing in CI:

Terminal window
schemathesis run --base-url http://localhost:7123 \
--header "X-Api-Key: $KEY" \
--checks all \
openapi.yaml

Or for a happy-path smoke without the full schemathesis property explosion, use Dredd:

Terminal window
dredd openapi.yaml http://localhost:7123 \
--header "X-Api-Key: $KEY"
SymptomModeCause + fix
Connection refused on curlAllconfig.svc not running. Check journalctl -u config.svc or stderr of the foreground process.
401 unauthenticated on every authed callAllWrong header (X-Api-Key:), or the key’s SHA-256 hash doesn’t match the entry in apikeys.yaml. Regenerate.
403 forbidden on a specific pathAllAPI key valid but the role’s expr policy denies this action. Check schemas/access/<hive>.expr semantics.
SSE shows only :connected then closesAll(Fixed in current versions) — clear write deadline on the SSE writer; reproduce only on very old builds.
Snapshot version stuck at 1A.1 / A.2Edited file isn’t named exactly config.yaml. The fs source filters on that name.
No SSE event after file save (macOS)A.1–A.3Editor uses atomic-rename-on-save; fsnotify Darwin can lose the watch. Workaround: write via REST.
Git changes don’t propagateA.4 / A.5Default poll is 60s. Either wait, or configure a webhook.
git: not a repository on bootA.4local_open: true requires the path to be a git work tree. git init it.
pq: connect failed on bootA.6DSN wrong, PG not running, or sslmode setting mismatched.
Direct SQL UPDATE doesn’t fire SSEA.6The trigger emitting NOTIFY is missing — re-apply config.svc generate pgsource.
Consumer reads stale dataB.4Check the consumer’s SSE connection on /subscribers. If it’s not listed, the consumer hasn’t connected.
Consumer “tenant not found” but tenant clearly existsB.2 / B.3 / B.4CPET in ctx is incomplete. chassis TenantMiddleware extracts from headers; ensure your test request includes them.
Bootstrap-inline tenants ignoredB.1Tenant key: must be the full CPET (4 colon-segments). 1–3 segments are prefix layers, not addressable tenants.