Vault End-to-End Testing

Role	What they verify
Customer developer	”Can my service get its secrets / mint a JWT from the local agent socket on my dev box?“
kis.ai DevSecOps	”Did my install land correctly? Server up, agent up, attestation working, secrets/JWT round-trips clean, audit chain healthy, rotation actually rotating?”

This document is a self-contained test plan for the Vault service. Every command is copy-pasteable. Every API call has the matching curl and Bruno entry. We cover every non-cloud combination of vault.svc + vault-agent — fsvault / OpenBao / self-hosted HCP Vault for the backend, in-process RSA / SoftHSM2 for the signer, and every topology from “single binary on laptop” to “3-node HA in docker-compose.”

What works today vs. what’s scaffolded. A handful of routes return HTTP 501 with a phase marker (/workload/secret/get|write|delete, /workload/jwt returns a stub token, /operator/audit/*, /operator/root/*, /operator/backend/migrate, /operator/signer/test, /operator/policy/lint). Each test section calls these out so you know whether a 501 means “broken” or “expected pending phase X.”

1. Prerequisites

1.1 Software

Tool	Min version	Why
curl	7.40	UDS support via --unix-socket
jq	1.6	Parse JSON responses
openssl	3.0	Generate test CAs and operator certs
bruno	1.20 (download)	API explorer; OSS replacement for Postman
kis CLI	latest release	Convenience wrapper over the agent socket
docker + docker-compose	24.x / v2	Multi-container topologies
softhsm2	2.6	Local HSM emulation for PKCS#11 tests
vault (HashiCorp)	1.14	Optional — backend tests
bao (OpenBao)	2.0	Optional — backend tests
PostgreSQL	15+	Vault’s own store — certificates, secrets and their audit trails

macOS / Linux: every command in this doc works on both. Windows users should run inside WSL2.

1.2 Materialise the database first

Vault keeps its certificates, secrets and audit trails in Postgres, and the schema has to exist before the service will start. Create the database, point the config at it, then:

vault data bootstrap -f .vault.yaml

That diffs the composed schema against the empty database and applies the whole plan in one shot — every table and index, the per-tenant schema namespace, and the migration ledger’s own tables. It is idempotent: re-running emits only IF NOT EXISTS no-ops, so it is safe in a provisioning flow that may run twice.

vault data bootstrap -f .vault.yaml --tenant acme:app:prod:eu1   # a specific tenant
vault data bootstrap -f .vault.yaml --dry-run                    # print the plan, apply nothing

Reach for --dry-run before the first run against anything you care about — it prints exactly what would be applied.

Vault signs its own certificate on first boot

Vault is a certificate authority, so a standalone root-mode boot issues and stores its own service certificate rather than asking anyone for one. That is the end-to-end check worth running after bootstrap: start the service, confirm /health answers, and confirm the certificate landed in Postgres. If it did, the CA, the engine and the store are all working together.

Symptom	Cause	Fix
relation "vault.certificates" does not exist	The schema was never materialised	Run vault data bootstrap -f .vault.yaml

1.3 Network ports used

Port	Service	Notes
7999	vault.svc HTTPS	Configurable via httpserver.listenaddress
7991	vault.svc gRPC (future)	Reserved
n/a — UDS	vault-agent	/run/vault/vault.sock (root) or /tmp/vault/vault.sock (rootless dev)
8200	HashiCorp Vault (optional backend)	Default
8200	OpenBao (optional backend)	Default — same as HCP Vault, run on different hosts

1.4 Working directory

Everything below assumes you’re in a scratch directory:

mkdir -p ~/kis-vault-test && cd ~/kis-vault-test

All generated certs, config files, and SoftHSM tokens land here.

2. The five-minute smoke test

Goal: prove vault.svc starts, accepts a healthz, and your toolchain (curl + jq + Bruno) talks to it. Run this before any of the longer recipes — it isolates “is my environment broken” from “is the test broken.”

2.1 Run a stripped-down server

# Generate a throwaway CA + leaf for vault.svc itself
openssl req -x509 -newkey rsa:2048 -nodes -keyout test-ca.key -out test-ca.crt \
  -days 30 -subj "/CN=kis-test-ca"

openssl req -newkey rsa:2048 -nodes -keyout vault-svc.key -out vault-svc.csr \
  -subj "/CN=vault.svc.local"

openssl x509 -req -in vault-svc.csr -CA test-ca.crt -CAkey test-ca.key -CAcreateserial \
  -out vault-svc.crt -days 30 \
  -extfile <(printf "subjectAltName=DNS:vault.svc.local,DNS:localhost,IP:127.0.0.1")

# Minimal config for fsvault + inproc signer, listening on 7999
cat > vault.svc.yaml <<'EOF'
service:
  name: vault.svc
  http:
    listenaddress: 127.0.0.1:7999

vaultmanager:
  store:
    fsvault:
      filepath: ./fsvault-data/secrets.yaml
      env: dev

vault:
  signer:
    type: inproc

tls:
  ca_certs: ["./test-ca.crt"]
  cert:    "./vault-svc.crt"
  key:     "./vault-svc.key"
  require_client_cert: false   # smoke test only; flip back on for real
EOF

mkdir -p fsvault-data

# Start it (foreground; ctrl-c to stop)
vault.svc -config ./vault.svc.yaml

2.2 Hit it

In another shell:

curl -sk https://127.0.0.1:7999/healthz | jq .
# expect: {"status":"ok"} or similar — anything 2xx is healthy

If you get JSON back, stop here and continue with the recipes. If not, fix the environment before going further.

2.3 Common smoke-test failures

Symptom	Likely cause	Fix
curl: (7) Failed to connect	server didn’t bind; check stderr	lsof -i :7999
SSL routines:tls_process_server_certificate:certificate verify failed	self-signed CA	use -k (smoke) or --cacert test-ca.crt
bad config: yaml: line N	YAML indentation	tabs vs. spaces; this config is spaces only
Server prints signer: inproc must be constructed via NewInProcWithKey	the inproc signer needs a key file; see §3.1

3. Setup recipes — every non-cloud combination

Each recipe lists: what you’re testing, how to start it, and what to verify. Recipes share the test-CA from §2.1; don’t regenerate.

3.1 Local binary, fsvault, inproc signer, no agent

The simplest end-to-end. Server only; consumers talk to it directly via HTTPS (legacy transport: remote).

Start

# Generate the CA private key the inproc signer will use as the issuer
openssl genrsa -out vault-ca.key 4096
openssl req -x509 -new -key vault-ca.key -out vault-ca.crt -days 3650 \
  -subj "/CN=kis Vault Root CA"

# Server config (replaces the smoke-test one)
cat > vault.svc.yaml <<'EOF'
service:
  name: vault.svc
  http: { listenaddress: 127.0.0.1:7999 }
vaultmanager:
  store:
    fsvault: { filepath: ./fsvault-data/secrets.yaml, env: dev }
vault:
  signer:
    type: inproc
    inproc:
      ca_cert: ./vault-ca.crt
      ca_key:  ./vault-ca.key
tls:
  ca_certs: ["./test-ca.crt"]
  cert:    "./vault-svc.crt"
  key:     "./vault-svc.key"
  require_client_cert: true
EOF

vault.svc -config ./vault.svc.yaml

Verify

# Server-side health
curl -sk https://127.0.0.1:7999/healthz | jq .

# Operator-issued cert (using the test CA's leaf as the operator's mTLS pair)
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key https://127.0.0.1:7999/healthz

Tests: server boot, fsvault initialization, inproc signer key load, mTLS handshake. Does NOT test: agent attestation, JWT minting, /workload/* routes.

3.2 Local binary + agent on same host

The reference dev setup. Server on 127.0.0.1:7999, agent on /tmp/vault/vault.sock, both share the test CA.

Bootstrap the agent’s node identity

The agent needs a node cert. We mint a single-use token from the server, then vault-agent init exchanges it.

# Issue an operator token (operator cert is test-ca + vault-svc leaf for now)
TOKEN=$(curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/operator/node-token/issue \
  -H 'Content-Type: application/json' \
  -d '{"dc":"local","cluster":"dev","ttl":"30m"}' \
  | jq -r .token)
echo "Issued token: $TOKEN"

Start the agent

sudo mkdir -p /run/vault /var/lib/vault
sudo chown $(whoami) /run/vault /var/lib/vault

vault-agent init \
  --server https://127.0.0.1:7999 \
  --token "$TOKEN" \
  --trust-bundle ./test-ca.crt \
  --dc local --cluster dev

vault-agent run --foreground &
AGENT_PID=$!

Verify

# Health probe (unauthenticated)
curl --unix-socket /run/vault/vault.sock http://agent/healthz

# Attested identity probe — uses the calling shell's pid for SO_PEERCRED
curl --unix-socket /run/vault/vault.sock http://agent/workload/identity \
  -X GET
# expect 401 unless you've registered a selector matching curl's binary

Tests: server↔agent handshake, node-cert issuance, refresh-token storage, agent socket listening, SO_PEERCRED attestation refusal for unregistered workloads.

3.3 Server + agent in separate Docker containers

Closer to production: server in one container, agent in another, on the same Docker network. Tests the mTLS hop “for real” (different routable hosts).

docker-compose.yaml

version: "3.8"
services:
  vault-svc:
    image: kis-ai/vault.svc:dev
    volumes:
      - ./test-ca.crt:/etc/vault/test-ca.crt:ro
      - ./vault-svc.crt:/etc/vault/vault-svc.crt:ro
      - ./vault-svc.key:/etc/vault/vault-svc.key:ro
      - ./vault-ca.crt:/etc/vault/vault-ca.crt:ro
      - ./vault-ca.key:/etc/vault/vault-ca.key:ro
      - vault-data:/var/lib/vault
    command: ["-config", "/etc/vault/vault.svc.yaml"]
    networks: { kisnet: { aliases: [vault.svc] } }
    ports: ["7999:7999"]

  vault-agent:
    image: kis-ai/vault-agent:dev
    depends_on: [vault-svc]
    volumes:
      - ./test-ca.crt:/etc/vault/test-ca.crt:ro
      - agent-state:/var/lib/vault
      - /run/vault:/run/vault
    cap_add: [SYS_PTRACE]   # for /proc/<pid>/exe reads of peer workloads
    command:
      - run
      - --foreground

volumes: { vault-data: {}, agent-state: {} }
networks: { kisnet: {} }

docker compose up -d
# Bootstrap the agent (still requires a token issued by the operator)
TOKEN=$(docker compose exec vault-svc /usr/local/bin/vault.svc operator node-token issue \
  --dc local --cluster dev --ttl 30m | jq -r .token)
docker compose exec vault-agent vault-agent init \
  --server https://vault.svc:7999 --token "$TOKEN" \
  --trust-bundle /etc/vault/test-ca.crt --dc local --cluster dev

Tests: real DNS resolution, container-isolated trust bundle, two-process /healthz, mTLS-over-bridge-network handshake.

3.4 Server VM + agent VM (two-host)

Same as §3.3 but the network hop is real (different machines, real interfaces). Use Vagrant, two laptops, or two cloud VMs (skip cloud-managed services — use plain t3.micro running our binaries).

# On VM-A (server):
ip addr show           # note the LAN IP, e.g. 192.168.1.20
vault.svc -config /etc/vault/vault.svc.yaml

# On VM-B (agent):
echo "192.168.1.20 vault.svc.local" | sudo tee -a /etc/hosts
vault-agent init --server https://vault.svc.local:7999 \
  --token "$TOKEN" --trust-bundle /etc/vault/test-ca.crt --dc local --cluster dev
vault-agent run --foreground

Tests: all of §3.3 plus real-network NIC behavior (MTU, TLS-SNI), recovery from packet loss (drop a few ICMPs with tc to simulate).

3.5 OpenBao backend

OpenBao is wire-compatible with HashiCorp Vault. Useful for sites that want OSS-licensed backend without the BSL.

Start OpenBao alongside the server

# Run OpenBao in dev mode (root token printed on stdout — capture it)
docker run --rm -d --name openbao -p 8200:8200 \
  -e BAO_DEV_ROOT_TOKEN_ID=dev-root \
  -e BAO_DEV_LISTEN_ADDRESS=0.0.0.0:8200 \
  openbao/openbao:latest server -dev

# Enable the v2 KV secrets engine at the prefix the Vault service writes under
docker exec openbao bao secrets enable -version=2 -path=secret kv

# The backend authenticates with AppRole or a client cert — never a root token.
# AppRole is the least ceremony for a local run:
docker exec openbao bao auth enable approle
docker exec openbao bao write auth/approle/role/vault \
  token_policies=default token_ttl=1h token_max_ttl=4h
docker exec openbao bao read -field=role_id auth/approle/role/vault/role-id
docker exec openbao bao write -f -field=secret_id auth/approle/role/vault/secret-id

Vault.svc config block

vaultmanager:
  store:
    openbao:
      url: http://127.0.0.1:8200
      auth_method: approle
      secretspathprefix: secret          # default
      certificatespathprefix: kisaicerts # default
      approle:
        role_id: <role_id from above>
        secret_id_env: BAO_SECRET_ID     # export before starting the server
        mount_path: approle              # default

Caution

There is no token auth method. auth_method accepts cert (the default, mTLS against /v1/auth/cert/login) and approle — nothing else. A config naming a root token is ignored and the backend fails to connect with vault-hcpvault-failed-to-connect.

Verify

# Put a secret via the server, confirm it shows up in OpenBao directly
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/secret/myname \
  -H 'Content-Type: application/json' \
  -d '{"secret":"hunter2","dc":"local","cluster":"dev"}'

# Read back via OpenBao CLI to confirm storage location
docker exec -e BAO_TOKEN=dev-root openbao bao kv get secret/secret/local/dev/myname

Tests: backend swap, version cap, key-collision behaviour.

3.6 HashiCorp Vault self-hosted (no HCP cloud)

Identical to §3.5 with bao → vault and the hcpvault store key instead of openbao. Useful if your org already runs HashiCorp Vault on-prem. The two backends share one implementation, so the config shape is the same — swapping the key is the whole change.

docker run --rm -d --name hcv -p 8200:8200 \
  -e VAULT_DEV_ROOT_TOKEN_ID=dev-root \
  hashicorp/vault:latest

docker exec hcv vault secrets enable -version=2 -path=secret kv
docker exec hcv vault auth enable approle
docker exec hcv vault write auth/approle/role/vault \
  token_policies=default token_ttl=1h token_max_ttl=4h

vaultmanager:
  store:
    hcpvault:
      url: http://127.0.0.1:8200
      auth_method: approle
      approle:
        role_id: <role_id>
        secret_id_env: VAULT_SECRET_ID

For a production deployment use the default cert method instead, which authenticates with a client certificate:

vaultmanager:
  store:
    hcpvault:
      url: https://vault.internal.example.com:8200
      auth_method: cert
      certpath:   /etc/vault/certs/vault-client.crt
      keypath:    /etc/vault/certs/vault-client.key
      cacertpath: /etc/vault/certs/issuer.crt
      authname:   vault-cert

3.7 PKCS#11 signer via SoftHSM2

Tests the HSM path without buying hardware.

# Init a SoftHSM token + generate a CA key on it
mkdir -p softhsm-tokens
cat > softhsm2.conf <<EOF
directories.tokendir = $PWD/softhsm-tokens
objectstore.backend = file
EOF
export SOFTHSM2_CONF=$PWD/softhsm2.conf

softhsm2-util --init-token --slot 0 --label vault-test \
  --pin 1234 --so-pin 5678

# Generate a 4096-bit RSA key labeled vault-ca-key
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
  --login --pin 1234 \
  --keypairgen --key-type RSA:4096 \
  --label vault-ca-key --id 01

# Tell vault.svc to use it
cat >> vault.svc.yaml <<'EOF'
vault:
  signer:
    type: pkcs11
    pkcs11:
      module_path: /usr/lib/softhsm/libsofthsm2.so
      slot_label: vault-test
      pin_env: SOFTHSM_PIN
      key_label: vault-ca-key
EOF

export SOFTHSM_PIN=1234
vault.svc -config ./vault.svc.yaml

Issue a leaf and verify the issuer’s pubkey matches the HSM’s pubkey:

# Export the HSM pubkey for comparison
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --pin 1234 \
  --read-object --label vault-ca-key --type pubkey --output-file hsm-pub.der
openssl pkey -inform DER -pubin -in hsm-pub.der -text -noout

# Issue a workload cert and inspect its issuer
# (uses the legacy /certificate/generate/internal endpoint; substitute your CN)
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/certificate/generate/internal \
  -H 'Content-Type: application/json' \
  -d '{"common_name":"hsm-test","dc":"local","cluster":"dev"}'

Tests: PKCS#11 module load, key location by label, signing through the HSM session. Failure modes: missing pin → server refuses to boot with a clear pkcs11.pin_env is empty error.

3.8 File-mode delivery (Postgres-style workload)

Use this when a workload (Postgres, ClickHouse, third-party binary) can’t speak the agent socket protocol.

Agent config

# /etc/vault/agent.yaml (or wherever your agent reads from)
file_deliveries:
  - name: postgres-prod
    identity: spiffe://kis.dev/workload/postgres
    cert_path:         /var/lib/postgresql/server.crt
    key_path:          /var/lib/postgresql/server.key
    trust_bundle_path: /var/lib/postgresql/trust.pem
    owner: postgres
    group: postgres
    mode_cert:   "0644"
    mode_key:    "0600"
    mode_bundle: "0644"
    allow_chown: true
    rotation_interval: 12h
    reload:
      type: signal
      signal: SIGHUP
      pid_file: /var/run/postgresql/postgres.pid

Verify

# Watch the agent rotation log
journalctl -u vault-agent -f &

# Force a rotation immediately
sudo systemctl reload vault-agent

# Confirm Postgres saw the new cert
sudo -u postgres openssl x509 -in /var/lib/postgresql/server.crt -noout -dates
# notBefore should be within the last minute

Tests: atomic write, fsync, chown without disrupting Postgres, SIGHUP delivery. Edge case to confirm: kill Postgres mid-rotation. Should never see a half-written cert because rename is atomic.

3.9 Canary transport (transport: both)

The migration tool (kis vault migrate-service) recommends transport: both as the canary state. To verify shape parity between the two transports, run them side by side in two terminals and compare:

# Terminal A — direct HTTPS (legacy)
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X PATCH https://127.0.0.1:7999/secret/myname \
  -H 'Content-Type: application/json' \
  -d '{"dc":"local","cluster":"dev"}'

# Terminal B — through the agent
curl --unix-socket /run/vault/vault.sock http://agent/workload/secret \
  -X POST -H 'Content-Type: application/json' \
  -d '{"name":"vault:myname"}'

# Diff the bodies — shapes should match (modulo the agent's selector-derived tenant scoping)

3.10 3-node HA cluster (docker-compose)

For DevSecOps validating HA failover.

docker-compose.ha.yaml

version: "3.8"
services:
  vault-svc-a: { image: kis-ai/vault.svc:dev, ... networks: [kisnet] }
  vault-svc-b: { image: kis-ai/vault.svc:dev, ... networks: [kisnet] }
  vault-svc-c: { image: kis-ai/vault.svc:dev, ... networks: [kisnet] }
  nginx:
    image: nginx:alpine
    volumes: [./nginx.conf:/etc/nginx/nginx.conf:ro]
    ports: ["7999:7999"]
    networks: [kisnet]
  agent-1: { image: kis-ai/vault-agent:dev, ... }
  agent-2: { image: kis-ai/vault-agent:dev, ... }
  agent-3: { image: kis-ai/vault-agent:dev, ... }

nginx.conf round-robins TCP over the three server replicas. Backend is shared OpenBao or a shared fsvault-data volume.

Tests: kill vault-svc-a mid-request, expect retry to land on -b or -c. Refresh-token flow must work across the failover.

4. Test plan — every API surface with curl

We exercise four route families. All examples assume a running setup from §3.

4.1 Health

Method	Path	Auth	Notes
GET	/healthz	none	Liveness, no chain-of-trust

curl -sk https://127.0.0.1:7999/healthz
# response: {"status":"ok"}

Agent’s parallel:

curl --unix-socket /run/vault/vault.sock http://agent/healthz
# response: {"status":"ok"}

4.2 Workload endpoints (agent → server)

These are called by vault-agent against vault.svc over mTLS, on behalf of attested workloads. You typically don’t curl them directly except for testing.

POST /workload/identity

curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/workload/identity \
  -H 'Content-Type: application/json' \
  -d '{
    "selectors": {
      "binary_path": "/usr/local/bin/vulcan",
      "uid": 1000
    }
  }'

# response (200):
# {"spiffe_id":"spiffe://kis.dev/workload/vulcan","policy":"vulcan-default"}

# response (404 when no entry matches):
# {"code":"vault-workload-error","message":"no registration entry matches..."}

# response (409 ambiguous):
# {"code":"vault-workload-error","message":"selectors match multiple entries..."}

Pre-register an entry so the 200 path works:

curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/operator/identity \
  -H 'Content-Type: application/json' \
  -d '{
    "spiffe_id": "spiffe://kis.dev/workload/vulcan",
    "selectors": {"binary_path": "/usr/local/bin/vulcan"},
    "policy": "vulcan-default"
  }'

POST /workload/jwt

curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/workload/jwt \
  -H 'Content-Type: application/json' \
  -d '{
    "selectors": {"binary_path": "/usr/local/bin/vulcan"},
    "audience": ["janus"]
  }'

Today’s response (stub):

{
  "token": "vault.stub-jwt.spiffe://kis.dev/workload/vulcan.aud=janus",
  "spiffe_id": "spiffe://kis.dev/workload/vulcan",
  "note": "Phase-4-jwt: stub token; real SVID issuance lands with the JWT-SVID work item"
}

When real JWT-SVID lands, the token field becomes a signed JWS with rcv_aud: ["janus"]. The stub is intentional so consumers can wire shape today.

POST /workload/secret/{get,write,delete}

Currently 501. Calling them confirms the routes exist:

curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/workload/secret/get \
  -H 'Content-Type: application/json' \
  -d '{"selectors":{"binary_path":"/x"},"name":"vault:foo"}'

# response (501):
# {"code":"vault-operator-scaffold","message":"workload secret-get is scaffolded;
#  SPIFFE→tenant mapping lands in Phase 4-secrets"}

4.3 Agent handshake

POST /agent/join

curl -sk --cacert ./test-ca.crt \
  -X POST https://127.0.0.1:7999/agent/join \
  -H 'Content-Type: application/json' \
  -d '{
    "secret": "hj_v1_abc12_DEADBEEF...",
    "attributes": {
      "hostname": "test-host",
      "machine_id": "uuid-here",
      "os": "linux"
    },
    "dc": "local",
    "cluster": "dev"
  }'

# 200 — returns node cert chain PEM + key PEM + trust bundle PEM + refresh token
# 401 — token invalid / consumed / scope mismatch

POST /agent/renew

curl -sk --cacert ./test-ca.crt \
  -X POST https://127.0.0.1:7999/agent/renew \
  -H 'Content-Type: application/json' \
  -d '{"refresh_token":"rt_v1_xxx..."}'

# 200 — fresh node cert chain + (typically same) refresh token
# 401 — refresh token expired or revoked

POST /agent/decommission

curl -sk --cacert ./test-ca.crt \
  -X POST https://127.0.0.1:7999/agent/decommission \
  -H 'Content-Type: application/json' \
  -d '{"node_spiffe_id":"spiffe://kis.dev/node/abc","reason":"reimage"}'

# 200 — {"status":"ok"}

4.4 Operator endpoints

All require an operator mTLS cert. The cert’s SAN encodes the role (today not yet enforced — P1 follow-up; any cert the cluster trusts is treated as operator-privileged).

Node tokens

# Issue
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/operator/node-token/issue \
  -H 'Content-Type: application/json' \
  -d '{"dc":"local","cluster":"dev","ttl":"30m"}'
# 200: {"token":"hj_v1_...","id":"abcde","expires_at":"..."}

# List (secret stripped)
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  https://127.0.0.1:7999/operator/node-token
# 200: {"tokens":[{"id":"abcde","dc":"local","cluster":"dev","not_after":"..."}]}

# Revoke
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X DELETE https://127.0.0.1:7999/operator/node-token/abcde
# 200: {"status":"ok","id":"abcde"}

Identity

# Register
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/operator/identity \
  -H 'Content-Type: application/json' \
  -d '{
    "spiffe_id":"spiffe://kis.dev/workload/test",
    "selectors":{"binary_path":"/usr/bin/test"},
    "policy":"default"
  }'

# List
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  https://127.0.0.1:7999/operator/identity

# Revoke (note: scheme stripped, leading slash kept by the router)
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X DELETE https://127.0.0.1:7999/operator/identity/kis.dev/workload/test

Audit / Root / Backend / Signer / Policy (all 501 today)

for ep in audit/query audit/verify root/rotate-plan \
          backend/migrate signer/test policy/lint; do
  echo "=== /operator/$ep ==="
  curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
    -X POST https://127.0.0.1:7999/operator/$ep \
    -H 'Content-Type: application/json' -d '{}'
  echo
done

# All return 501 with phase markers. This is expected scaffolding.

4.5 Legacy secret / certificate routes

These predate /workload/* and stay during migration. Verifying they still work guards against regressions.

# Save (POST creates, PUT updates)
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X POST https://127.0.0.1:7999/secret/db_password \
  -H 'Content-Type: application/json' \
  -d '{
    "secret":"hunter2",
    "datacenter":"local","cluster":"dev",
    "customer":"acme","product":"widget","environment":"dev","tenant":"acme-1"
  }'

# Retrieve
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X PATCH https://127.0.0.1:7999/secret/db_password \
  -H 'Content-Type: application/json' \
  -d '{"datacenter":"local","cluster":"dev","customer":"acme","product":"widget","environment":"dev","tenant":"acme-1"}'

# Exists
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X PATCH https://127.0.0.1:7999/checksecret/db_password \
  -H 'Content-Type: application/json' \
  -d '{"datacenter":"local","cluster":"dev","customer":"acme","product":"widget","environment":"dev","tenant":"acme-1"}'

# Delete
curl -sk --cert ./vault-svc.crt --key ./vault-svc.key \
  -X DELETE https://127.0.0.1:7999/secret/db_password \
  -H 'Content-Type: application/json' \
  -d '{"datacenter":"local","cluster":"dev","customer":"acme","product":"widget","environment":"dev","tenant":"acme-1"}'

4.6 Direct agent-socket calls

curl --unix-socket works against the agent the same way it does against the server.

# Health
curl --unix-socket /run/vault/vault.sock http://agent/healthz

# Identity (attested — must be called by a binary that matches a selector)
curl --unix-socket /run/vault/vault.sock http://agent/workload/identity

# Mint a JWT
curl --unix-socket /run/vault/vault.sock \
  -X POST http://agent/workload/jwt \
  -H 'Content-Type: application/json' \
  -d '{"audience":["janus"]}'

# Read a secret (501 today; routes exist)
curl --unix-socket /run/vault/vault.sock \
  -X POST http://agent/workload/secret \
  -H 'Content-Type: application/json' \
  -d '{"name":"vault:db_password"}'

The kis CLI wraps these so you don’t have to remember the socket path:

kis vault status      # GET /healthz + /workload/identity
kis vault jwt --audience janus
kis vault get vault:db_password

5. API collection — single OpenAPI 3.0 spec

The full API surface ships as a single OpenAPI 3.0 spec for the vault service — one file, no directory tree. Importable into Bruno, Insomnia, Postman, and Swagger UI without modification.

5.1 Open it in Bruno

1. Open Bruno → Collections → Import Collection → OpenAPI Specification.

2. Pick the vault service OpenAPI spec file.

3. Bruno generates one folder per tags: group (Health, Workload, Agent handshake, Operator — node tokens, …) with one request per operation.

4. Create an environment in Bruno with these vars (matching the OpenAPI servers: and securitySchemes:):

   Variable	Local dev value	Notes
   baseUrl	https://127.0.0.1:7999	Override per environment
   agentSocket	/run/vault/vault.sock	For agent UDS routes
   operatorCert	~/kis-vault-test/vault-svc.crt	mTLS client cert
   operatorKey	~/kis-vault-test/vault-svc.key	mTLS client key
   caBundle	~/kis-vault-test/test-ca.crt	Server trust anchor
   dc, cluster	local, dev	Sample tenancy

5. In Bruno’s collection-level Auth tab, select mTLS Client Certificate and point at {{operatorCert}} / {{operatorKey}} / {{caBundle}}. Every operation that declares mtlsOperator security inherits this automatically.

5.2 Open it in Insomnia / Postman / Swagger UI

• Insomnia: Create → Import → File → kis-vault.openapi.yaml. Auto-generates a workspace with all routes.

• Postman: Import → File → pick the YAML. Postman converts to its collection format.

• Swagger UI: point a local swagger-ui Docker container at the file:

  docker run --rm -p 8080:8080 \
    -e SWAGGER_JSON=/spec/kis-vault.openapi.yaml \
    -v $PWD:/spec \
    swaggerapi/swagger-ui

  Browse http://localhost:8080. The “Try it out” button works directly against any of the servers: URLs.

5.3 Validate the spec in CI

The spec is the source of truth — keep it in sync with the code.

# Lint
docker run --rm -v $PWD:/specs stoplight/spectral lint \
  /specs/kis-vault.openapi.yaml

# Contract-test using openapi-cli or schemathesis
schemathesis run --checks all kis-vault.openapi.yaml \
  --base-url https://127.0.0.1:7999 \
  --request-tls-verify false

The CI gate runs both on every PR that touches the REST layer or the spec file.

5.4 Variable chaining

The spec includes example bodies for every operation. Bruno’s post-response variable extraction (and Insomnia’s response chaining) lets you wire happy-path flows by hand:

Source operation	Captured field	Used by
POST /operator/node-token/issue	res.body.token → issuedToken	POST /agent/join (in secret field)
POST /agent/join	res.body.refresh_token → refreshToken	POST /agent/renew
POST /operator/identity	res.body.spiffe_id → lastIdentity	DELETE /operator/identity/{spiffe}

In Bruno: open the request → Vars tab → Post-response → issuedToken: res.body.token. The variable persists across requests in the same run.

6. End-to-end verification scenarios

The recipes give you reproducible setups. The scenarios below are flows — sequences that exercise multiple subsystems together.

6.1 Cold-start: server, then agent, then workload

Step	Action	Expected
1	Start vault.svc	/healthz returns 200 in <2s
2	Issue node-token	200 with hj_v1_... body
3	Run vault-agent init with that token	/var/lib/vault/node.crt, node.key, trust_bundle.pem, refresh.dat all created
4	Run vault-agent run	Agent’s /healthz reachable; systemd reports active; sd_notify READY fires
5	Register workload identity for curl’s binary	200
6	curl --unix-socket .../workload/identity	200 with SPIFFE ID

If any step fails, the next ones can’t succeed — diagnose top-down.

6.2 Rotation under load

Step	Action	Expected
1	Set node_cert_lifetime: 5m in vault.svc.yaml so renewal kicks in fast	restart server
2	Run vault-agent for 7 minutes	At ~150s the agent’s stderr logs rotation due then node cert renewed; lifetime: ...
3	While rotating, run a tight loop of kis vault status from another shell	Every call returns 200; no errors during the cert swap

This catches: race conditions in agent identity reload, mTLS-cache misses on the server side.

6.3 Server failure → agent serves cached creds

Step	Action	Expected
1	Bring up §3.2 setup	Healthy
2	Stop vault.svc (pkill vault.svc)	Agent’s /healthz still 200
3	Call /workload/jwt via agent	Either succeeds from cache OR returns a clear “server backend: … not reachable” 5xx — never a hang
4	Wait beyond cache TTL, retry	Clear 5xx
5	Restart vault.svc	Next call succeeds without manual intervention

This catches: the agent must not crash when the server is gone (degraded-mode contract).

6.4 Token replay

Step	Action	Expected
1	Issue node-token	200
2	vault-agent init with it	200
3	vault-agent init AGAIN with the same token	401 with ErrTokenAlreadyUsed shape
4	List tokens via operator endpoint	The consumed token doesn’t appear (or shows uses_left:0)

6.5 Backend swap (live data)

Step	Action	Expected
1	§3.5: Run with fsvault. Write 10 secrets.	All readable.
2	Stop server. Switch config to openbao.	(Backend has no data yet — expected.)
3	Start server, attempt to read a secret	404
4	Migrate the backend contents — see the backend migration guide	Produces a manifest and copies values
5	Verify via direct OpenBao CLI	Same 10 secrets present

The migration command is scaffolded; until it’s implemented, this scenario serves as the acceptance test for that future work.

6.6 Audit chain integrity

Step	Action	Expected
1	Issue 5 node-tokens via the operator endpoint	5×200
2	Query the audit chain (via the audit store directly until /operator/audit/query lands)	5 entries, each chained to the previous via prev_hash
3	Hex-dump one entry’s signature in the audit store’s storage; flip one bit	Tamper
4	Re-run audit verify	Chain breaks at the tampered entry

This catches a class of “silently lost audit” bugs — the chain must refuse to validate if any link is mutated.

6.7 File-mode delivery survives Postgres restart

Step	Action	Expected
1	§3.8 setup	Postgres comes up with cert from /var/lib/postgresql/server.crt
2	Kill Postgres mid-pg_dump	TLS-mid-flight breaks; expected
3	Wait for next agent rotation tick	New cert written atomically
4	Start Postgres again	Postgres reads new cert; serves cleanly
5	Inspect /var/lib/postgresql/server.crt’s notAfter	At least the configured lifetime from now

7. Troubleshooting matrix

Symptom → likely-cause → fastest fix.

Symptom	Likely cause	Fix
connection refused on /healthz	server not listening	lsof -i :7999; check stderr
bad certificate from server	client cert not signed by trusted CA	Re-issue using the test CA, or pass --cacert
Agent socket 401 “workload attestation failed”	no registration entry matches the calling binary’s selectors	POST /operator/identity with the binary’s path / systemd unit
Agent 401 ALWAYS	running on macOS — attest.Attest is Linux-only	Use the Docker setup (§3.3) for attestation testing
socket /run/vault/vault.sock unreachable	agent not running, or /run/vault ownership wrong	systemctl status vault-agent; ls -ld /run/vault
agent /workload/jwt: HTTP 501	(legacy path is 200 — see what changed)	If you hit /workload/secret/get, that’s currently 501 by design
awskms: GetPublicKey ... no route to host	constructor tried to dial AWS; you’re running on a laptop	Use type: inproc or type: pkcs11 for non-cloud tests
Sign: pkcs11: 0x132: CKR_GENERAL_ERROR	SoftHSM key was generated with mismatched mechanism	Re-gen with --key-type RSA:4096
vault-agent init succeeds but vault-agent run exits with node identity not on disk	init wrote to a different state dir	Both commands must agree on state_dir; default is /var/lib/vault
Server log spam: dead JWT cache for vault:NAME	the cache layer logs every cache miss in info	Lower log level or ignore — it’s not an error
Comparing transports	Run both side by side	Two terminals, same request, compare the responses

8. Capacity / soak / chaos

For DevSecOps signing off a release. Skip if you’re a customer dev — these are infra-validation tests.

8.1 Issuance throughput

# 1000 sequential JWT mints over the agent socket
time for i in $(seq 1 1000); do
  curl -s --unix-socket /run/vault/vault.sock -X POST \
    http://agent/workload/jwt \
    -H 'Content-Type: application/json' \
    -d '{"audience":["bench"]}' >/dev/null
done

Expected (laptop, inproc signer): ~30–50ms p50 per call, no errors. Expected (PKCS#11/SoftHSM): ~80–120ms p50 — SoftHSM is software, but the protocol overhead is real.

8.2 Soak

Leave the §3.10 HA setup running for 24 hours under a steady 10 rps. Verify:

• No memory growth over 100 MB per process
• No goroutine leak (go tool pprof http://vault.svc:7999/debug/pprof/goroutine if pprof is enabled)
• Rotation happens at the configured interval; no missed ticks
• Audit chain has the expected event count (issuance × 10 rps × 86400 s = ~864k entries)

8.3 Chaos

• Kill -9 vault.svc at random times. Agent should never crash.
• Drop the backend (docker stop openbao). Server should return 5xx from /secret/* with a clear message; /healthz itself should report degraded.
• Clock skew: skew one host’s clock by +5 minutes. Cert validity windows should still accept the leaf within their not_before grace.
• OOM the server: stress-ng --vm 4 --vm-bytes 80%. Server should restart cleanly (systemd Restart=always) and re-load identity material without re-bootstrapping.

Appendix A — One-shot setup script

#!/usr/bin/env bash
# Saves to ~/kis-vault-test and starts §3.2 (server + agent on same host).
set -euo pipefail

cd "$(mkdir -p ~/kis-vault-test && cd ~/kis-vault-test && pwd)"

[[ -f test-ca.crt ]] || {
  openssl req -x509 -newkey rsa:2048 -nodes -keyout test-ca.key -out test-ca.crt \
    -days 30 -subj "/CN=kis-test-ca"
  openssl req -newkey rsa:2048 -nodes -keyout vault-svc.key -out vault-svc.csr \
    -subj "/CN=vault.svc.local"
  openssl x509 -req -in vault-svc.csr -CA test-ca.crt -CAkey test-ca.key -CAcreateserial \
    -out vault-svc.crt -days 30 \
    -extfile <(printf "subjectAltName=DNS:vault.svc.local,DNS:localhost,IP:127.0.0.1")
  openssl genrsa -out vault-ca.key 4096
  openssl req -x509 -new -key vault-ca.key -out vault-ca.crt -days 3650 \
    -subj "/CN=kis Vault Root CA"
}

cat > vault.svc.yaml <<EOF
service: { name: vault.svc, http: { listenaddress: 127.0.0.1:7999 } }
vaultmanager:
  store: { fsvault: { filepath: ./fsvault-data/secrets.yaml, env: dev } }
vault:
  signer: { type: inproc, inproc: { ca_cert: ./vault-ca.crt, ca_key: ./vault-ca.key } }
tls:
  ca_certs: ["./test-ca.crt"]
  cert:    "./vault-svc.crt"
  key:     "./vault-svc.key"
  require_client_cert: true
EOF

mkdir -p fsvault-data /run/vault /var/lib/vault
echo "Server config at $PWD/vault.svc.yaml"
echo "Run:    vault.svc -config ./vault.svc.yaml"
echo "Then:   see §3.2 for agent bootstrap"

Appendix B — Bruno collection bootstrap

mkdir -p bruno/kis-vault/{environments,Health,Workload,Agent\ handshake,Operator,Legacy\ secret}

cat > bruno/kis-vault/bruno.json <<'EOF'
{"version":"1","name":"kis Vault","type":"collection","ignore":["node_modules",".git"]}
EOF

# environments/local-dev.bru and per-folder .bru files — copy from §5.3-5.4

After that, open Bruno → Collection → Open Collection → point at bruno/kis-vault/.

Done. If a request in this doc doesn’t behave as described, file an issue with the recipe number (§3.x) and the response body — that’s enough context to reproduce locally.