Operations

Bootstrap from zero

1. Install — kvm install vault.svc (or build from source; the release pipeline stamps each cluster’s root CA into the vault-agent binary — dev builds carry a placeholder root and refuse sensitive operations).

2. Generate config:

vault.svc config generate bootstrap > /etc/vault/vault.yaml     # production
vault.svc config generate bootstrap-dev > /etc/vault/vault.yaml # dev (fsvault)

3. Initialize the storage backend (HCP Vault / OpenBao), one time:

   • operator init + unseal the backend
   • auth enable cert and register the vault.svc client certificate role
   • apply the access policy and cert-auth role definitions (the .hcl files ship with the service)
   • secrets enable -path=secret kv-v2 and secrets enable -path=kisaicerts kv-v2

4. Root of trust — generate the CA with the built-in certificate subcommands (root CA → intermediate → service certs). There is no unseal ceremony for vault.svc itself: protect the CA key material (or use an HSM/KMS signer and it never touches disk).

5. Run — on start the service loads the root issuer, connects the backend, initializes signer → envelope → audit → revocation, and idempotently generates its own JWT signing certificate once backend connectivity is confirmed.

vault.svc server -f /etc/vault/vault.yaml
curl -k https://localhost:7999/ready

There are no hand-run SQL migrations — the secrets/certificates index schema is embedded and applied at boot.

Local dev, single process:

vault.svc server -f .vault.svc.yaml --port 9443 --datacenter dev --cluster local --sid vault-local --mode root

Enrolling a node (vault-agent)

# operator: mint a single-use join token, scope-bound to the cluster
kis vault node-token issue --scope dc=dc1,cluster=main --ttl 1h --uses 1
# → hj_v1_<id>_<random>

# on the node:
vault-agent install
vault-agent init          # consumes the join token → node identity + 24h cert
vault-agent run
vault-agent status        # or: doctor, upgrade, uninstall

The agent renews its node certificate at half-life; the refresh token rotates on every renewal (default 30-day refresh window). Register workloads so the agent can attest them:

kis vault identity register   # binds a SPIFFE ID to selectors (binary path/hash, uid, unit…)
kis vault identity list
kis vault identity revoke

CLI

Developer commands go through the local agent, which attests the calling shell — see Vault CLI for the full list: kis vault get|put|exists|delete <name>, kis vault jwt --audience <aud>, kis vault status.

Operator commands use an mTLS operator cert (--operator-cert/--operator-key, the kis context, or KIS_VAULT_OPERATOR_{CERT,KEY}): node-token, identity, audit query|verify, root rotate-plan|rotate-execute, backend migrate, signer test, policy lint, migrate-service <name> (prints the per-service runbook for switching a consumer to the agent transport). root rotate-*, backend migrate, signer test, and policy lint currently hit server-side scaffolds (501).

Rotation

Material	Policy
Secrets	Write a new record (kis vault put); previous becomes inactive; max 3 active per key
Service certificates	90-day default validity; use the days filter on the list endpoints to find expiring certs
Node certificates	24h, auto-renewed by the agent at half-life
Refresh tokens	Rotated on every renewal
Root CA	A planned ceremony: always run root rotate-plan (dry run) before root rotate-execute

Revocation

Revoking an internal cert (DELETE /certificate/internal) records the serial in the backend-persisted revocation list and emits a cert.revoke audit event. The CRL is re-signed with 24h validity and served at GET /pki/crl; OCSP at POST /pki/ocsp (1h response validity). Alert if the log shows the revocation list read as empty — that means clients see revoked certs as good.

Audit

Vault is a client of the platform audit service (a tamper-evident Merkle-log store). Configure audit.endpoint; without it a no-op client logs locally and only the mutable legacy audit entity records secret operations.

• Log-ID convention: vault/<dc>/<cluster>/<tenant>, …/_cluster, vault/_global.
• Event types: agent.join|renew|decommission, secret.read|write|delete|exists, cert.issue|retrieve|revoke, jwt.issue|verify, policy.grant|deny, operator.
• Emission is best-effort and non-blocking (~7s retry budget). Alarm on the audit-emit-failure metric being non-zero — sustained failure means the tamper-evident chain has gaps and should be treated as a security incident.
• Read back through kis vault audit query / audit verify.

Boot ordering note: the audit service fetches its tree-head signing key from Vault — Vault starts without audit and begins emitting once the audit service is up.

Failure modes

Symptom	Cause / action
/ready returns 503	Storage backend unreachable
Refuses to boot with fsvault	env: prod with the file backend — use a real backend
vault-not-connected / vault-failed-to-retrieve-secret	Backend down. Consumers have no secret cache — expect immediate impact
Backend token not renewable	Missing policy on the backend side
Envelope-encrypted secret read fails with an explicit refusal	The instance reading it has no envelope key configured
Audit emit failures	Operations continue, but chain durability is broken — page on it
TLS chain mismatch	Client/CA certs out of sync between vault.svc and the backend

Metrics: OpenTelemetry, scrape 127.0.0.1:9098/metrics (server) and 127.0.0.1:9099 (agent).