Deploy: vault-agent

Audience: DevSecOps — engineers installing vault-agent on every node that runs kis.ai workloads. You are either a customer employee deploying agents across the customer’s fleet, or a kis.ai employee doing the same on the customer’s behalf. Either way, this doc assumes you do not have the Vault source code — you have a per-cluster release binary that embeds the cluster’s root CA at build time, plus a join token issued by your operator.

The agent is the local representative of the trust plane on each box — workloads talk to it over a Unix socket instead of HTTPS to vault.svc.

For the server side, see the single-node deployment guide and the HA deployment guide. The agent’s internal design and its role in the overall architecture are documented separately.

1. Why deploy the agent

Today’s consumers reach the Vault service over HTTPS via lib-vault.RemoteVaultClient. That works, but: every JWT verification, every secret fetch, every chassis-middleware call is a network round-trip to vault.svc. vault.svc is the SPOF.

With the agent on each node:

Every workload’s calls go to the localhost socket — microsecond latency, no network failure mode.
The agent caches credentials per workload, rotates them proactively, and re-fetches when the workload reconnects.
vault.svc becomes a regional / cluster-level authority; per-node load drops by orders of magnitude.

The agent is opt-in per cluster. Until vault.client.transport: agent is flipped in the cluster config, consumers stay on remote.

2. Prerequisites

Requirement	Detail
OS	Linux x86_64 / arm64. The agent does NOT target macOS / Windows; local dev uses `one.svc` or the `remote` transport.
Init system	systemd. The unit and `sd_notify` integration depend on it.
Privileged user	The installer runs as root once; the daemon then drops to the `horus` system user (UID/GID auto-assigned).
Filesystem	`/etc/horus/`, `/var/lib/horus/`, `/run/horus/` (tmpfs, created by systemd).
Network	Outbound mTLS to vault.svc (default `:7999`). No inbound listening port — the socket is local-only.
Time	NTP-synced. The agent renews node certs before they expire (default lifetime 24h, rotates at T-6h); large clock skew breaks the renewal window.
Binary	A per-cluster build — the agent embeds the cluster’s root CA at build time. A `kis-prod` binary will refuse to init against a `kis-staging` server, by design.

3. Install

The agent is a single static binary. The install subcommand handles the one-time per-node setup:

# 1. Download the cluster-specific binary.
curl -fsSL -o /tmp/vault-agent \
    "https://harbor.${CLUSTER}.example.com/horus/${VERSION}/vault-agent-linux-amd64"
sudo install -m 0755 /tmp/vault-agent /usr/local/bin/vault-agent

# 2. Verify the binary embeds the expected cluster identity.
vault-agent version --verbose
# Cluster:      kis-prod
# Trust domain: kis.ai
# Embedded root SHA-256: 8f3a2c4d6e1f...

# 3. Run the installer (idempotent).
sudo vault-agent install

The installer creates the horus system user, the /etc/horus and /var/lib/horus directories, the systemd unit, and writes a default /etc/horus/agent.yaml. It does NOT start the service — init does.

3.1 What `install` does

[install] Cluster:            kis-prod
[install] Trust domain:       kis.ai
[install] Embedded root sha:  8f3a2c4d6e1f...
[install] Binary build time:  2026-05-15T10:23:00Z

[install] Create system user 'horus'... ok
[install] Create /etc/horus... ok
[install] Create /var/lib/horus... ok
[install] Install binary to /opt/kis/bin... ok
[install] Install systemd unit... ok
[install] Write default agent.yaml... ok
[install] systemctl daemon-reload... ok

✓ Horus Agent installed but not initialized.

Next step:
  sudo vault-agent init <vault-server-address> <secret-key>

The cluster ID + root SHA printed at the top let you catch “wrong binary” mistakes before init.

4. Initialize (join the cluster)

The agent joins the cluster by presenting a single-use join token to vault.svc. The operator mints the token via the CLI:

# From an operator workstation:
kis vault node-token issue \
    --scope dc=us-west,cluster=prod \
    --ttl 1h \
    --uses 1
# → hj_v1_n8x2k_a1b2c3d4e5f6g7h8j9k0

Hand the token to the operator on the new node:

sudo vault-agent init \
    horus.prod.example.com:7999 \
    hj_v1_n8x2k_a1b2c3d4e5f6g7h8j9k0

Or, for automation:

echo "$TOKEN" | sudo vault-agent init horus.prod.example.com:7999 --token-stdin
# or:
sudo vault-agent init horus.prod.example.com:7999 --token-from-file /run/secrets/token

init does the following, in order:

TLS-verifies the server using only the embedded root (no system trust store).
Sends JoinRequest{secret, node_attributes} over gRPC.
Receives JoinResponse with node SPIFFE ID, cert, key, trust bundle, and refresh token.
Writes the identity material atomically under /var/lib/horus/.
Updates /etc/horus/agent.yaml with the server address.
Deletes the token file (if --token-from-file was used).
systemctl enable --now vault-agent.
Waits for sd_notify(READY=1).

Verify:

vault-agent status

You should see the node SPIFFE ID, cert expiry, and “Server connected”.

5. Steady-state behaviour

The daemon (vault-agent run, invoked by systemd) does:

Maintains an mTLS connection to vault.svc using the node cert.
Renews the node cert at T-6h (configurable via behavior.cert_renewal_threshold).
Refreshes the trust bundle hourly (or on server push).
Serves the local socket /run/vault/vault.sock (kis-native gRPC).
Optionally serves /run/horus/spiffe.sock (SPIFFE Workload API) if listeners.spiffe.enabled: true.
For each declared file-mode delivery, fetches the cert/key/bundle and writes them atomically to disk; triggers reload on rotation.
Emits metrics on 127.0.0.1:9099/metrics.
Logs structured events to journald.

6. File-mode delivery (Postgres, ClickHouse, etc.)

For workloads that can’t speak the socket protocol, the agent writes cert files to disk and triggers a reload. Configure in /etc/horus/agent.yaml:

file_deliveries:
  - name: postgres-prod
    identity: spiffe://kis.ai/customer/acme/product/postgres/env/prod
    paths:
      cert:         /var/lib/postgresql/tls/server.crt
      key:          /var/lib/postgresql/tls/server.key
      trust_bundle: /var/lib/postgresql/tls/ca.crt
    permissions:
      owner: postgres
      group: postgres
      mode_cert:   0644
      mode_key:    0400
      mode_bundle: 0644
    reload:
      type:     signal
      signal:   SIGHUP
      pid_file: /var/run/postgresql/16-main.pid
      # Alternative (mutually exclusive):
      # type:    exec
      # command: [/usr/bin/systemctl, reload, postgresql.service]

SIGHUP after writing all three files. The agent must be able to chown to the target user (postgres), which means either:

(a) Add horus to the postgres group and chmod the target dir group-writable. Preferred — keeps the agent’s capabilities narrow.

(b) Grant the agent CAP_CHOWN + CAP_FOWNER. Wider blast radius; do this only if (a) is infeasible.

After editing agent.yaml:

sudo systemctl reload vault-agent   # SIGHUP — picks up file_deliveries changes

Within one rotation cycle (default: half the cert lifetime), files appear under the configured paths. Verify:

sudo ls -la /var/lib/postgresql/tls/
sudo openssl x509 -in /var/lib/postgresql/tls/server.crt -noout -dates

7. Day-2 operations

7.1 Inspect status

vault-agent status              # human-readable
vault-agent status --json       # machine-readable

Output covers:

Agent version + cluster ID + trust domain
Node SPIFFE ID + cert expiry
Server address + connection state
Workloads attested in the last minute
File-mode deliveries: identity + next rotation

7.2 Diagnose

sudo vault-agent doctor

Runs a battery of checks (binary integrity, system user, config valid, state-dir perms, systemd unit installed + enabled, agent running, server reachable, workload sockets listening, file deliveries up-to-date). Exit 0 if all-ok-or-warnings; 1 on failure.

7.3 Logs

journalctl -u vault-agent -f

Structured JSON with these event kinds:

node.attested — initial join
node.cert.rotated — periodic cert refresh
workload.attested — a workload connected and was issued credentials
workload.cert.rotated — periodic per-workload rotation
server.connect / server.disconnect
error.* — failures

The agent never logs cert bodies, private keys, or JWT signature bytes. Cert serials, SPIFFE IDs, and audit-relevant identifiers are at info level.

7.4 Reload config without restart

sudo systemctl reload vault-agent   # SIGHUP

Reloadable fields: log.*, file_deliveries, behavior.*, metrics.*, telemetry.*. Changes to immutable fields (socket paths, trust domain, server address) require a service restart.

8. Upgrade

In-place binary swap:

sudo vault-agent upgrade

Implementation: downloads the new binary from the cluster’s distribution endpoint, verifies the lib-sign signature, replaces the on-disk binary atomically, restarts the systemd unit. The new binary MUST embed the same cluster ID and root SHA — refuses cross-cluster upgrades.

Manual path:

sudo curl -fsSL -o /tmp/vault-agent \
    "https://harbor.${CLUSTER}.example.com/horus/${NEW_VERSION}/vault-agent-linux-amd64"
sudo systemctl stop vault-agent
sudo install -m 0755 /tmp/vault-agent /opt/kis/bin/vault-agent
sudo systemctl start vault-agent
vault-agent status   # confirm it's back

Per-node upgrade is non-disruptive at the cluster level — other nodes’ agents remain up. The workloads on the upgraded node re-attest on the agent’s restart; total local outage is ~5 s.

9. Decommission

sudo vault-agent uninstall

Stops the service, notifies vault.svc (best-effort) that this node is decommissioning, removes the systemd unit, /etc/horus, /var/lib/horus, the binary, and the horus user.

The server marks the node SPIFFE ID as revoked. Any workload cert previously issued by this node remains valid until natural expiry — short TTLs are the mitigation.

Flags:

--keep-data: preserves /var/lib/horus/ for a re-install.
--offline: skip the server-side decommission notification (use when the server is unreachable).
--yes: skip the confirmation prompt.

10. Troubleshooting

Symptom	Likely cause	Action
`install` fails: `must run as root`	Not running with sudo.	Re-run with sudo.
`install` fails: `requires Linux with systemd`	Wrong OS or non-systemd init.	The agent is Linux+systemd only by design.
`init` fails: “server cert does not chain to embedded root”	Wrong binary for this cluster, OR server cert reissued under a different root.	Run `vault-agent version --verbose`; compare embedded root SHA to the cluster’s known root SHA. If mismatched, get the right binary.
`init` fails: “token already consumed”	The join token was used once; tokens are single-use.	Mint a new token (`kis vault node-token issue`).
`init` fails: “scope mismatch”	Token was minted with a scope that excludes this node’s attributes.	Mint a token with the right `--scope` for this dc/cluster.
`status` shows “Server: disconnected”	Network issue, server cert expired, or node cert expired without renewal.	Check `journalctl -u vault-agent`. If node cert is expired, the agent enters degraded mode and serves cached credentials only; re-init with `--force` to recover.
Workload connects but gets “no matching registration entry”	Workload’s selectors (binary path, systemd unit, cgroup) don’t match any registered identity.	Server-side: `kis vault identity list` — verify a matching entry exists. Add one if not (`kis vault identity register`).
File-mode delivery: cert file appears but workload doesn’t reload	Reload command wrong.	Check `signal:` matches what the workload expects (Postgres: SIGHUP). For `type: exec`, run the command manually to verify it works.
`journalctl` floods with `attestation_failed`	Some local process is hitting the socket with no matching registration.	Identify via PID in the log line; either register it or ACL the socket.
Agent process flapping (frequent restarts)	Persistent identity-refresh failure.	`journalctl -u vault-agent --since "10 min ago"`. If refresh fails repeatedly, the agent exits non-zero and systemd restarts. Re-init with `--force`.