Deploy: High Availability

Audience: DevSecOps — engineers running multiple vault.svc instances behind a load balancer for fault tolerance. You are either a customer employee operating Vault for your company, or a kis.ai employee operating it on the customer’s behalf. Either way, this doc assumes you do not have the Vault source code — you have release binaries, config files, and your cluster’s load balancer + storage-backend infrastructure to wire together.

Appropriate for any deployment where a single host failure must not interrupt the trust plane.

For single-instance deployments, see single-node deployment. For the per-node agent, see the Vault agent.

1. Topology

                    ┌──────────────────┐
                    │  Load balancer   │
                    │   (round-robin)  │
                    └────────┬─────────┘
                             │ HTTPS :7999
        ┌────────────────────┼────────────────────┐
        ▼                    ▼                    ▼
  ┌───────────┐        ┌───────────┐        ┌───────────┐
  │  horus-1  │        │  horus-2  │        │  horus-3  │   ← N horus-server
  │           │        │           │        │           │     instances
  └───────────┘        └───────────┘        └───────────┘
        │                    │                    │
        └────────────────────┼────────────────────┘
                             ▼
        ┌────────────────────────────────────────┐
        │   Storage backend (HCP / OpenBao /     │
        │   Azure KV / AWS SM)                   │
        │   — provides HA + consistency itself.  │
        └────────────────────────────────────────┘
                             │
                             ▼
        ┌────────────────────────────────────────┐
        │   Data API (entity store + audit chain)  │
        │   — separate HA story; see the Data API SOP. │
        └────────────────────────────────────────┘

Stateless instances. vault.svc is stateless beyond its config file and TLS material. All persistent state lives in the backend (HCP/OpenBao/Azure KV/AWS SM) and Seshat. Losing all N instances does not lose data; bring up new ones pointed at the same backend.

No internal leader election. Unlike config.svc, the Vault service has no reconciler that needs serialization. Every instance handles every operation independently against the shared backend. The backend’s own consistency model (HCP/OpenBao Raft, Azure KV strong consistency, AWS SM eventual within a region) is the serialization point.

Read scaling. The LB load-balances all reads across instances. Cert verification (consumer-side JWT validation) is the highest-QPS path; it hits one instance per chassis-middleware call.

Write serialization. Writes (PUT/POST/DELETE) go through whichever instance the LB hands them to. The backend serializes per key. For HCP/OpenBao with KV-v2 versioning, concurrent writes to the same key produce monotonic versions; no vault.svc-side conflict logic is needed.

2. Prerequisites

Requirement	Detail
Backend cluster	A production-grade backend deployment. HCP/OpenBao: 3 or 5 Raft nodes. Azure KV / AWS SM: regional service, multi-AZ by default.
Backend TLS / IAM	Cluster-grade auth: cert auth for HCP/OpenBao, managed identity for Azure, IRSA / instance profile for AWS.
Load balancer	TCP or HTTP/L7. Idle timeout ≥ 600 s for long-lived consumer streams (gRPC / SSE).
N ≥ 2 instances	One per AZ minimum. Stateless beyond the backend connection.
Time	NTP-synced. JWT validation and cert expiry depend on monotonic clocks.
TLS material	Per-instance server cert (matching the LB-facing DNS or the per-instance DNS, depending on LB mode). Common CA for client validation.
Seshat	Same Data API cluster across all vault.svc instances (vault.svc writes audit + entity rows; Seshat is the single source of truth for those).

3. Backend choice

Backend	When
HCP Vault	Default for on-prem / multi-cloud. Mature, well-understood, but BUSL-licensed.
OpenBao	Same wire protocol; Apache-2.0. Pick when license is a blocker.
Azure KV	AKS-hosted / Azure-native deployments. Phase-4.
AWS SM	EKS-hosted / AWS-native deployments. Phase-4.
FsVault	Never for HA. Refuses to start in prod regardless.

For HCP/OpenBao, the backend cluster’s HA is separate from the Vault service’s HA — bring up the Raft cluster first per the backend’s HA documentation, then point all vault.svc instances at it.

4. Bootstrap YAML — HA mode

Same shape as single-node (§3 of single-node deployment), with the following adjustments:

service: horus
version: 1.0.0
# sid: MUST be unique per instance. Used by audit, OTel resource
# attributes, and any future per-instance bookkeeping.
sid: ${HOSTNAME}

host: ""
port: "7999"
zerotrust: true

vaultmanager:
  store:
    hcpvault:
      url: https://vault.internal.example.com:8200
      certpath:   /etc/horus/certs/vault-client.crt
      keypath:    /etc/horus/certs/vault-client.key
      cacertpath: /etc/horus/certs/issuer.crt
      authname: horus-cert
      secretspathprefix: secret
      certificatespathprefix: kisaicerts
      retryduration: 2s

horus:
  signer:
    # In HA, ALL instances must use the same signer custody.
    # inproc is fine ONLY if every instance reads the same key from
    # the same trusted source (e.g. config-as-code with secret
    # mounts). Once you move to pkcs11 / awskms / azurekv-keys /
    # gcpkms, ALL instances point at the same HSM/KMS key.
    type: inproc

vault:
  client:
    transport: remote   # consumers use the LB-facing URL

accessrules: {}

audit:
  endpoint: https://chitragupta.internal:8443/submit
  timeout:  5s

telemetry:
  collector:   otel-collector.internal:4317
  insecure:    false
  environment: prod
  traces: { enabled: true, sampling_rate: 0.1 }

Each instance has the same YAML except sid: (unique per instance, typically ${HOSTNAME}).

5. Deployment

5.1 Bring instances up

Start instances one at a time. Each comes up independently against the shared backend; no coordination between instances.

for host in horus-1 horus-2 horus-3; do
    ssh "$host" 'sudo systemctl start horus'
    until curl -sf "https://${host}.internal:7999/ready" > /dev/null; do
        sleep 1
    done
done

5.2 LB configuration

Health probe: GET /ready. Out-of-rotation on 503.
Idle timeout: ≥ 600 s for long-lived consumer streams.
Backend pool: every instance. Reads are equivalent across instances.
TCP or HTTP/L7 — both work; L7 enables future path-based routing (e.g. split audit query traffic off to a dedicated pool).
Sticky sessions NOT required.

5.3 First-boot verification

# Through the LB
curl -s "https://horus.example.com:7999/version"
# {"version":"1.0.0", ...}

# mTLS health check from outside
curl -s --cert client.crt --key client.key --cacert ca.crt \
    "https://horus.example.com:7999/certificate/internal/issuer"

Per-instance:

for host in horus-1 horus-2 horus-3; do
    echo "--- $host ---"
    ssh "$host" "sudo journalctl -u horus -n 5 --no-pager"
done

Each instance should show the successfully connected to vault backend hcpvault log line shortly after start.

6. Day-2 operations

6.1 Adding an instance

Provision the new host (§2 of single-node deployment).
Copy the bootstrap YAML and TLS material from an existing instance. Change sid: to the new host’s identifier.
Start the service.
Add to the LB backend pool.
Verify /ready returns 200 from the new host.

No backend or Seshat reconfiguration; the new instance just joins the read/write pool.

6.2 Removing an instance

Drain from LB (stop accepting new connections).
Wait ~30 s for in-flight connections to drain.
Stop the service.

Stateless — nothing to migrate off.

6.3 Rolling upgrade

for host in horus-1 horus-2 horus-3; do
    ssh "$host" '
        sudo systemctl stop horus &&
        sudo tar -C /opt/horus -xzf /tmp/horus-${NEW_VERSION}.tar.gz &&
        sudo systemctl start horus
    '
    until curl -sf "https://${host}.internal:7999/ready" > /dev/null; do
        sleep 1
    done
done

The cluster remains fully available throughout; reads continue from the other N-1 instances. Total LB pool reduction: one instance at a time.

6.4 Backend maintenance

Backend operations (HCP/OpenBao Raft snapshot, key rotation, member churn) are independent of the Vault service. vault.svc reconnects automatically on backend churn (the cluster behind the URL just changes; the URL doesn’t).

6.5 Rotating the CA / Signer key (phase-5+)

The “root rotation ceremony” is privileged. Cross-sign the new root with the old (or vice versa) and gradually re-issue intermediates over a transition window. Procedure documented in the threat model, §6.

7. Monitoring

vault.svc exposes OTel signals (traces + metrics) when telemetry.collector is set. Required metrics for HA monitoring:

Metric	Purpose
`horus_backend_latency_seconds`	P99 should stay < 200 ms under normal load.
`horus_backend_failures_total`	Spike = backend health issue.
`horus_signer_latency_seconds`	P99 < 500 ms for inproc; < 1 s for KMS / PKCS#11.
`horus_jwt_issued_total`	JWT issuance QPS per audience.
`horus_cert_issued_total`	Cert issuance QPS per realm.
`horus_audit_emit_failures_total`	Audit write failures. MUST be zero in steady state.
`horus_policy_deny_total{rule}`	Per-rule deny rate. Spike from one workload = probe / misconfiguration.
`horus_instances_healthy`	Derived from LB health probes — N_expected vs N_responding.

Alert thresholds (starting points; tune per site):

Backend latency P99 > 1 s sustained 10 m → page.
Backend failures > 0 in last 5 m → page.
Audit emit failures > 0 in last 5 m → page (chain durability at risk).
Policy deny rate from any single workload > 100 /min → page (compromise probe).
N_healthy < N_expected for > 5 m → page.

8. Backup & disaster recovery

8.1 What to back up

Backend (HCP/OpenBao/Azure KV/AWS SM): the backend operator’s responsibility. Confirm backup is configured (HCP Raft snapshot schedule, Azure KV soft-delete + recovery window, AWS SM resource policies + cross-region replication).

Seshat (entity + audit): Seshat operator’s responsibility. The audit chain is mirrored to Chitragupta (audit.svc) for durability; the Data API copy is the queryable index.

vault.svc instances: stateless. No backup needed beyond the bootstrap YAML + TLS material in config-as-code.

8.2 Loss of all instances

Bring up new ones pointed at the same backend + Seshat. No data recovery needed.

8.3 Loss of backend

Restore from backend backup. While the backend is down:

Reads from already-cached cert public keys (chassis middleware) continue.
Cert issuance / secret reads / writes all fail.
The audit chain head publication continues (Chitragupta), so external observers can verify pre-incident state is intact.

8.4 Loss of Seshat

The audit chain is partially in Chitragupta already (mirrored on every write). Restore Seshat from its backup; vault.svc has no special role.

8.5 Loss of the CA private key (Signer key)

Catastrophic. Procedure: declare a security incident, rotate the root CA via cross-signing (see the threat model, §6), re-issue every intermediate and leaf cert in the cluster, communicate to all consumers. This is the worst case the threat model is designed against.

9. Troubleshooting

Symptom	Likely cause	Action
One instance lags in `/ready`	Local backend connection issue on that instance.	`journalctl -u horus -n 500` on the lagging host. Restart.
All instances 503 simultaneously	Backend unreachable from this network.	Check backend cluster health, network policy, DNS resolution from horus subnet.
Writes succeed on one instance, reads from another don’t see them immediately	Backend eventual consistency (rare for HCP; possible for AWS SM cross-AZ reads within < 100 ms).	Verify the backend’s consistency model. If unacceptable, wait or switch to a strongly-consistent backend.
Audit chain heads drift between Chitragupta and Seshat	Audit-emit failures on one instance.	`journalctl -u horus
LB reports an instance unhealthy intermittently	Slow backend response causing `/ready` to time out.	Check `horus_backend_latency_seconds` on that instance vs others. May indicate a single-AZ backend issue.
All instances issue different JWT signing keys	Each instance’s `inproc` Signer loaded a different key file.	Use a shared Signer (`pkcs11` / `awskms` / `azurekv-keys` / `gcpkms`) — they all point at the SAME backend key, so all instances sign identically.