Deploy: Single-Node

Audience: DevSecOps: engineers deploying vault.svc (the Vault service) as a single instance backed by HCP Vault (or OpenBao). You are either a customer employee operating a Vault for your company, or a kis.ai employee operating one on the customer’s behalf. Either way, this guide assumes you do not have the Vault source code, you have a release tarball, a binary, and a config file to write. Everything here can be done with what’s in the tarball.

Appropriate for: smaller production deployments, customer-prem single-site installations, and dev / staging environments where the trust-plane availability requirement is lower than the consuming platform.

For HA deployments with multiple vault.svc instances, see the HA deployment guide. For deploying vault-agent on each consumer node, see the agent guide.

1. Prerequisites

Requirement	Detail
OS	Linux x86_64 / arm64. macOS supported for dev only.
Privileged user	Service runs as `vault`. Do not run as root.
Backend storage	A running HCP Vault, OpenBao, Azure Key Vault, or AWS Secrets Manager (HCP/OpenBao are the only ones supported end-to-end in v1; Azure/AWS are scaffolded but phase-4).
Filesystem	A persistent local directory for state and cert material (default `/var/lib/vault`, `/etc/vault`).
Network	One TCP port (default `:7999`). Outbound to the backend (HCP: 8200; etc.). Outbound HTTPS to Chitragupta (`audit.svc`, typically `:8443`) for audit submission.
TLS material	mTLS required. PEM-format server cert/key, the CA bundle that signs your clients, and (for HCP/OpenBao) a separate client cert for `vault.svc` → backend Vault auth.
Time	NTP-synced. JWT `nbf/exp` and cert validity windows assume reasonable wall-clock accuracy.

2. Install

2.1 From a release tarball

sudo install -d -o root -g root /opt/vault
sudo tar -C /opt/vault -xzf vault.svc-${VERSION}-linux-amd64.tar.gz
sudo install -d -o root -g vault -m 0750 /etc/vault
sudo install -d -o vault -g vault -m 0700 /var/lib/vault
sudo install -d -o root -g vault -m 0750 /etc/vault/certs

2.2 systemd unit

[Unit]
Description=vault.svc trust plane (kis Vault server)
After=network.target

[Service]
Type=simple
User=vault
Group=vault
ExecStart=/opt/vault/vault.svc server -f /etc/vault/vault.yaml
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

# Hardening — single-node, no shared FS:
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/vault /var/log/vault
NoNewPrivileges=true
PrivateTmp=true
LockPersonality=true
RestrictSUIDSGID=true

# Resource limits.
MemoryMax=1G
TasksMax=256

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now vault.svc

3. Bootstrap YAML: single-node

Generate a template:

sudo /opt/vault/vault.svc config generate bootstrap > /etc/vault/vault.yaml
# Or pick a specific shape:
sudo /opt/vault/vault.svc config list

Edit /etc/vault/vault.yaml. Production-style example using HCP Vault (works against OpenBao identically, swap hcpvault → openbao and the cert paths):

# Service identity — chassis convention.
service: vault
version: 1.0.0
sid:     ${HOSTNAME}

# Listener.
host: ""
port: "7999"
zerotrust: true

# Storage backend — HCP Vault (KV-v2) with cert auth.
vaultmanager:
  store:
    hcpvault:
      url: https://vault.internal.example.com:8200
      certpath:   /etc/vault/certs/vault-client.crt
      keypath:    /etc/vault/certs/vault-client.key
      cacertpath: /etc/vault/certs/issuer.crt
      authname: vault-cert
      secretspathprefix: secret
      certificatespathprefix: kisaicerts
      retryduration: 2s

# JWT defaults.
servicejwt:
  expiry: 1h
  httponly: true

vault:
  # Signing custody — start with inproc; upgrade to pkcs11 / awskms once
  # the HSM/KMS plan is in place. See the signer-*.yaml examples.
  signer:
    type: inproc

  # Default transport for the chassis lib-vault clients in this cluster.
  # Keep "remote" until every consumer node has vault-agent installed
  # and validated — flipping to "agent" before that breaks consumers.
  client:
    transport: remote

# Tenant-aware access rules. See accessrules.yaml example for the
# canonical shape, including the mandatory tenant check.
accessrules: {}

# Audit emission. Required in prod; the default NoOpEmitter is dev-only.
audit:
  endpoint: https://chitragupta.internal:8443/submit
  timeout:  5s

# OTLP telemetry.
telemetry:
  collector:   otel-collector.internal:4317
  insecure:    false
  environment: prod
  traces: { enabled: true, sampling_rate: 0.1 }

3.1 Templates available

vault.svc config list                      # list all
vault.svc config generate bootstrap        # HCP + inproc + JS rules
vault.svc config generate bootstrap-dev    # FsVault (dev only)
vault.svc config generate backend-openbao  # OpenBao snippet
vault.svc config generate backend-azurekv  # Azure KV snippet (phase-4)
vault.svc config generate backend-awssm    # AWS SM snippet (phase-4)
vault.svc config generate signer-pkcs11    # HSM signer (phase-5)
vault.svc config generate signer-awskms    # AWS KMS signer (phase-5)
vault.svc config generate accessrules      # canonical JS rule sample
vault.svc config generate agent            # reference per-node agent config

4. Backend setup (HCP Vault / OpenBao)

A one-time bootstrap of the backend, per the release documentation. Summary:

vault operator init (or bao operator init), unseal with the issued keys, store the root token offline.
vault auth enable cert: turn on certificate auth.
Write the kisai-policy granting access to secret/* and kisaicerts/*.
vault write auth/cert/certs/vault-cert display_name=vault_cert policies=kisai-policy certificate=@/etc/vault/certs/vault-client.crt
vault secrets enable -path=secret kv-v2
vault secrets enable -path=kisaicerts kv-v2

Then point vault.svc’s bootstrap (vaultmanager.store.hcpvault.*) at the URL + cert paths + the registered authname (vault-cert above). Validate with vault.svc startup logs, the first successfully connected to vault backend hcpvault should land within 2 s of boot.

5. TLS material

For any non-loopback deployment, two cert pairs:

Pair	Purpose	Where
Server cert	Terminates incoming mTLS from consumers / operator	`host`/`port` listener
Vault client cert	Authenticates `vault.svc` to HCP/OpenBao	`vaultmanager.store.hcpvault.{certpath,keypath}`

Same CA bundle for both (the cluster’s kisai issuer chain). Generate with vault.svc cert rootca / intermediateca / svccert (the release documentation covers the procedure end-to-end), or with your existing CA.

# Lock down perms
sudo chown -R root:vault /etc/vault/certs
sudo chmod -R 0640 /etc/vault/certs/*.key
sudo chmod 0644 /etc/vault/certs/*.crt

6. First boot

sudo systemctl start vault.svc
sudo systemctl status vault.svc

Verify:

curl -s http://localhost:7999/version          # unauth
curl -s http://localhost:7999/ready            # 200 → ready
curl -s --cert client.crt --key client.key --cacert ca.crt \
    https://localhost:7999/certificate/internal/issuer

Tail logs:

journalctl -u vault.svc -f

Look for successfully connected to vault backend hcpvault (or your backend’s name) within 2 s of start.

7. Day-2 operations

7.1 Operator commands via the CLI

From any host with the kis CLI:

kis vault status   --server https://vault.example.com:7999
kis vault audit query --since 1h
kis vault node-token issue --scope dc=us-west,cluster=prod

Operator auth: an mTLS operator cert resolved from the kis context or --operator-cert / --operator-key. The server enforces the operator-role scope.

7.2 Issuing a join token (for a vault-agent)

kis vault node-token issue \
    --scope dc=us-west,cluster=prod \
    --ttl 1h \
    --uses 1
# → hj_v1_n8x2k_a1b2c3d4e5f6g7h8j9k0

Give the printed token to the operator running vault-agent init on the new node. See the agent guide.

7.3 Inspecting the audit chain

kis vault audit query --since 24h --actor 'spiffe://kis.ai/customer/acme/...'
kis vault audit verify   # walks the chain from genesis

audit verify is slow (whole-chain replay); run it as a periodic CI job rather than interactively.

7.4 Rotating an API-key-style secret

kis vault put vault:db_password "$(openssl rand -hex 24)" \
    --end-date "$(date -u -d '+90 days' +%Y-%m-%dT%H:%M:%SZ)"

The previous active record is marked inactive in the entity store but the corresponding vault blob is not deleted immediately; the janitor sweeps expired blobs after a grace window.

8. Backup & restore

8.1 What to back up

Two stores:

Backend (HCP Vault / OpenBao / Azure KV / AWS SM): the source of truth for secret + cert material. Backup is the backend’s responsibility (Vault Raft snapshots, AWS SM is implicitly backed up, Azure KV has its own export). Confirm your backend’s backup procedure runs.
Data audit + entity tables: vault.svc writes to these. Audit is ALSO mirrored to Chitragupta (audit.svc) for chain durability; the Data block copy is the queryable index. Backup is the Data block operator’s responsibility.

The vault.svc binary itself is stateless, losing the host doesn’t lose data so long as the backend + Data survive.

8.2 Restoring after host loss

Provision a new host, same OS, same package install (§2).
Copy /etc/vault/vault.yaml and /etc/vault/certs/* from config-as-code (your cluster repository).
sudo systemctl start vault.svc.
Verify /ready returns 200 and the connect log fires within 2 s.

No backend or audit state needs reconstruction; vault.svc picks up where the previous instance left off.

9. Upgrade

sudo systemctl stop vault.svc
sudo tar -C /opt/vault -xzf vault.svc-${NEW_VERSION}-linux-amd64.tar.gz
sudo systemctl start vault.svc

Downtime: typically <2 s; consumers using RemoteVaultClient auto-retry. Read the release notes before crossing a minor version boundary, backend or signer interface changes are surfaced there.

10. Troubleshooting

Symptom	Likely cause	Action
`/ready` returns 503	Backend not reachable / not authenticated.	`journalctl -u vault.svc -n 200`: look for `failed to connect to <backend>`. Verify the backend is up, the auth role exists, the cert file is readable by the `vault` user.
`vault.svc: refusing prod` (fsvault)	Mistake: fsvault is configured with `env: prod`.	Switch the backend to `hcpvault` / `openbao` / your real prod backend.
TLS handshake fails on listener	Server cert + client cert chain mismatch.	Verify the CA bundle (`cacertpath`) signs both. Check cert expiry: `openssl x509 -in cert.crt -noout -enddate`.
Vault token “non-renewable” error in logs	The cert role lacks `auth/token/renew-self` permission.	Re-apply kisai-policy on the backend.
Audit emission failing	The audit endpoint is unreachable.	Verify `audit.endpoint` resolves and is reachable. Operations continue, but the audit chain is not durable until it does: investigate immediately.
`vault.svc config list` returns no examples	Broken release binary: examples weren’t embedded at build time.	Re-download the release tarball and verify checksum; if it persists, file a bug with the build supplier.

11. Failure modes

Process crash: systemd restart kicks in. Backend state is unaffected (it’s external). Consumers using RemoteVaultClient retry transparently via lib-chassis backoff. Total recovery: under 5 s in normal operation.

Backend down: vault.svc’s connect loop retries (retryduration). While the backend is unreachable, in-flight secret operations fail with vault-failed-to-retrieve-secret / vault-not-connected. Consumers do NOT have a local cache today (the lib-vault secret cache is wired but disabled, see the consumer guide). Audit + cert verification paths that rely on cached cert public keys keep working.

Single-node has no redundancy. If the host dies, restore per §8.2. For deployments that need fault tolerance, see the HA deployment guide.