Deploy: Single-Node

Audience: DevSecOps — deploy and operate the config service in single-node (bbolt) mode. Appropriate for: edge / air-gapped sites, small Pattern-1 deployments, dev/staging environments.

For HA deployments with etcd + leader election, see the HA deployment guide.

1. Prerequisites

Requirement	Detail
OS	Linux x86_64 / arm64. macOS supported for dev only.
Privileged user	Service runs as config (uid/gid created by package). Do not run as root.
Filesystem	A persistent local directory for bbolt (default /var/lib/config). Must survive process restart; NOT a tmpfs.
Network	One TCP port (default :7123). Optional outbound HTTPS to the audit service if audit is enabled.
TLS material	Recommended for any non-loopback deployment. PEM-format cert/key + the CA bundle that signs your clients (when require_client_cert: true).
Time	NTP-synced. Lease TTLs and JWT nbf/exp rely on monotonic, reasonably-accurate wall clock.

2. Install

2.1 From a release tarball

sudo install -d -o root -g root /opt/config
sudo tar -C /opt/config -xzf config.svc-${VERSION}-linux-amd64.tar.gz
sudo install -d -o root -g config -m 0750 /etc/config
sudo install -d -o config -g config -m 0700 /var/lib/config

2.2 systemd unit

/etc/systemd/system/config.svc.service

[Unit]
Description=config control plane
After=network.target

[Service]
Type=simple
User=config
Group=config
ExecStart=/opt/config/config.svc server -f /etc/config/config.yaml
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

# Hardening (single-node only — no etcd, no shared FS):
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/config /var/log/config
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now config.svc

3. Bootstrap YAML — single-node

Write /etc/config/config.yaml:

# Bootstrap for single-node, bbolt-backed config service.
# Generated by: config.svc generate bootstrap-single-node

# Service identity — chassis convention is flat top-level keys.
# `sid:` is the instance ID used by the leader-election driver.
service: config
version: 2.0.0
sid:     ${HOSTNAME}

storage:
  mode: bbolt
  bbolt:
    path: /var/lib/config

# Listener — chassis svcboot reads these top-level keys.
host: ""
port: "7123"
zerotrust: true   # mTLS on; set false for plain-HTTP loopback only.
# timeouts:
#   read:       30s
#   write:      30s
#   readheader: 5s

# TLS material is resolved from container/<service>-ktls when
# zerotrust:true — wire it via the standard kis-tls flags or
# chassis cert-bundle mechanism.

hives:
  cluster:
    sources:
      - name: platform-cluster-defaults
        backend: git
        url: https://git.example.com/platform/cluster-defaults.git
        branch: main
        owns: ["**"]
        priority: 0
        writable: false
        auth_username: x-access-token
        auth_token: ${env:GIT_TOKEN}

  tenant:
    sources:
      - name: platform-tenant-defaults
        backend: git
        url: https://git.example.com/platform/tenant-defaults.git
        branch: main
        owns: ["**"]
        priority: 0
        writable: false
        auth_username: x-access-token
        auth_token: ${env:GIT_TOKEN}
      # Break-glass overrides: a WRITABLE postgres source at higher priority.
      # Git is READ-ONLY via the config API (the server holds no push creds),
      # so API writes (PUT/PATCH/DELETE) land here and override the git
      # defaults; change the git-backed defaults through your normal git flow.
      - name: tenant-overrides
        backend: postgres
        dsn: ${env:CONFIG_PG_DSN}
        table: config_overrides
        owns: ["**"]
        priority: 40
        writable: true

auth:
  apikey_file: /etc/config/apikeys.yaml
  # JWT, mTLS sub-blocks omitted = unconfigured (API key only).

# Optional. Omit entirely or leave endpoint empty to skip audit
# emission (server falls back to NoOpEmitter — dev only; required
# for any environment that needs an append-only audit trail).
# audit:
#   endpoint: https://chitragupta.internal/events
#   timeout:  5s

# OTLP telemetry. Set collector + signals.* to enable; missing
# collector → exporter not constructed (no "produced zero addresses"
# spam during boot).
# telemetry:
#   collector: otel-collector.internal:4317
#   insecure: false
#   environment: prod
#   traces: { enabled: true, sampling_rate: 0.1 }

3.1 Generate a template

config.svc generate bootstrap-single-node > /etc/config/config.yaml

Other templates: bootstrap, bootstrap-airgap, bootstrap-dev. List with config.svc generate.

4. API keys

The bootstrap references an API-key file. Initial admin key:

/etc/config/apikeys.yaml

keys:
  - key:          <plain-text-secret>     # compared constant-time at request time
    subject_type: user                    # "user" | "service"
    subject_id:   bootstrap-admin
    attrs:                                # arbitrary; exposed to expr policy as `subject.attrs`
      role: platform_admin

The key is stored in plain text and compared in constant time against the incoming X-Api-Key header. No hashing. Protect the file with tight filesystem permissions instead (see below). Rotate keys by regenerating the file and reloading the service.

Generate a fresh random key:

KEY=$(openssl rand -hex 32)
echo "API key (give to consumer + paste into the `key:` field above): $KEY"

Permissions:

sudo chown root:config /etc/config/apikeys.yaml
sudo chmod 0640                  /etc/config/apikeys.yaml

5. TLS material

For any deployment beyond a single host, generate a server cert signed by your internal CA:

# Server cert (example with cfssl)
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
    -config=cfssl.json -profile=server server-csr.json | \
    cfssljson -bare server

sudo install -m 0644 -o root -g config server.pem    /etc/config/server.pem
sudo install -m 0640 -o root -g config server-key.pem /etc/config/server.key
sudo install -m 0644 -o root -g config clients-ca.pem /etc/config/clients-ca.pem

mTLS toggle: set listeners.http.tls.require_client_cert: true only after every client has its own client cert + the CA chain enrolled. Until then, leave it false so API-key clients can connect over TLS.

6. First boot

sudo systemctl start config.svc
sudo systemctl status config.svc

API smoke tests (every endpoint is covered in the HTTP API; these are the minimal post-install verifications):

# Unauthenticated probes
curl -s http://localhost:7123/health   # → {"status":"ok"}
curl -s http://localhost:7123/ready    # → {"status":"ready"}
curl -s http://localhost:7123/version  # → JSON with build info

# Authenticated — confirms the api-key file is parsed and policy
# allows the bootstrap admin to list hives
curl -s -H "X-Api-Key: $ADMIN_KEY" http://localhost:7123/hives | jq
# → {"hives":["cluster","tenant"]}

# Confirm the binary self-registered in its own SD
curl -s -H "X-Api-Key: $ADMIN_KEY" \
    http://localhost:7123/services/config/instances | jq
# → {"instances":[{...sid, addr, tags...}]}

If any of these fail post-install, jump to the troubleshooting table in §10.

Tail logs:

journalctl -u config.svc -f

Look for config subsystems wired; reconcilers running within ~2 seconds of start.

7. Day-2 operations

7.1 Reading config

# Resolved cluster config for a service at a specific path
curl -H "X-Api-Key: $KEY" \
  http://localhost:7123/hives/cluster/us-west/prod-cluster-01/forge-meta/2.0.0

# Resolved tenant config
curl -H "X-Api-Key: $KEY" \
  http://localhost:7123/hives/tenant/acme/forge/prod/acme-prod/forge-meta

7.2 Writing config (break-glass)

Writes go to a source marked writable: true — a postgres (or etcd) source. Git is read-only via the API (the server holds no push creds); its values are the defaults, and a writable pg/etcd source at higher priority overrides them. To change git-backed defaults, commit to the repo directly. The reconciler picks up the write and publishes a new snapshot.

curl -X PUT -H "X-Api-Key: $WRITE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"log_level": "debug"}' \
  http://localhost:7123/hives/tenant/acme/forge/prod/acme-prod/forge-meta
# → 200, response body + ETag header carry the source version, e.g. "3".

For optimistic concurrency, send If-Match: "<version>" using the version from a prior write response to that path (not the GET snapshot ETag — that is the per-hive snapshot version, a different counter). A stale version returns 409 Conflict with the current version:

curl -X PUT -H "X-Api-Key: $WRITE_KEY" -H "If-Match: \"3\"" \
  -H "Content-Type: application/json" -d '{"log_level": "info"}' \
  http://localhost:7123/hives/tenant/acme/forge/prod/acme-prod/forge-meta

7.3 Subscribing to changes (SSE)

curl -N -H "X-Api-Key: $KEY" \
  "http://localhost:7123/stream/config?hives=tenant"

You’ll see :connected, periodic event: heartbeat keepalives, and event: snapshot / event: delete for live changes. Resume after disconnect by setting Last-Event-Id: <last-snapshot-version> in the request header.

7.4 Listing subscribers

curl -H "X-Api-Key: $ADMIN_KEY" http://localhost:7123/subscribers

8. Backup & restore

The entire state is the bbolt directory.

# Backup (consistent point-in-time snapshot — bbolt's Tx model)
sudo systemctl stop config.svc
sudo tar -C /var/lib -czf "config-$(date +%F).tgz" config
sudo systemctl start config.svc

For hot backups, the bbolt CLI provides bbolt cp for online copies:

bbolt cp /var/lib/config/snapshots.bbolt /backup/snapshots-$(date +%F).bbolt
bbolt cp /var/lib/config/leases.bbolt    /backup/leases-$(date +%F).bbolt
bbolt cp /var/lib/config/revert.bbolt    /backup/revert-$(date +%F).bbolt

Restore: stop service, replace the directory contents, start service.

9. Upgrade

# Single-instance upgrade is a stop-replace-start.
sudo systemctl stop config.svc
sudo tar -C /opt/config -xzf config.svc-${NEW_VERSION}-linux-amd64.tar.gz
sudo systemctl start config.svc

Subscribers reconnect automatically via lib-chassis backoff. Downtime window: typically <2 s. The bbolt files are forward- compatible across patch versions; read the release notes before crossing a minor version boundary.

10. Troubleshooting

Symptom	Likely cause	Action
/ready returns 503	Initial reconcile failed for a hive (invalid YAML, schema rejection).	journalctl -u config.svc — look for initial reconcile failed: hive=<name>. Fix the source content.
Subscribers see heartbeats but no snapshot events	Source watcher not picking up changes, or validator silently rejecting.	Log lines at debug level on the reconcile path will identify which. Set log.level: debug and restart.
/hives/<name>/<path> returns 404	Path not yet reconciled (resolves to empty), or wrong path levels for the hive.	config.svc explain --hive <name> --path <path> shows the resolution walk.
Boot blocks on Git fetch	Git source URL unreachable / auth wrong.	Sources init synchronously at boot. Check auth_token env var is populated; verify outbound HTTPS to the Git host.
OTel exporter spam in logs	telemetry.collector set but unreachable.	Either unset telemetry.collector (disables exporter entirely) or fix the collector address.

11. Failure modes

Process crash: systemd restart kicks in (Restart=on-failure). bbolt is crash-safe; snapshots written before the crash survive. Subscribers reconnect via SSE backoff. Total recovery: under 5 seconds in normal operation.

Disk full: Writes start failing with ENOSPC; reads continue to work from in-memory cache. /ready will eventually return 503 once the reconciler can’t make progress. Free space; service recovers.

Single-node mode has no redundancy. If the host dies, configure restore from the most recent backup. Critical deployments should run the HA topology — see the HA deployment guide.