Deploy: High Availability

Audience: DevSecOps — engineers running the config service in HA mode with etcd backend + leader election. Appropriate for: production deployments, Pattern-2/3 sites, anywhere a single-host failure must not interrupt config availability.

For single-node deployments, see single-node deployment.

1. Topology

                    ┌──────────────────┐
                    │  Load balancer   │
                    │   (round-robin)  │
                    └────────┬─────────┘
                             │ HTTP/HTTPS :7123
        ┌────────────────────┼────────────────────┐
        ▼                    ▼                    ▼
  ┌───────────┐        ┌───────────┐        ┌───────────┐
  │  ic-1     │        │  ic-2     │        │  ic-3     │   ← N config service
  │ (leader)  │        │           │        │           │     instances
  └───────────┘        └───────────┘        └───────────┘
        │                    │                    │
        └────────────────────┼────────────────────┘
                             ▼
                   ┌──────────────────┐
                   │   etcd cluster   │   ← 3 or 5 nodes,
                   │   (3 or 5 node)  │     odd quorum
                   └──────────────────┘

Read scaling: every instance serves reads from its in-memory view of the etcd snapshot store. The LB load-balances reads across all instances.

Write serialization: writes go through whichever instance the LB hands them to. Each write transacts directly against the etcd source store; etcd’s revision counter is the serialization point. The leader is not the write coordinator — it’s only the reconciler driver. Writes bypass the leader.

Reconciliation: a single instance (the elected leader) runs the per-hive reconcile loops. Followers do not reconcile but do serve reads from the same etcd store. Leader election lease TTL: 15 s default (configurable). On leader loss, a new leader is elected within ~2× TTL and resumes the reconcile loop.

2. Prerequisites

Requirement	Detail
etcd cluster	3 or 5 members. Production etcd hygiene applies (quorum, snapshotting, member health). Same network reachability for all config service instances.
etcd TLS	Strongly recommended. Generate client certs for the config service; pin the etcd CA.
Load balancer	TCP or HTTP/L7. Sticky sessions NOT required (reads are stateless). For SSE, configure long-lived connection timeouts on the LB (≥ 600 s idle).
N ≥ 3 instances	One per AZ minimum. Instances are stateless beyond the etcd connection.
Time	NTP-synced across all config + etcd hosts. Leader election relies on monotonic clocks; large skew causes flapping.
TLS material	Each instance gets its own server cert (matching the public DNS / IP). Common CA for client validation.

3. etcd preparation

Create a dedicated user for the config service in etcd with read/write access to the key prefixes it uses:

ETCDCTL_API=3 etcdctl user add config
ETCDCTL_API=3 etcdctl role add config-rw
ETCDCTL_API=3 etcdctl role grant-permission config-rw \
    --prefix=true readwrite /infinity-config/
ETCDCTL_API=3 etcdctl user grant-role config config-rw
ETCDCTL_API=3 etcdctl auth enable

Key prefixes used (the binary’s etcd storage layout — internal, shown here so etcd RBAC + monitoring rules can be scoped tightly):

Prefix	Contents
`/infinity-config/snapshots/<hive>/`	Resolved snapshots, keyed by path.
`/infinity-config/leases/<service>/`	Discovery leases.
`/infinity-config/versions/`	Per-hive monotonic version counters.
`/infinity-config/election/`	Leader election lease + leader identity.

4. Bootstrap YAML — HA mode

# Service identity — chassis convention is flat top-level keys.
# sid: MUST be unique per instance (used by leader election).
service: config
version: 2.0.0
sid:     ${HOSTNAME}

storage:
  mode: etcd
  etcd:
    endpoints:
      - etcd-0.internal:2379
      - etcd-1.internal:2379
      - etcd-2.internal:2379
    username: config
    password: ${env:CONFIG_ETCD_PASSWORD}
    dial_timeout: 5s
    tls:
      ca_file:   /etc/config/etcd-ca.pem
      cert_file: /etc/config/etcd-client.pem
      key_file:  /etc/config/etcd-client.key
    leader_election:
      enabled: true
      ttl:     15s

# Listener — chassis svcboot reads these top-level keys. TLS
# material is resolved from container/<service>-ktls when
# zerotrust:true.
host: ""
port: "7123"
zerotrust: true

hives:
  cluster:
    sources:
      - name: platform-cluster-defaults
        backend: git
        url: https://git.example.com/platform/cluster-defaults.git
        branch: main
        owns: ["**"]
        priority: 0
        writable: false
        auth_username: x-access-token
        auth_token: ${env:GIT_TOKEN}

      # API writes land in this etcd-backed source (highest priority).
      - name: runtime-overlay
        backend: etcd
        endpoints: [etcd-0.internal:2379, etcd-1.internal:2379, etcd-2.internal:2379]
        prefix: /infinity-config/source-overlays/cluster/
        owns: ["**"]
        priority: 100
        writable: true

  tenant:
    sources:
      - name: platform-tenant-defaults
        backend: git
        url: https://git.example.com/platform/tenant-defaults.git
        branch: main
        owns: ["**"]
        priority: 0
        writable: false
        auth_username: x-access-token
        auth_token: ${env:GIT_TOKEN}

      - name: runtime-overlay
        backend: etcd
        endpoints: [etcd-0.internal:2379, etcd-1.internal:2379, etcd-2.internal:2379]
        prefix: /infinity-config/source-overlays/tenant/
        owns: ["**"]
        priority: 100
        writable: true

auth:
  apikey_file: /etc/config/apikeys.yaml
  jwt:
    jwks_url: https://auth.example.com/.well-known/jwks.json
    audience: config
    issuer:   https://auth.example.com/
  mtls:
    trusted_ca_file: /etc/config/clients-ca.pem
    spiffe_id_pattern: "spiffe://example.com/ns/*/sa/*"

audit:
  endpoint: https://chitragupta.internal/events
  timeout:  5s

telemetry:
  collector: otel-collector.internal:4317
  insecure:  false
  environment: prod
  traces:    { enabled: true, sampling_rate: 0.1 }

Each instance has the same YAML except sid: — the chassis instance ID MUST be unique per instance (typically ${HOSTNAME}). It’s used by the etcd leader-election driver and by the self-registration lease.

5. Deployment

5.1 Rolling out N instances

Bring instances up one by one. The first instance to start wins the leader election; the rest become followers. Reads start serving as soon as /ready returns 200 on each instance — typically <3 seconds after process start (etcd dial + warm-load).

5.2 LB configuration

Health probe: GET /ready. Mark instance out-of-rotation on 503.
Idle timeout: ≥ 600 s for SSE long-polls. Heartbeats fire every 20 s on the server side, so 600 s gives 30× margin.
Backend pool: every instance. Reads are equivalent across instances.
Layer-7 LB recommended for path-based routing in the future, but TCP works fine today (single port, mTLS terminated on the config service instances themselves).

5.3 First-boot verification

API smoke tests (full reference: HTTP API). From any node that can reach the LB:

# Unauth probes
curl -s "https://config.internal:7123/version"   # JSON build info
curl -s "https://config.internal:7123/ready"     # {"status":"ready"}

# Authenticated read (mTLS path; swap in --header X-Api-Key for API-key auth)
curl -s --cert client.pem --key client.key \
    "https://config.internal:7123/hives"
# → {"hives": ["cluster", "tenant"]}

# Confirm all N instances self-registered
curl -s --cert client.pem --key client.key \
    "https://config.internal:7123/services/config/instances" | jq
# → "instances": [N entries, one per running instance]

Verify leadership:

# On any instance:
sudo journalctl -u config.svc | grep -E "won election|lost election" | tail

You should see exactly one “won election” log line across all instances during a steady-state window. The others stay quiet (follower mode).

6. Leader election semantics

Property	Value
Election library	`go.etcd.io/etcd/client/v3/concurrency.Election`
Key	`/infinity-config/election/` (single-key campaign)
Lease TTL	15 s (`storage.etcd.leader_election.ttl`)
Re-election window	≤ 2× TTL after leader loss (~30 s worst case)
What the leader does	Runs the per-hive `Reconciler.Run` loops. Initial reconcile + watch-driven incremental reconciles.
What followers do	Serve reads. Accept writes (writes go directly to the etcd source, not via the leader).

The leader is purely a reconcile-loop driver. There is no single-writer bottleneck on the read or write paths.

6.1 Leader loss

When the leader dies / loses its etcd lease:

Followers detect the election key’s lease expiry (≤ 15 s).
They campaign; one wins.
The winner runs initial reconcile (idempotent; produces the same snapshots that the previous leader did).
Reconcile-watch loops resume.

During the gap: reads continue to serve from the etcd snapshot store (no degradation). Writes continue to land in the etcd source overlay. Only new source-side changes (Git pushes, fs events on fs sources) are delayed until a new leader picks them up. Total unavailability of the reconcile pipeline: ≤ 30 s.

7. Day-2 operations

7.1 Adding an instance

Provision the new host with bootstrap YAML (identical to other instances except instance_id).
Start the service.
Add to LB backend pool.
Verify /ready returns 200.

The new instance immediately serves reads. It will join the leader election; it stays follower unless the current leader steps down or dies.

7.2 Removing an instance

Drain from LB (stop accepting new connections).
Wait ~30 s for in-flight SSE streams to drain (existing subscribers will reconnect to surviving instances).
Stop the service.
If this was the leader, a new leader is elected automatically.

7.3 Rolling upgrade

for host in ic-1 ic-2 ic-3; do
    ssh $host '
        sudo systemctl stop config.svc &&
        sudo tar -C /opt/config -xzf /tmp/config.svc-${NEW_VERSION}.tar.gz &&
        sudo systemctl start config.svc
    '
    # Wait for /ready before moving to the next host
    until curl -sf https://${host}.internal:7123/ready > /dev/null; do
        sleep 1
    done
done

The cluster remains fully available throughout; reads continue from the other N-1 instances, leader fails over if the upgraded host was the leader. Total LB pool reduction: one instance at a time.

7.4 etcd maintenance

Snapshotting / compaction on etcd is independent of the config service. Standard etcd ops apply. The config service reconnects automatically on etcd member churn (etcd client library handles endpoint failover).

8. Monitoring

config service exposes OTel signals (traces + metrics) when telemetry.collector is set. Required metrics for HA monitoring:

Metric	Purpose
`infinity_config_reconcile_latency_seconds`	P99 should stay < 1 s under normal load.
`infinity_config_schema_failures_total`	Spike = bad config push.
`infinity_config_writes_total` (labeled hive)	Write rate per hive.
`infinity_config_subscribers`	Number of active SSE consumers.
`infinity_config_drift_behind_total`	Subscribers reporting they’re behind. Non-zero = back-pressure or network.
`infinity_config_leader_elections_total`	Leader changes. Frequent = unstable network / etcd issues.

Alert thresholds (starting point — tune per site):

Reconcile latency P99 > 5 s sustained for 10 m → page.
Schema failures > 0 in last 5 m → page.
Leader elections > 3 in last 15 m → investigate etcd / network.
Subscribers < expected baseline → check LB + DNS.

9. Backup & disaster recovery

9.1 What to back up

The source of truth lives in:

Source stores: Git repos (already version-controlled and remote-replicated by your Git provider).
etcd: snapshot via etcdctl snapshot save regularly; this captures both the source-overlay state and the resolved snapshots.

config service instances themselves are stateless beyond their local config files. Lose all instances → bring up new ones pointed at the same etcd cluster + Git repos → state is intact.

9.2 Restore from etcd snapshot

Standard etcd restore procedure:

etcdctl snapshot restore snapshot.db \
    --name etcd-restore-1 \
    --initial-cluster etcd-restore-1=https://etcd-restore-1:2380 \
    --initial-cluster-token etcd-cluster-restore \
    --initial-advertise-peer-urls https://etcd-restore-1:2380
# … repeat for each etcd member, then start the restored cluster.

Point config service instances at the restored etcd endpoints in their bootstrap YAML; restart. Snapshot state and source overlays return to the snapshot point-in-time.

9.3 Loss of all etcd data without backup

Source stores (Git) still hold the source-of-truth. Bring up a fresh etcd cluster, start the config service — the reconciler walks every source and rebuilds the snapshot store from scratch. Source-overlay (API-write) state since the last Git commit is lost.

10. Troubleshooting

Symptom	Likely cause	Action
Frequent leader elections	etcd lease can’t be renewed reliably.	Check etcd quorum health, network latency between config service and etcd members, clock skew.
One instance lags others on reads	Local clock skew or etcd connection issue on that instance.	`journalctl -u config.svc -n 500` on the lagging instance. Restart if stuck.
Writes succeed but subscribers don’t see them	Reconciler not running anywhere — no current leader.	`journalctl
`/ready` 503 across all instances	etcd unreachable / auth failed.	Check etcd cluster health. Verify config service can authenticate to etcd.
Subscribers see snapshots from old leader and new leader interleaved	Normal during election transition.	No action needed — subscribers’ `Last-Event-Id` resume eventually picks the higher-versioned snapshot.