Rotate a secret across a fleet

Goal

A credential replaced everywhere it is used, with no request failing during the change, and the old value retired only after the new one is proven.

When you finish rotation is a command rather than a maintenance window, which is what makes it something you do on a schedule instead of after an incident.

Why bother

The reason most secrets are years old is that rotating them is frightening.

Without this	With this
Rotation needs a maintenance window	It runs while traffic flows
The old secret is deleted and something breaks an hour later	It is retired only after the new one is verified
Nobody knows which hosts got the new value	The run names every one
It is done by hand, so it is done once	It runs on a schedule

The shape that avoids an outage

Every safe rotation is the same four moves, and the order is the whole trick:

Generate the new secret and store it alongside the old one.
Distribute it, so every consumer holds both.
Switch consumers to the new one and reload them.
Retire the old one — only once step 3 is verified everywhere.

The overlap in steps 1–2 is what removes the outage. A rotation that replaces the value in one move has a window, however short, in which some consumers hold the old secret and the issuer has already forgotten it.

Before you start

You need	Why
A backend that accepts two valid credentials at once	Without it there is no overlap, and no safe rotation
A list of consumers	You cannot verify what you have not enumerated
A reload that does not drop connections	Otherwise step 3 is an outage of its own

name: rotate-secret
list: true
vars:
  secret_name: api-signing-key
tasks:
  - name: generate
    secret:
      pattern: password
      length: 48
      setvar: new_value

  - name: stage
    vault:
      operation: write
      path: "{{secret_name}}-next"
      value: "{{new_value}}"

Writing to <name>-next rather than over <name> is step 1’s entire point: nothing reads it yet.

function main() {
  const generated = secret.generate({ pattern: 'password', length: 48 });
  if (!generated.success) throw new Error(generated.error);

  // staged under a separate name — nothing reads it yet
  log.info(`generated ${secret_name}-next`);
  return { staged: `${secret_name}-next`, value: generated.secret };
}

secret.generate returns the value as secret, not value. The operation reference has the full parameter set.

Step 2 — distribute, so every consumer holds both

Flow
Script

tables:
  hosts:
    type: csv
    file: ./consumers.csv

  - name: distribute
    map:
      items_path: hosts
      task: push-one
      max_concurrency: 5
    next:
      go: verify-held

  - name: push-one
    ssh:
      host: "{{_item.host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        install -m 0600 /dev/stdin /etc/app/secret.next <<'EOF'
        {{new_value}}
        EOF

const hosts = csv.read({ path: './consumers.csv' }).records;

for (const h of hosts) {
  const c = ssh.connect({ host: h.host, user: user, keyPath: keypath });
  try {
    const r = ssh.execute({
      clientId: c.clientId,
      commands: [`install -m 0600 /dev/stdin /etc/app/secret.next <<'EOF'\n${newValue}\nEOF`],
    });
    if (!r.success) throw new Error(`${h.host}: ${r.error}`);
    log.info(`staged on ${h.host}`);
  } finally {
    ssh.close({ clientId: c.clientId });
  }
}

Mode 0600 on the way in, not afterwards. A secret written world-readable and chmodded a second later was world-readable for a second, and that is long enough to end up in a backup.

Step 3 — switch and reload

Flow
Script

  - name: switch-one
    ssh:
      host: "{{_item.host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        mv /etc/app/secret.next /etc/app/secret
        systemctl reload app
        sleep 2
        curl -fsS http://127.0.0.1:8080/ready > /dev/null

reload, not restart — a reload re-reads configuration without dropping connections. Where a service has no reload, this step is a safe restart instead, one host at a time.

for (const h of hosts) {
  const c = ssh.connect({ host: h.host, user: user, keyPath: keypath });
  try {
    const r = ssh.execute({
      clientId: c.clientId,
      commands: [
        'set -e',
        'mv /etc/app/secret.next /etc/app/secret',
        'systemctl reload app',
        'sleep 2',
        'curl -fsS http://127.0.0.1:8080/ready > /dev/null',
      ],
    });
    if (!r.success) throw new Error(`${h.host} failed to reload: ${r.error} — STOPPING, fleet is now mixed`);
    log.info(`switched ${h.host}`);
  } finally {
    ssh.close({ clientId: c.clientId });
  }
}

Stop at the first failure. Continuing leaves a fleet where some hosts hold the new secret and some the old, which is the state you were trying to pass through quickly, not settle in.

Step 4 — retire the old one, last

Only after every consumer is verified on the new value:

  - name: retire
    vault:
      operation: write
      path: "{{secret_name}}"
      value: "{{new_value}}"

Then revoke the old credential at the issuer. This is the irreversible step, and it is last for that reason: everything before it can be abandoned by leaving secret.next unused.

Which surface to use

	Flow	Script
Which hosts got the new value	Recorded per node	Only what you logged
Failure halfway	Resume from the last host	Re-run, and work out where you were
Runs on a schedule	Yes	Needs a wrapper

Rotate with the flow. The list of hosts that completed is not a nicety here — it is the difference between finishing a partial rotation and starting one again from a fleet in an unknown state.

Verify

Before step 4, prove every consumer is on the new value:

# every host reports the new key id, and none reports the old one
kis flow -f rotate-secret.yaml -t verify-held

Then, after retiring: make one authenticated call per consumer and confirm it still succeeds. A rotation is verified by traffic, not by file contents.

Adapt it

Change	How
Certificates rather than a shared secret	Renew certificates
The backend allows only one credential	Roll host by host, and accept a brief per-host failure window
On a schedule	Attach with the `cron:` operation
Different secret per host	One row per host in the CSV, generate inside the loop