Restore from a backup

Goal

A restore you have performed before, into a scratch target, with a check that the data arrived — and the same flow able to restore for real behind an explicit flag.

When you finish you will know your backups work, which is a different claim from knowing your backup job succeeds.

Why bother

This is the guide people skip, and the omission is the whole risk.

Without this	With this
Backups are a hypothesis nobody has tested	A rehearsal you have run and timed
The first restore happens during an incident, from memory	The steps are written down and were followed last month
”How long does a restore take?” is a guess	You have measured it, so your RTO is a number
An archive that restores into a corrupt state looks fine on the shelf	The verify step reads the data back
The restore command is typed under pressure	It is a flow, run with one flag

A backup job proves a file was written. Only a restore proves the file is worth having. Run this on a schedule against scratch, not just when something breaks.

Before you start

You need	Why
An archive from Back up a service	The input
A scratch target	Somewhere a mistake costs nothing
The service’s restore tool	Matching the tool that produced the archive

Step 1 — make the destructive path opt-in

The default must be the safe one. A flow that restores over production unless told otherwise is a loaded weapon.

name: restore-service
vars:
  host: scratch.example.com
  user: service
  keypath: /path/to/key
  archive_store: /mnt/backups
  archive: ""
  target: scratch
  confirm: "no"

target: scratch and confirm: "no" are the defaults. Restoring for real requires saying so.

Step 2 — pick the archive explicitly

tasks:
  - name: select
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        if [ -z "{{archive}}" ]; then
          echo "available archives:"
          ls -1t {{archive_store}}/backup-*.tar.gz | head -10
          echo "ERROR: pass -v archive=<filename>"
          exit 1
        fi
        test -f {{archive_store}}/{{archive}}
        echo "selected {{archive}}"

Refusing to guess is deliberate. “Restore the latest” is right until the latest is the one that captured the corruption.

Step 3 — verify the archive before touching anything

  - name: verify-archive
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        tar -tzf {{archive_store}}/{{archive}} > /dev/null
        echo "archive opens cleanly"

Step 4 — guard the real target

  - name: guard
    check: |
      "{{target}}" !== "scratch" && "{{confirm}}" !== "yes"
    shell: |
      echo "REFUSING: target={{target}} requires -v confirm=yes"
      exit 1

Step 5 — restore

  - name: restore
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        echo "restoring {{archive}} into {{target}}"
        <service-restore-command> --input {{archive_store}}/{{archive}} --target {{target}}

Step 6 — read the data back

The step that separates a restore from a hopeful copy.

  - name: verify-restore
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        <service-query-command> --target {{target}} --count
        echo "RESTORE VERIFIED"

Count something you can predict. A restore that produces an empty but valid database passes every check except this one.

name: restore-service
list: true
vars:
  host: scratch.example.com
  user: service
  keypath: /path/to/key
  archive_store: /mnt/backups
  archive: ""
  target: scratch
  confirm: "no"

tasks:
  - name: select
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        if [ -z "{{archive}}" ]; then
          echo "available archives:"
          ls -1t {{archive_store}}/backup-*.tar.gz | head -10
          echo "ERROR: pass -v archive=<filename>"
          exit 1
        fi
        test -f {{archive_store}}/{{archive}}
        echo "selected {{archive}}"

  - name: verify-archive
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        tar -tzf {{archive_store}}/{{archive}} > /dev/null
        echo "archive opens cleanly"

  - name: guard
    check: |
      "{{target}}" !== "scratch" && "{{confirm}}" !== "yes"
    shell: |
      echo "REFUSING: target={{target}} requires -v confirm=yes"
      exit 1

  - name: restore
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        echo "restoring {{archive}} into {{target}}"
        <service-restore-command> --input {{archive_store}}/{{archive}} --target {{target}}

  - name: verify-restore
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        <service-query-command> --target {{target}} --count
        echo "RESTORE VERIFIED"

function main() {
  // the guard is the most important line in a restore
  if (target !== 'scratch' && confirm !== 'yes') {
    throw new Error(`refusing to restore into '${target}' without confirm=yes`);
  }

  const c = ssh.connect({ host: host, user: user, keyPath: keypath });
  const run = (cmd) => {
    const r = ssh.execute({ clientId: c.clientId, commands: [cmd] });
    if (!r.success) throw new Error(`${cmd}: ${r.error}`);
    return r.output;
  };

  try {
    const chosen = archive || run(`ls -1t ${archive_store}/backup-*.tar.gz | head -1`).trim();
    log.info(`restoring ${chosen} into ${target}`);

    run(`tar -tzf ${chosen} > /dev/null`);               // refuse a corrupt archive
    run(`mkdir -p /tmp/restore && tar -C /tmp/restore -xzf ${chosen}`);
    run(`psql ${target} < /tmp/restore/app.sql`);

    // a restore you have not counted is a restore you have not done
    const rows = Number(run(`psql -tAc 'select count(*) from customers' ${target}`).trim());
    if (rows === 0) throw new Error('restore produced an empty table');

    return { archive: chosen, target, rows };
  } finally {
    ssh.close({ clientId: c.clientId });
  }
}

kis script run restore-service.js --env dr.yaml --vars target=scratch

Rehearse with either; run the real thing as a flow. A restore is the operation you least want to repeat from the beginning, and the flow is the one that can tell you the extract succeeded and the load did not.

Rehearse into scratch:

kis flow -f restore-service.yaml -v archive=backup-20260731T020000Z.tar.gz

Restore for real, deliberately:

kis flow -f restore-service.yaml \
  -v archive=backup-20260731T020000Z.tar.gz \
  -v target=production -v confirm=yes

Verify

The flow verifies itself, twice — once that the archive opens, once that the restored data is readable. Time the run and record it: that number is your recovery time, and it is worth knowing before someone asks.

Adapt it

Change	Where
Restore to a point in time	Add a timestamp argument to the restore command
Rehearse monthly	Schedule it against `scratch` — see Control tasks
Stronger verification	Compare a row count or checksum against the source

Back up a service — the other half