Back up a service

Goal

A dated archive of a service’s data, written to storage that is not the machine being backed up, verified before it is trusted, with old archives pruned on a window you choose.

When you finish you will have a flow you can attach to a schedule, and — more importantly — a backup you have reason to believe in, because the flow refuses to report success on an archive it could not verify.

Why bother

Most backup systems work. The ones that fail do so quietly, and the failure is discovered by the person who needed the backup.

Without this	With this
A cron line on one host that nobody has read since it was written	A flow in version control that someone reviewed
The dump exits zero, the disk was full, the archive is empty	test -s and tar -tzf fail the run instead
Backups accumulate until the volume fills — and the first symptom is a failed backup	A retention window prunes on every run
Filenames from date on the host, in whatever locale it has	One UTC stamp generated in the flow, sorting correctly everywhere
The backup sits on the disk you are protecting against	It is moved off, and the move is a step you can see
Restores are theory	The same flow shape restores, and you rehearse it

The verify step is the whole argument. A backup job that reports success on a broken archive is worse than no backup job, because it buys confidence you have not earned.

The cost is one afternoon. The alternative is finding out during an incident.

Caution

A backup you have never restored is a hypothesis. Pair this with Restore from a backup and rehearse it — the first restore should not happen during an incident.

Before you start

You need	Why
SSH access to the host	The dump runs where the data is
A separate volume or bucket	A backup on the same disk does not survive the disk
The service’s own dump tool	Use it rather than copying files under a running process

Step 1 — name the archive after the moment it was taken

Shell date arithmetic drifts between hosts and locales. Generate the stamp once, in the flow, in UTC, and every archive across every host sorts correctly.

name: back-up-service
vars:
  host: app.example.com
  user: service
  keypath: /path/to/key
  workdir: /var/backups/staging
  archive_store: /mnt/backups
  retention_days: 14

tasks:
  - name: stamp
    script:
      language: javascript
      code: |
        function main() {
          const d = new Date();
          const p = (n) => String(n).padStart(2, '0');
          const name = `${d.getUTCFullYear()}${p(d.getUTCMonth() + 1)}${p(d.getUTCDate())}`
                     + `T${p(d.getUTCHours())}${p(d.getUTCMinutes())}${p(d.getUTCSeconds())}Z`;
          return { archive: `backup-${name}.tar.gz` };
        }
      setvar: stamped

What main() returns is captured under setvar, so later tasks refer to {{stamped.archive}}. Sorting is lexicographic and correct because the format is year-month-day.

Step 2 — take the dump

Use the service’s own dump command. Copying data files while the process is running gives you an archive that restores into a corrupt state — and you find out at restore time.

  - name: dump
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        mkdir -p {{workdir}}
        <service-dump-command> --output {{workdir}}/{{stamped.archive}}

Step 3 — verify before trusting

The step people skip. A dump command can exit zero and leave a truncated or empty archive — a full disk is the usual cause. Check the archive is readable and non-trivial before it counts as a backup.

  - name: verify
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        test -s {{workdir}}/{{stamped.archive}}
        tar -tzf {{workdir}}/{{stamped.archive}} > /dev/null
        echo "archive verified: $(du -h {{workdir}}/{{stamped.archive}} | cut -f1)"

test -s catches the empty file; tar -tzf catches the truncated one. Both are cheap, and together they turn “the command ran” into “the archive opens”.

Step 4 — move it off the machine

Until this step you have a copy on the disk you are protecting against.

  - name: store
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        mkdir -p {{archive_store}}
        mv {{workdir}}/{{stamped.archive}} {{archive_store}}/

For off-host storage, replace the mv with an upload to your object store — see Files & Transfer tasks.

Step 5 — prune on a window

Unbounded backups fill the volume and the first symptom is a failed backup, which is the worst time to discover it.

  - name: prune
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        find {{archive_store}} -name 'backup-*.tar.gz' \
          -mtime +{{retention_days}} -print -delete

-print before -delete means the run log records exactly what was removed.

The finished thing

Flow

name: back-up-service
list: true
vars:
  host: app.example.com
  user: service
  keypath: /path/to/key
  workdir: /var/backups/staging
  archive_store: /mnt/backups
  retention_days: 14

tasks:
  - name: stamp
    script:
      language: javascript
      code: |
        function main() {
          const d = new Date();
          const p = (n) => String(n).padStart(2, '0');
          const name = `${d.getUTCFullYear()}${p(d.getUTCMonth() + 1)}${p(d.getUTCDate())}`
                     + `T${p(d.getUTCHours())}${p(d.getUTCMinutes())}${p(d.getUTCSeconds())}Z`;
          return { archive: `backup-${name}.tar.gz` };
        }
      setvar: stamped

  - name: dump
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        mkdir -p {{workdir}}
        <service-dump-command> --output {{workdir}}/{{stamped.archive}}

  - name: verify
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        test -s {{workdir}}/{{stamped.archive}}
        tar -tzf {{workdir}}/{{stamped.archive}} > /dev/null
        echo "archive verified: $(du -h {{workdir}}/{{stamped.archive}} | cut -f1)"

  - name: store
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        mkdir -p {{archive_store}}
        mv {{workdir}}/{{stamped.archive}} {{archive_store}}/

  - name: prune
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        find {{archive_store}} -name 'backup-*.tar.gz' \
          -mtime +{{retention_days}} -print -delete

Script

// back-up-service.js — vars arrive from --vars or --env
function main() {
  const p = (n) => String(n).padStart(2, '0');
  const d = new Date();
  const stamp = `${d.getUTCFullYear()}${p(d.getUTCMonth() + 1)}${p(d.getUTCDate())}`
              + `T${p(d.getUTCHours())}${p(d.getUTCMinutes())}${p(d.getUTCSeconds())}Z`;
  const archive = `backup-${stamp}.tar.gz`;

  const c = ssh.connect({ host: host, user: user, keyPath: keypath });
  try {
    const run = (cmd) => {
      const r = ssh.execute({ clientId: c.clientId, commands: [cmd] });
      if (!r.success) throw new Error(`${cmd}: ${r.error}`);
      return r.output;
    };

    run(`mkdir -p ${workdir}`);
    run(`pg_dump app > ${workdir}/app.sql`);
    run(`tar -C ${workdir} -czf ${workdir}/${archive} app.sql`);

    // verify before trusting: a tar that will not list is not a backup
    run(`tar -tzf ${workdir}/${archive} > /dev/null`);
    const size = Number(run(`stat -c %s ${workdir}/${archive}`).trim());
    if (size < 1024) throw new Error(`archive suspiciously small: ${size} bytes`);

    run(`cp ${workdir}/${archive} ${archive_store}/${archive}`);
    run(`find ${archive_store} -name 'backup-*.tar.gz' -mtime +${retention_days} -delete`);

    log.info(`backed up ${archive} (${size} bytes)`);
    return { archive, size };
  } finally {
    ssh.close({ clientId: c.clientId });
  }
}

kis script run back-up-service.js --env staging.yaml

One connection for the whole backup rather than one per step, which is the script’s real advantage over the flow here — and its real cost, because closing it is now your job. Hence the finally.

The flow is the one to run on a schedule. Both do the same work, but a backup that fails at the upload step should tell you that it failed at the upload step, with the dump and the archive recorded as done. That is a run record, and only the flow keeps one.

kis flow -f back-up-service.yaml

Rehearse without writing anything:

kis flow -f back-up-service.yaml --dryrun

Point it at a different host without editing the file:

kis flow -f back-up-service.yaml -v host=other.example.com

Verify

The flow fails rather than reporting a bad backup, so a green run means the archive opened. To confirm the whole chain, list the store and check the newest entry is today’s:

ls -lh /mnt/backups | tail -5

Then rehearse a restore. That is the only check that tests the backup rather than the flow.

Adapt it

Change	Where
Off-host object storage	Replace store with an upload task
A different retention	retention_days
Run it nightly	Attach to a schedule — see Control tasks
Per-environment paths	An environment file, passed with -e env.yaml -n production
Several services	One flow each. A shared flow that backs up everything fails as a unit

Related

• Automate — the engine behind kis flow
• Files & Transfer tasks — archives, object storage
• Execution operations — the script: task