Restart a service safely

Goal

A restart that takes an instance out of rotation, restarts it, waits for it to actually report ready, and fails loudly if it does not — instead of systemctl restart followed by hope.

When you finish restarting is boring, and a restart that breaks something tells you at the time rather than at the next page.

Why bother

sleep 10 && curl is the most common restart procedure in the world and it is wrong in both directions.

Without this	With this
A fixed sleep that is too short on a slow day	Poll until ready, with a real timeout
A fixed sleep that is too long on every other day	Continue the moment it is up
The service fails to start and nobody notices until traffic does	The flow fails immediately
Requests in flight are dropped	Drained first
Restarting three instances takes all three down together	One at a time, stopping at the first failure

The readiness poll is the substance. A sleep encodes a guess about startup time; a poll encodes the actual question, which is is it serving.

Before you start

You need	Why
SSH access	The restart runs on the host
A readiness endpoint	Something that answers only when the service can work
To know how your service drains	Connection draining differs per service

Step 1 — parameterise the instance, not the flow

name: restart-service
vars:
  host: app.example.com
  user: service
  keypath: /path/to/key
  service_name: my-service
  ready_url: http://127.0.0.1:8080/ready
  timeout_seconds: 60
  drain_seconds: 5

Step 2 — record what you are changing from

If the restart makes things worse, you want the before state.

tasks:
  - name: before
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        systemctl is-active {{service_name}} || true
        systemctl show -p ActiveEnterTimestamp {{service_name}} || true

Step 3 — drain

Give in-flight requests a moment to finish. Even a few seconds turns dropped connections into completed ones.

  - name: drain
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        echo "draining for {{drain_seconds}}s"
        sleep {{drain_seconds}}

If your edge supports it, mark the instance unhealthy here instead — draining at the load balancer is better than draining by waiting.

Step 4 — restart

  - name: restart
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        sudo systemctl restart {{service_name}}
        echo "restart issued"

Step 5 — wait for ready, with a deadline

  - name: wait-ready
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        deadline=$(( $(date +%s) + {{timeout_seconds}} ))
        until curl -fsS {{ready_url}} > /dev/null 2>&1; do
          if [ $(date +%s) -ge $deadline ]; then
            echo "NOT READY after {{timeout_seconds}}s"
            sudo systemctl status {{service_name}} --no-pager | tail -20
            exit 1
          fi
          sleep 2
        done
        echo "ready after $(( {{timeout_seconds}} - (deadline - $(date +%s)) ))s"

Dumping the last lines of status on failure means the run log already contains the reason, so you are not SSHing in to find out.

The finished thing

Flow

name: restart-service
list: true
vars:
  host: app.example.com
  user: service
  keypath: /path/to/key
  service_name: my-service
  ready_url: http://127.0.0.1:8080/ready
  timeout_seconds: 60
  drain_seconds: 5

tasks:
  - name: before
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        systemctl is-active {{service_name}} || true
        systemctl show -p ActiveEnterTimestamp {{service_name}} || true

  - name: drain
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        echo "draining for {{drain_seconds}}s"
        sleep {{drain_seconds}}

  - name: restart
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        sudo systemctl restart {{service_name}}
        echo "restart issued"

  - name: wait-ready
    ssh:
      host: "{{host}}:22"
      username: "{{user}}"
      privatekeypath: "{{keypath}}"
      commands: |
        set -e
        deadline=$(( $(date +%s) + {{timeout_seconds}} ))
        until curl -fsS {{ready_url}} > /dev/null 2>&1; do
          if [ $(date +%s) -ge $deadline ]; then
            echo "NOT READY after {{timeout_seconds}}s"
            sudo systemctl status {{service_name}} --no-pager | tail -20
            exit 1
          fi
          sleep 2
        done
        echo "service is ready"

Script

restart-service.js

function main() {
  const c = ssh.connect({ host: host, user: user, keyPath: keypath });
  const run = (cmd) => ssh.execute({ clientId: c.clientId, commands: [cmd] });

  try {
    const before = run(`systemctl is-active ${service_name} || true`).output.trim();
    log.info(`before: ${before}`);

    run(`sleep ${drain_seconds}`);                       // let in-flight requests finish
    const r = run(`sudo systemctl restart ${service_name}`);
    if (!r.success) throw new Error(`restart failed: ${r.error}`);

    // poll for readiness — never sleep and hope
    const deadline = `$(( $(date +%s) + ${timeout_seconds} ))`;
    const ready = run(
      `deadline=${deadline}; ` +
      `until curl -fsS ${ready_url} > /dev/null 2>&1; do ` +
      `  if [ $(date +%s) -ge $deadline ]; then ` +
      `    sudo systemctl status ${service_name} --no-pager | tail -20; exit 1; fi; ` +
      `  sleep 2; done; echo ready`
    );
    if (!ready.success) throw new Error(`${service_name} did not become ready in ${timeout_seconds}s`);

    return { host, service: service_name, ready: true };
  } finally {
    ssh.close({ clientId: c.clientId });
  }
}

kis script run restart-service.js --env prod.yaml --vars host=app-02.example.com

For a rolling restart across a fleet, use the flow. Driving it from a table: gives one independent run per instance, stopping at the first failure — so a bad build takes out one instance rather than all of them. A loop in a script has no such record, and nothing stops the next iteration.

kis flow -f restart-service.yaml

Another instance, same flow:

kis flow -f restart-service.yaml -v host=app-02.example.com

Rolling a fleet

Restart instances one at a time and stop at the first failure, so a bad build takes out one instance rather than all of them:

tables:
  instances:
    type: csv
    file: ./instances.csv

Drive the same tasks with table: instances and leave continueonerror off. The default — stop on failure — is what you want here.

Caution

Do not raise concurrency on a rolling restart. Parallelism is the one thing you do not want: the whole value is that instance two waits until instance one is proven healthy.

Verify

/ready answering is necessary but not sufficient — it says the process started, not that it is serving correctly. Check the metric that matters for your service before calling the restart done.

Related

• Stand up metrics collection (planned) — so you can see the effect
• API conventions — what health and ready mean here