Clone or refresh a repository fleet

Goal

Every repository your build needs, present on disk and current — cloned if missing, fast-forwarded if already there — driven from a list you maintain rather than a script you edit.

When you finish you will have one flow that takes a machine with nothing checked out to a machine ready to build, and that is safe to re-run any time.

Why bother

A fleet of repositories is the point where “just clone it” stops scaling.

Without this	With this
A README with thirty `git clone` lines that drifts the week it is written	One list, read by the flow, reviewed like code
A new machine takes an hour of copy-paste	One command, and you get coffee
Nobody is sure whether their checkout is current	Re-running is the answer, and it is cheap
A new repository is added and half the team never gets it	Add a row; everyone picks it up on their next run
One missing repository aborts the whole script	The run continues and tells you which one

The list is the deliverable as much as the flow is. It becomes the answer to “what makes up this product”, in a file, with a history.

Before you start

You need	Why
Read access to the repositories	Cloning
A working directory with room	Everything lands under it
A list of repositories	A CSV with a name and a URL

Step 1 — put the fleet in a list, not the flow

The flow should not change when the fleet does.

name,url,build
service-alpha,https://git.example.com/org/service-alpha.git,TRUE
service-beta,https://git.example.com/org/service-beta.git,TRUE
tool-gamma,https://git.example.com/org/tool-gamma.git,FALSE

The build column is not used here, but it will be by Build every service from trunk — one list, several flows.

Step 2 — declare the list as a table

name: refresh-fleet
workingdirectory: ~/workspace
continueonerror: true
vars:
  branch: main
tables:
  repos:
    type: csv
    file: ./fleet.csv

continueonerror: true matters. One unreachable repository should not stop the other twenty-nine from updating; you want the run to finish and tell you what it could not do.

Step 3 — clone what is missing

A task with table: runs once per row, with each column available as a variable.

tasks:
  - name: clone
    table: repos
    check: |
      "{{name}}" !== ""
    shell: |
      if [ ! -d "{{name}}/.git" ]; then
        echo "cloning {{name}}"
        git clone --branch {{branch}} {{url}} {{name}}
      else
        echo "present: {{name}}"
      fi

Step 4 — fast-forward what exists

Separate from cloning, because the failure modes differ: a clone fails on access, an update fails on local changes. Keeping them apart tells you which happened.

  - name: update
    table: repos
    shell: |
      cd {{name}} 2>/dev/null || exit 0
      git fetch --prune --quiet
      git checkout {{branch}} --quiet
      git pull --ff-only --quiet && echo "updated: {{name}}" \
        || echo "SKIPPED {{name}} — local changes or diverged"

--ff-only is the safety. It refuses to merge, so a repository with local work is reported and left alone rather than silently altered.

Step 5 — report

  - name: summary
    shell: |
      echo "fleet under $(pwd):"
      ls -1d */ | wc -l

name: refresh-fleet
list: true
workingdirectory: ~/workspace
continueonerror: true
vars:
  branch: main
tables:
  repos:
    type: csv
    file: ./fleet.csv

tasks:
  - name: clone
    table: repos
    check: |
      "{{name}}" !== ""
    shell: |
      if [ ! -d "{{name}}/.git" ]; then
        echo "cloning {{name}}"
        git clone --branch {{branch}} {{url}} {{name}}
      else
        echo "present: {{name}}"
      fi

  - name: update
    table: repos
    shell: |
      cd {{name}} 2>/dev/null || exit 0
      git fetch --prune --quiet
      git checkout {{branch}} --quiet
      git pull --ff-only --quiet && echo "updated: {{name}}" \
        || echo "SKIPPED {{name}} — local changes or diverged"

  - name: summary
    shell: |
      echo "fleet under $(pwd):"
      ls -1d */ | wc -l

function main() {
  const repos = csv.read({ path: './fleet.csv' }).records;
  const cloned = [], updated = [], dirty = [];

  for (const repo of repos) {
    const dir = `~/workspace/${repo.name}`;

    if (!file.exists({ path: `${dir}/.git` }).exists) {
      const r = git.clone({ url: repo.url, directory: dir, depth: 1 });
      if (!r.success) throw new Error(`clone ${repo.name}: ${r.error}`);
      cloned.push(repo.name);
      continue;
    }

    // never discard someone's work to make a build succeed
    const st = git.status({ directory: dir });
    if (!st.clean) { dirty.push(repo.name); log.error(`skipping ${repo.name}: working tree dirty`); continue; }

    const r = git.pull({ directory: dir, remote: 'origin' });
    if (!r.success) throw new Error(`pull ${repo.name}: ${r.error}`);
    updated.push(repo.name);
  }

  log.info(`cloned ${cloned.length}, updated ${updated.length}, skipped ${dirty.length} dirty`);
  return { cloned, updated, dirty };
}

kis script run refresh-fleet.js --vars branch=main

git.status is what makes this safe, and it is the call with no natural flow equivalent — deciding whether to pull is a branch, and branches are what scripts are for.

Either works here, and the script is arguably clearer: the clone-or-pull decision is a conditional, which reads better in code than as a choice: node. Use the flow when this runs as one stage of a larger build.

kis flow -f refresh-fleet.yaml

Track a release branch instead of the mainline:

kis flow -f refresh-fleet.yaml -v branch=release-1.1

Verify

The run log names every repository as cloned, present, updated or skipped. Anything marked SKIPPED has local work — deal with those by hand, which is the point of not merging for you.

Adapt it

Change	Where
A different branch per repository	Add a `branch` column and use `{{branch}}` from the row
Shallow clones for speed	`git clone --depth 1` — fine for building, not for tagging
Run it much faster	See Build a fleet in parallel for the `map` pattern

Build every service from trunk — the next step
Data & Templating tasks — tables and CSV sources