Build every service from trunk

Goal

A complete set of binaries built from the mainline of every repository in your fleet, laid out by version and architecture, with repositories you do not want built opted out by a column in the list rather than by editing the flow.

When you finish you will have an artefact tree that a release flow can consume without knowing anything about how it was produced.

Why bother

Building thirty services by hand is not hard, it is just unrepeatable.

Without this	With this
Build order and flags live in someone’s shell history	They live in a file with a diff history
Every developer’s binary is built slightly differently	One matrix — version, architecture, flags — applied to all
A service that should not ship gets built and shipped	A FALSE in the list, visible to everyone
Artefacts land wherever the person was standing	A predictable tree, so release automation needs no arguments
”Which commit is this binary?”	The tree is versioned and the flow records it

The flag matrix is the hidden value. ldflags, target architecture and version being set once means every binary in the tree was built the same way — which is what makes the tree trustworthy as a release input.

Before you start

You need	Why
The fleet on disk	See Clone or refresh a repository fleet
A prepared build machine	See Prepare a build machine
The same CSV list	Reused, with its build column now doing work

Step 1 — declare the build matrix

Everything that makes a binary different from another binary goes here, and nowhere else.

name: build-trunk
workingdirectory: ~/workspace
continueonerror: true
vars:
  version: "1.2.0"
  osarch: linux-amd64
  ldflags: "-s -w"
  only: ""
tables:
  repos:
    type: csv
    file: ./fleet.csv

only is an escape hatch: empty means everything, otherwise a name to build just one. It costs one variable and saves you writing a second flow.

Step 2 — lay out the artefact tree first

Create the destination before anything writes to it, so a build never fails on a missing directory halfway through.

tasks:
  - name: prepare-tree
    shell: |
      mkdir -p build/{{version}}/{{osarch}}

Versioning the tree means two builds can coexist, which is what makes a rollback a copy rather than a rebuild.

Step 3 — say out loud what you are skipping

A silent skip and a silent failure look identical in a log.

  - name: skip
    table: repos
    check: |
      "{{build}}" === "FALSE"
    shell: |
      echo "not building {{name}} (build=FALSE)"

Step 4 — build the rest

  - name: build
    table: repos
    check: |
      "{{build}}" === "TRUE" && ("{{only}}" === "" || "{{only}}" === "{{name}}")
    shell: |
      set -e
      cd {{name}}
      echo "building {{name}} @ {{version}}"
      GOFLAGS=-trimpath go build -ldflags "{{ldflags}}" \
        -o ../build/{{version}}/{{osarch}}/{{name}} ./...

The check is the whole opt-in mechanism: a row builds only if it says TRUE, and only if it matches only when only is set.

Step 5 — prove the tree is complete

  - name: manifest
    shell: |
      cd build/{{version}}/{{osarch}}
      ls -1 > MANIFEST.txt
      echo "built $(wc -l < MANIFEST.txt) artefacts for {{version}}/{{osarch}}"
      sha256sum * > SHA256SUMS 2>/dev/null || true

The checksums are what let a release flow verify it is shipping what this flow produced.

The finished thing

Flow

name: build-trunk
list: true
workingdirectory: ~/workspace
continueonerror: true
vars:
  version: "1.2.0"
  osarch: linux-amd64
  ldflags: "-s -w"
  only: ""
tables:
  repos:
    type: csv
    file: ./fleet.csv

tasks:
  - name: prepare-tree
    shell: |
      mkdir -p build/{{version}}/{{osarch}}

  - name: skip
    table: repos
    check: |
      "{{build}}" === "FALSE"
    shell: |
      echo "not building {{name}} (build=FALSE)"

  - name: build
    table: repos
    check: |
      "{{build}}" === "TRUE" && ("{{only}}" === "" || "{{only}}" === "{{name}}")
    shell: |
      set -e
      cd {{name}}
      echo "building {{name}} @ {{version}}"
      GOFLAGS=-trimpath go build -ldflags "{{ldflags}}" \
        -o ../build/{{version}}/{{osarch}}/{{name}} ./...

  - name: manifest
    shell: |
      cd build/{{version}}/{{osarch}}
      ls -1 > MANIFEST.txt
      echo "built $(wc -l < MANIFEST.txt) artefacts for {{version}}/{{osarch}}"
      sha256sum * > SHA256SUMS 2>/dev/null || true

Script

build-trunk.js

function main() {
  const repos = csv.read({ path: './fleet.csv' }).records;
  const built = [], failed = [], skipped = [];

  for (const repo of repos) {
    if (only && repo.name !== only) { skipped.push(repo.name); continue; }

    const out = `./dist/${osarch}/${repo.name}`;
    const r = shell.execute({
      script: `go build -ldflags "${ldflags} -X main.version=${version}" -o ${out} ./cmd/${repo.name}`,
      workingDir: `~/workspace/${repo.name}`,
      capture: true,
    });

    if (r.success) { built.push(repo.name); log.info(`built ${repo.name}`); }
    else { failed.push({ repo: repo.name, error: r.error }); log.error(`FAILED ${repo.name}: ${r.error}`); }
  }

  // the report is the deliverable — a build that hides failures is worse than one that stops
  log.info(`built ${built.length}, failed ${failed.length}, skipped ${skipped.length}`);
  if (failed.length) throw new Error(`${failed.length} repositories failed to build`);
  return { built, skipped };
}

kis script run build-trunk.js --vars version=1.2.0 --vars osarch=linux-amd64

Note the throw at the end rather than in the loop: every repository is attempted, and the run still fails. That is what continueonerror: true plus a final check buys you in the flow.

Reach for the flow when the fleet is more than a handful. It records which repositories completed, so a rerun after fixing one broken repository does not rebuild the other thirty.

kis flow -f build-trunk.yaml

Build one service while iterating:

kis flow -f build-trunk.yaml -v only=service-alpha

Cut a different version without editing:

kis flow -f build-trunk.yaml -v version=1.2.1

Verify

MANIFEST.txt should list what you expect. Compare its line count against the number of TRUE rows in the list — a mismatch means a build failed and continueonerror let the run finish, which is the behaviour you want but only if you check.

Adapt it

Change	Where
Cross-compilation	Add GOOS/GOARCH from osarch in the build task
Non-Go services	A second table and a second build task; the tree layout is unchanged
Much faster	See Build a fleet in parallel
Reproducible from tags	See Build a tagged release (planned)

Related

• Build a fleet in parallel
• Publish artefacts to object storage (planned)