Build a fleet in parallel

Goal

The same fleet build, finishing in a fraction of the time, by running repositories concurrently with a ceiling you choose — without losing the ability to say which repository failed.

When you finish a build that took forty minutes serially will take as long as its slowest few repositories.

Why bother

Serial builds waste a machine. A build host with sixteen cores compiling one repository at a time is idle in fifteen of them.

Without this	With this
Build time is the sum of every repository	Build time approaches the slowest one
A sixteen-core builder runs one compile	The machine is actually used
Nobody re-runs the full build because it takes too long	It is cheap enough to run on every change
Unbounded `&` in a shell script exhausts memory and the OOM killer picks a victim	A ceiling you set, and can lower on a small machine

The ceiling is the part people skip. Unbounded parallelism on a fleet build is how you discover your build host’s memory limit — during a release.

Before you start

You need	Why
A working serial build	See Build every service from trunk
An idea of your machine’s limits	Cores, and memory per compile

Step 1 — extract the per-repository work into its own task

Fan-out needs something to fan out to. The unit of parallelism is a task, so the work for one repository has to be a task by itself.

tasks:
  - name: build-one
    shell: |
      set -e
      cd {{name}}
      go build -ldflags "{{ldflags}}" \
        -o ../build/{{version}}/{{osarch}}/{{name}} ./...

This is not called directly. It is the body that the fan-out invokes once per row.

Step 2 — fan out over the table

  - name: build-all
    map:
      items_path: repos
      task_name: build-one
      max_concurrency: 4

items_path is the table, task_name is the task to run per item, max_concurrency is the ceiling. Four is a conservative default that suits most builders.

Step 3 — choose the ceiling deliberately

The right number is bounded by memory, not cores. Compilers are memory-hungry, and the failure mode of guessing too high is a killed process with a confusing message.

Machine	Reasonable ceiling
4 cores, 8 GB	2
8 cores, 16 GB	4
16 cores, 32 GB	6–8
CI container with a hard memory cap	Divide the cap by the largest single compile

Make it a variable so it can be lowered without editing tasks:

vars:
  concurrency: 4

  - name: build-all
    map:
      items_path: repos
      task_name: build-one
      max_concurrency: "{{concurrency}}"

Step 4 — sequence the groups that must be ordered

Parallel within a group, ordered between groups. If libraries must exist before services build, express that as two fan-outs chained with next, not as one big pool.

  - name: build-libs
    map:
      items_path: libs
      task_name: build-one
      max_concurrency: "{{concurrency}}"
    next:
      go: build-services

  - name: build-services
    map:
      items_path: services
      task_name: build-one
      max_concurrency: "{{concurrency}}"

name: build-parallel
workingdirectory: ~/workspace
continueonerror: true
vars:
  version: "1.2.0"
  osarch: linux-amd64
  ldflags: "-s -w"
  concurrency: 4
tables:
  libs:
    type: csv
    file: ./libs.csv
  services:
    type: csv
    file: ./services.csv

tasks:
  - name: prepare-tree
    shell: |
      mkdir -p build/{{version}}/{{osarch}}
    next:
      go: build-libs

  - name: build-one
    shell: |
      set -e
      cd {{name}}
      go build -ldflags "{{ldflags}}" \
        -o ../build/{{version}}/{{osarch}}/{{name}} ./...

  - name: build-libs
    map:
      items_path: libs
      task_name: build-one
      max_concurrency: "{{concurrency}}"
    next:
      go: build-services

  - name: build-services
    map:
      items_path: services
      task_name: build-one
      max_concurrency: "{{concurrency}}"
    next:
      go: manifest

  - name: manifest
    shell: |
      cd build/{{version}}/{{osarch}}
      ls -1 > MANIFEST.txt
      sha256sum * > SHA256SUMS 2>/dev/null || true
      echo "built $(wc -l < MANIFEST.txt) artefacts"

// build-parallel.js — libraries first, then services
function buildOne(name) {
  const r = shell.execute({
    script: `go build -ldflags "${ldflags} -X main.version=${version}" -o ./dist/${osarch}/${name} ./cmd/${name}`,
    workingDir: `~/workspace/${name}`,
    capture: true,
  });
  return { name, ok: r.success, error: r.error };
}

function main() {
  const libs = csv.read({ path: './libs.csv' }).records;
  const services = csv.read({ path: './services.csv' }).records;

  // libraries must finish before services start — the ordering is the point
  const libResults = libs.map((l) => buildOne(l.name));
  if (libResults.some((r) => !r.ok)) {
    throw new Error('library build failed; not starting services');
  }

  const svcResults = services.map((s) => buildOne(s.name));
  const failed = svcResults.filter((r) => !r.ok);
  if (failed.length) throw new Error(`${failed.length} services failed`);

  return { libraries: libResults.length, services: svcResults.length };
}

kis script run build-parallel.js --vars version=1.2.0

This is the guide where the flow is not merely tidier, it is the feature. map: with max_concurrency and -w gives you a bounded worker pool, per-item output buffering so logs stay readable, and a record of which items completed — none of which the script has. Use the script to develop buildOne; use the flow to run the fleet.

kis flow -f build-parallel.yaml

Throttle on a small machine:

kis flow -f build-parallel.yaml -v concurrency=2

Verify

Compare MANIFEST.txt against the serial build’s. Parallelism must not change what is produced — if the artefact list differs, you have a dependency between repositories that the group ordering does not express.

Adapt it

Change	Where
More groups	Another table and another `map`, chained with `next`
Fail fast instead of finishing	Drop `continueonerror`
Spread across machines	`-w` raises flow workers; see Run a flow across many hosts (planned)

Execution tasks — parallel execution and map nodes
Build every service from trunk