Bootstrap and key material

A first boot creates nothing. iam.svc opens no database connection during startup and issues no DDL at serve time: tables come from the migration commands below, and pools, engines and signing keys are built lazily on the first request that names a CPET.

Prerequisites

Thing	Who creates it
PostgreSQL cluster and the database	You — infrastructure-as-code. iam.svc never creates a database.
The tenant’s physical PG schema	iam.svc data bootstrap (CREATE SCHEMA IF NOT EXISTS).
Entity tables, indexes, audit tables	iam.svc data bootstrap
Migration ledger tables	iam.svc data bootstrap
First operator (control plane)	iam.svc seed, or POST /auth/bootstrap
First tenant admin user	iam.svc seed, iam.svc tenant provision, or the provision route
Token signing key	Generated on first mint, unless you supply one

PostgreSQL is the only backend with a connection-pool factory. A dbpool declaring any other type does not fail at boot — it fails the first request that touches that tenant.

Boot proves nothing about the database

Startup resolves configuration, builds the process schema, mounts routes and flips /ready and /health to 200. None of that touches PostgreSQL. Both probes are process liveness only: they never consult the config client, a pool or a tenant, so an instance whose every request returns 500 engine_unavailable still answers ready:true.

The first evidence that credentials, host or schema are wrong is a request, not a log line at startup.

Caution

The default log level is error, and the log.level config key is applied after the -l/--loglevel flag, so the flag is overridden on every boot. To see the per-tenant build breadcrumbs, put the level in the file:

log:
  level: info

Ordered commands, one box, from nothing

iam.svc config list
iam.svc config generate bootstrap-dev > .iam.yaml

Edit .iam.yaml before running anything else — see control-plane addressing below, and set jwks.issuers.iam to the URL this instance is reachable at. Then:

iam.svc data bootstrap -f .iam.yaml --dry-run
iam.svc data bootstrap -f .iam.yaml
iam.svc data bootstrap -f .iam.yaml --tenant default:iam:dev:superadmin
iam.svc seed -f .iam.yaml --email [email protected] --password 'Passw0rd!' --firstname Admin --lastname User
iam.svc -f .iam.yaml

1. --dry-run prints the plan and applies nothing. Read the schema-qualified DDL in it — that is the check that the tenant resolved the pool you intended.
2. data bootstrap with no --tenant uses default:iam:dev:default. It creates the schema, every entity table and the ledger in one lock-less plan, and is safe to re-run: a second pass emits IF NOT EXISTS no-ops.
3. The control plane migrates exactly like a tenant, addressed by its own CPET.
4. seed writes the first operator into the control plane governing --tenant’s product and environment, and a tenant admin (roles: [admin]) into the tenant itself. Both steps are idempotent — an existing operator, or a users row with the same email, is left untouched.
5. Serving starts on host:port — 5030 in every shipped profile. port is mandatory; with neither the key nor -p, boot aborts with invalid/empty port provided.

Smoke-test the result with a login:

curl -s localhost:5030/auth/login \
  -H 'Content-Type: application/json' \
  -H 'X-Customer: default' -H 'X-Product: iam' -H 'X-Env: dev' -H 'X-Tenant: default' \
  -d '{"identity":"[email protected]","password":"Passw0rd!"}'

Caution

--tenant superadmin is not accepted, in any command. Every --tenant value is parsed as a four-segment CPET:

invalid --tenant "superadmin" (want customer:product:env:tenant): key "superadmin" has 1 parts; want 4 (customer:product:env:tenant)

Config profiles

iam.svc config list prints the profiles embedded in the binary; config generate <name> writes one to stdout, -d <dir> to a directory. The reserved name all writes every profile and requires -d.

Profile	Shape
bootstrap-dev	Single file, inline tenant hive, local PostgreSQL, zero-trust off
bootstrap-managed-session	As dev, with a managed-session preset on the tenant
bootstrap-config-server	Tenants, datastores and pools come from config.svc
bootstrap-vault	Pool passwords are Vault secret names; zero-trust mTLS
bootstrap-full	Config server, Vault, mTLS, service discovery

Two keys shipped in those files are read by nothing: enablecache and telemetry.enabled. Neither has an effect.

Caution

zerotrust defaults to true when the key is absent. A hand-written bootstrap that omits it demands TLS material at listener start and fails to start without it. Set it explicitly.

Control-plane addressing

The control plane is a reserved tenant slot — <customer>:<product>:<env>:superadmin — with its own PG schema, its own signing keyring and its own refresh-token table. There is one per product and environment, not one per deployment. It holds the operator roster, the tenant registry and operator grants; see Superadmin and the control plane.

It resolves its pool through the same tenant hive as any tenant, by prefix merge on the CPET. A node keyed superadmin is a prefix of superadmin:… only — it is not consulted for default:iam:dev:superadmin. Key the plane’s node by its full CPET:

hives:
  tenant:
    - path: default:iam:dev:superadmin
      name: superadmin
      active: true
      datastores:
        main:
          dbpool: ctrlpool
      dbpools:
        ctrlpool:
          type: postgres
          dbserver: localhost:5432
          database: erpdb
          dbuser: erpadmin
          password: erpadmin
          schema: superadmin
          maxconnections: 5
          connect_timeout: 5

Caution

If the plane’s CPET falls through to a tenant’s node, the plane is built on that tenant’s pool and its tables are created in the tenant’s schema. The plane carries its own copies of jwt_signing_key and user_refresh_token, so co-locating it merges operator credentials into the tenant’s tables — the separation the plane exists to provide is gone, silently and without an error.

Confirm before applying: data bootstrap --dry-run prints every statement schema-qualified.

A dbpool that omits schema: falls back to the tenant slot of the CPET as the physical schema name — not to iam.

What the migration commands create

data bootstrap is the fresh-database verb: it generates the same diff-based plan the normal lifecycle does, but without persistence (the ledger it would write to does not exist yet) and applies it immediately, printing every step. After it, the persisted, risk-gated plan lifecycle applies.

Ledger tables, created in the same pass and excluded from ordinary plans:

Table	Holds
data_db_version	Applied schema version per datastore
data_plans	Generated plans
data_applied_changes	Per-step apply ledger
data_locks	Cross-instance coordination locks
data_archive_handoff	Retention handoff rows
data_materialized_refresh_tasks	Materialized-refresh queue

Control-plane tables, created by a data bootstrap against the plane’s CPET:

Table	Holds
superadmin	The operator roster
superadmin_grant	Which tenants an operator may act on
tenant	The tenant registry
tenant_eventlog	Registry event log
jwt_signing_key	The plane’s own signing keyring
user_refresh_token	The plane’s own refresh tokens

Steady-state changes go through the persisted lifecycle:

iam.svc data plan generate -f .iam.yaml --tenant <cpet>
iam.svc data plan show <plan-id>
iam.svc data plan apply <plan-id> --confirm-risk medium
iam.svc data applied list --plan-id <plan-id>

Shared flags on the whole data tree: -f <config>, --tenant <cpet> (default default:iam:dev:default), --datastore <name>, --output table|json. --confirm-risk defaults to low and apply refuses a plan whose maximum risk exceeds it.

Note

Migration is per CPET. There is no all-tenants pass — rolling a schema change across a fleet is your own loop over the tenant inventory, and each control plane needs its own run.

The HTTP equivalents are gated differently from the library default on this service: generate and read stay at admin, while apply, seed, locks and maintenance require superadmin. A tenant admin can produce and review a plan; only an operator can execute it. Full route list at /api/iam/.

The first operator

Two paths create the first row in the superadmin entity. Both write into the plane governing the target product and environment, and both are one-time.

Out of band, no token required:

iam.svc seed -f .iam.yaml --tenant default:iam:dev:default --email [email protected] --password 'S3cret!'

Flag	Default	Effect
--email, --password	—	Required
--firstname, --lastname	Admin, User	Names on both rows
--mobile	reserved sentinel	E.164 mobile
--role	admin	Repeatable; roles for the tenant admin user
--superadmin	true	--superadmin=false skips the operator row
--tenant	default:iam:dev:default	Tenant to seed; its plane receives the operator
--datastore	schema default	Datastore to seed into

Pointing --tenant at a reserved plane creates the operator and skips the tenant-admin step with a message — a plane has no users entity.

Over HTTP, gated by the bootstrap.token cluster-plane config key:

curl -s localhost:5030/auth/bootstrap \
  -H 'Content-Type: application/json' \
  -H 'X-Customer: default' -H 'X-Product: iam' -H 'X-Env: dev' -H 'X-Tenant: default' \
  -d '{"token":"dev-bootstrap-secret-change-me","email":"[email protected]","password":"S3cret!","firstname":"Root","lastname":"Operator"}'

{ "ok": true, "id": "U01HVK…", "email": "[email protected]" }

Field	Required
token, email, password, firstname, lastname	Yes
mobile	No

The created operator carries properties.roles: ["root"]. The request is unauthenticated but still needs CPET headers — the plane is derived from the request’s product and environment, so bootstrapping under one product never seeds another’s operators.

Response	Meaning
200	First operator created
403 already_bootstrapped	bootstrap.token unset, token mismatch, or an operator already exists
400 bad_request / missing_fields / no_tenant	Body or tenant context
503 superadmin_not_deployed	The plane has no hive binding, or its schema is not bootstrapped
500 persist_failed	Write failed

Caution

403 already_bootstrapped is one envelope for three causes, deliberately — a prober learns nothing. The empty string never matches, so an unset bootstrap.token disables the endpoint rather than opening it. Rotate or remove the token once the first operator exists.

The endpoint is per product and environment. There is no cross-tenant bootstrap, and operators have no MFA or lockout policy — the control plane carries no MFA tables.

The first tenant

Registration and provisioning are separate steps, so you can register many tenants and provision each when its isolation config is ready.

curl -s localhost:5030/superadmin/registry/tenants \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H 'Content-Type: application/json' \
  -H 'X-Customer: default' -H 'X-Product: iam' -H 'X-Env: dev' -H 'X-Tenant: superadmin' \
  -d '{"slug":"acme:iam:prod:eu1","displayname":"Acme EU"}'

{ "id": "01HVK…", "slug": "acme:iam:prod:eu1" }

The row is written active: true, markedforseed: true. superadmin and shared are reserved and refused in any segment of a slug.

Provisioning then brings the registered tenant live — schema, tables, ledger, admin user, and clearing markedforseed. Over HTTP:

curl -s localhost:5030/superadmin/registry/tenants/provision \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H 'Content-Type: application/json' \
  -H 'X-Customer: default' -H 'X-Product: iam' -H 'X-Env: dev' -H 'X-Tenant: superadmin' \
  -d '{"slug":"acme:iam:prod:eu1","admin_email":"[email protected]","admin_password":"S3cret!"}'

{ "cpet": "acme:iam:prod:eu1", "bootstrap_steps_applied": 61,
  "admin_id": "U01HVK…", "admin_existed": false, "activated": true }

Or out of band, holding no token:

iam.svc tenant provision -f .iam.yaml --tenant acme:iam:prod:eu1 --email [email protected] --password 'S3cret!'

--tenant, --email and --password are required; --firstname/--lastname default to Admin/User; --datastore defaults to the schema’s default datastore. The command is idempotent: a re-run applies no DDL, reports admin_existed: true, and re-clears the flag.

Both paths require the tenant’s node — datastore, pool, schema: — to already exist in the hive, and they refuse a tenant that is not registered:

provision: tenant "acme:iam:prod:eu1" is not registered — POST /superadmin/registry/tenants first

The pure-CLI path that skips the registry entirely is data bootstrap followed by seed against the tenant’s CPET. Nothing else is needed to onboard a product: products register themselves from the first request that names one, so there is no product allow-list and no product-registration route.

There is no tenant-deletion or deactivation route. Removing a registry row is a delete on the generic entity surface under the plane’s headers, requires the root role, and drops only the row — no schema, table or data teardown exists anywhere in the service.

Signing keys

Keys are per tenant, and the control plane has its own. There is no platform-wide key: compromising one tenant’s key cannot forge tokens for another. The key for a tenant resolves in three tiers, highest first.

Tier	Trigger	Resolution
1. Vault secret	Tenant property jwt.signing.key set to a value not starting with -----BEGIN	The value is a secret name; the chassis Vault client fetches the PKCS#8 Ed25519 PEM
2. Inline PEM	The property value starts with -----BEGIN	The value is the private key. Development only
3. Generated	The property is absent	A keypair is generated into the tenant’s own jwt_signing_key table on first use and loaded thereafter

hives:
  tenant:
    - path: acme:iam:prod:eu1
      jwt:
        signing:
          key: jwt-signing-acme-eu1     # a Vault secret name

Caution

A vault secret name with no Vault client registered is a hard error, not a fallback. The service will not silently mint its own key when you clearly intended an external one.

Tier 1 and 2 keys are never written to the database, and their kid is the RFC 7638 JWK thumbprint — stable across instances and restarts. The default algorithm is EdDSA (Ed25519); ES256 is accepted on the row and served in JWKS for clients without an Ed25519 library.

Public keys are served per tenant at GET /.well-known/jwks.json, which requires CPET headers and no credentials, and returns both active and inactive keys so tokens signed before a rotation keep verifying. The response is cached in-process and caps at 100 keys with no indication that it truncated. Every response on this service carries Cache-Control: no-store, including this one, so implement your own JWKS caching rather than relying on HTTP caching.

Note

The session preset records a crypto suite, and fedramp names ECDSA P-384. That selection does not reach the signer: every token this service mints is PASETO v4.public signed with the tenant’s Ed25519 key. See Tokens.

Storage and encryption at rest

Generated keys live in the tenant’s jwt_signing_key table. The private-key column is marked no-log and no-export, and only the service’s own internal identity may read the entity — the external REST and admin surfaces are denied, admins included.

Encryption at rest for that column is AES-256-GCM and is installed only when a Vault client is registered. With no Vault client, the hooks are not installed and fields declared encrypted — signing-key PEMs and TOTP shared secrets — are written in plaintext. No error is raised.

Enabling it later needs a data migration: the read hook leaves an undecryptable value as-is, so existing plaintext rows keep working and stay plaintext until they are rewritten.

Rotation

There is no rotation endpoint and no rotation schedule. Rotation is a Go API on the key manager only, and it refuses externally-managed keys outright, since an external key always wins the resolution hierarchy and a database-rotated key would never sign anything.

For a tier 1 or 2 key, rotate at the source and then invalidate the tenant so the next request reloads:

curl -s -X DELETE localhost:5030/superadmin/tenant/eu1/cache \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H 'X-Customer: acme' -H 'X-Product: iam' -H 'X-Env: prod' -H 'X-Tenant: superadmin'

The :tenant segment is a bare tenant name; customer, product and environment come from the caller’s own headers. Invalidation drops that tenant’s pools, engines, schemas and cached keys — the next request rebuilds them.

Rotation is not revocation. Old keys stay published and valid until the tokens they signed expire. To cut access now, rotate and drop the principal’s refresh tokens or sessions; access tokens still live out their TTL. Refresh-token rotation tombstones have no retention sweep.

Config keys that bootstrap depends on

Key	Plane	Default	Effect
port	boot	none — required	Listener port; 5030 in every profile
host	boot	"" (all interfaces)	Bind address
zerotrust	boot	true	Demands TLS material at listener start
jwks.issuers.iam	boot, top level	none	Where this service’s own tokens are verified from
bootstrap.token	cluster	"" (endpoint disabled)	Gates POST /auth/bootstrap; compared constant-time
boot.retry_window	boot	90s	How long each dependency client retries before failing loud
<tenant>.dbpools.<name>.schema	tenant	the tenant slot name	The physical PG schema — the isolation boundary
<tenant>.dbpools.<name>.password	tenant	""	A Vault secret name when a Vault client is registered, otherwise a literal
<tenant>.jwt.signing.key	tenant	absent	Vault secret name, or an inline PEM

Caution

jwks.issuers.iam is top level, never nested under hives:. Nested, a token this service just minted returns 401 on its own /rest and /admin routes, with no configured URL for issuer in the log.

A pool password is tried against Vault first and used literally on failure, so a mistyped secret name in a development boot becomes a literal password and surfaces as a connection authentication failure rather than a configuration error.

Verifying a bootstrap

Check	Command or route	What proves it
The plan targets the schema you expect	data bootstrap --dry-run	Every statement is schema-qualified in the printed plan
Tables exist	iam.svc data applied list --plan-id <id>	Per-step ledger
Credentials and pool are correct	POST /auth/login with CPET headers	401 invalid_credentials proves reachability; 500 engine_unavailable does not
The key exists and is published	GET /.well-known/jwks.json with CPET headers	A non-empty key set for that tenant
The plane is separate	Inspect the plane’s schema:	superadmin tables in their own PG schema

Failures during bootstrap and their causes are listed in Error codes. Two are worth knowing here: 503 superadmin_not_deployed means the plane has no hive binding or has never been bootstrapped, and 500 engine_unavailable means the CPET has no resolvable tenant config or its pool failed to connect — the HTTP body is deliberately generic, and the underlying error is in the log.

Caution

The listener binds with SO_REUSEPORT by default, so a second instance on the same host and port starts without an address-in-use error and quietly splits traffic with the first. Shutdown closes the listener without draining in-flight requests; drain upstream.