Migrations

Every table, column and index in a tenant’s database is derived from that tenant’s entity definitions by the backend dialect. Applying that DDL is a separate, explicit operator action: a definition change on disk changes nothing in any database until you run a migration verb.

The platform rule

A service never hand-rolls SQL, a DSN, or a migration. The rule is structural, not a convention:

Concern	Who owns it
DDL text	The backend dialect, from the loaded definition object — CREATE TABLE, CREATE INDEX, ALTER COLUMN, audit/history side tables, CREATE SCHEMA
Connection string	The host, built from the tenant’s pool config with the password resolved through Vault
Migration	The plan pipeline — generate, review, apply, ledger

The planner is orchestration only; it never contains a backend-specific string. The only way author-written SQL enters a database is a declared init script or migration script, and both run through the same plan and ledger machinery as generated DDL.

The lifecycle

Stage	What it does	Produces	Touches the database
Discover	Introspects the live database for the target namespace, excluding tables prefixed data_	The live schema: tables, columns, indexes, side-table names	Reads only
Compare	Diffs the composed definitions against the live schema in physical type space	Diff{Ops, Refusals, Warnings}	No
Plan	Lowers each op to SQL through the dialect and assigns 1-based step order	Plan{PlanID, Source, MaxRisk, Warnings, Refusals, Steps}, persisted as one plan row plus one step row per step, all pending	Writes ledger rows
Apply	Runs steps in order under a datastore lock, transitioning ledger rows as it goes	ApplyResult{PlanID, Status, StepsApplied, StepsFailed, StepsSkipped, FailedStep, HaltedStep}	Yes — the only stage that does
Verify	Validation steps inside the plan run a read query and compare column 1 of row 1 to the step’s expected value	Step pass or fail	Reads only

Plans are deterministic on the declaration side. Entities sort by name, fields by declared position then name, indexes by name, and step order is op order — so the same definitions against the same live state produce the same additive plan, in the same order, every time. Two environments can be compared plan to plan.

Note

Two things are collected by walking the live schema’s own maps rather than a sorted list: drop_column ops and unknown-table warnings. Their relative order can differ between runs of the same diff. Everything driven by the definitions is ordered.

Comparison happens in physical type space, not logical. Several logical types map to one physical type, so a purely logical comparison would report a freshly created database as full of drift. The dialect is threaded through the diff so a matched column is compared as “what would CREATE TABLE emit for this field” against the live column’s raw type, and defaults are compared the same way. Generate → apply → generate therefore converges on an empty plan.

Automatic versus explicit

Thing	Automatic	Explicit
Deriving DDL from definitions	✅
Applying DDL		data bootstrap, or data plan generate + data plan apply
Creating the PostgreSQL schema (namespace)	✅ CREATE SCHEMA IF NOT EXISTS, issued by bootstrap before the plan
Creating the physical database		External — IaC or an operator, never the engine
Datastore init.scripts	✅ on the first engine build per process, per (tenant, datastore)	data init run forces a re-pass
migrations[].scripts		Appended to the plan, applied by plan apply
Seeds		data seed run
Engine rebuild after a definition change		DELETE /cache, a tenant config refresh, or a restart
Foreign keys, CHECK constraints, enum types		Never emitted by the plan pipeline — declare them in an init script
Dropping a table or an index		Never emitted at all; reported as a warning
Dropping a column		Opt in at generate and confirm high risk at apply

Nothing schema-shaped is ever applied at runtime. The engine build path runs datastore init scripts and nothing else; there is no migrate call anywhere on it. That is deliberate — a definition change reaching a running process must not silently reshape a production database.

Bootstrap: a fresh database

bootstrap is a separate verb because the plan that creates the plan ledger cannot be recorded in that ledger. It unions the engine’s own system entities into a copy of the composed schema, generates the plan unpersisted, and applies it without taking a lock — neither the ledger table nor the lock table exists yet.

data.svc -f data.yaml data bootstrap --tenant acme:shop:prod:eu1
data.svc -f data.yaml data bootstrap --tenant acme:shop:prod:eu1 --datastore main
data.svc -f data.yaml data bootstrap --tenant acme:shop:prod:eu1 --scope shared
data.svc -f data.yaml data bootstrap --tenant acme:shop:prod:eu1 --dry-run

It is idempotent: every generated statement is IF NOT EXISTS, and a re-run against a current database diffs to zero steps. After the first run, the persisted generate → apply lifecycle takes over.

Six engine-owned tables are created in the tenant’s own namespace alongside the entity tables:

Table	Holds
data_plans	One row per plan: id, source, max risk, status, trigger
data_applied_changes	The step-level ledger — one row per plan step or seed application, with status, timing, checksum and error
data_locks	Coordination locks, one row per held resource
data_archive_handoff	The retention queue between marking a row for archive and the archive writer confirming it
data_materialized_refresh_tasks	The outbox for cross-entity materialized computed-field refreshes
data_db_version	Created empty. Nothing in the shipped pipeline writes or reads it

Note

--dry-run prints the plan and applies no step, but the CREATE SCHEMA still runs — it is issued before the plan is generated, not as part of it.

Entities carrying scope: shared or scope: superadmin are served by the scope’s own engine and pool, so their DDL belongs to the deployment rather than to any one tenant. --scope shared and --scope superadmin migrate those planes once per deployment into the scope’s reserved binding and namespace; --tenant still names a tenant of the product whose schema declares them. Scoped entities never appear in a tenant plan.

Generate, show, apply

data.svc -f data.yaml data plan generate --tenant acme:shop:prod:eu1
data.svc -f data.yaml data plan show <plan-id> --tenant acme:shop:prod:eu1 --output json
data.svc -f data.yaml data plan apply <plan-id> --tenant acme:shop:prod:eu1 --confirm-risk medium
data.svc -f data.yaml data applied list --tenant acme:shop:prod:eu1 --plan-id <plan-id>

generate writes a plan row and one row per step, all pending, and executes no DDL. apply rehydrates that plan from the ledger, so generate and apply may happen in different processes, by different people, minutes or days apart. Rehydration does not restore warnings or refusals — a plan only reaches pending when it has none.

Each step carries the exact statement it will run. Read it before applying; --output json prints the full SQL of every step, which the table renderer truncates.

The change vocabulary

Every op the diff can emit, what triggers it, the steps it lowers to, and its risk tier:

Op	Emitted when	Steps	Risk
add_table	An entity has no live table	CREATE TABLE, plus one CREATE INDEX per declared index	low
add_column	A field has no live column and is nullable: true or not required:	ALTER TABLE … ADD COLUMN	low
add_column	The field is required: and not nullable:, with a default	ALTER TABLE … ADD COLUMN	medium
add_index	A declared index is missing	CREATE INDEX	low
add_audit_table / add_audit_index	History strategy typeaudit, <entity>_audit missing	One create each	low
add_history_table / add_history_index	History strategy type4, <entity>_history missing	One create each	low
change_column_default	The default drifted from what CREATE TABLE would emit	ALTER COLUMN … SET/DROP DEFAULT	low
change_column_nullable	Live NOT NULL → declared nullable	DROP NOT NULL	low
change_column_nullable	Live nullable → declared NOT NULL	A NULL-count validation expecting 0, then SET NOT NULL	medium
change_column_type	Physical types differ and the change is on the compatible list	Dialect-emitted ALTER COLUMN	medium
rename_column	Auto-detected 1:1 shape match	ALTER TABLE … RENAME COLUMN	medium
drop_column	A live column has no field and drops are opted in	An informational row count, then DROP COLUMN	high

Indexes over fields marked no-index or encrypted are skipped when the table is created — a plaintext index on ciphertext leaks ordering and equality without being useful.

Risk and the apply gate

Risk is low, medium or high. Plan.MaxRisk is the maximum across the plan’s ops, and apply compares it to what the caller confirmed:

Confirmed	Applies
low (the default)	Table, column and index additions, side tables, default changes, nullability loosening
medium	Also type widening, nullability tightening, renames, NOT NULL columns with a default
high	Also column drops

A plan whose risk exceeds the confirmed level is refused outright and no step runs — the run fails before the lock is even taken. The confirmed value is the operator’s acknowledgement, which is why it is a parameter and not a config setting: --confirm-risk on the CLI, confirm_max_risk on the HTTP route.

migrations[].risk in a definition is parsed and stored but never consulted; the gate uses the planner-computed maximum.

Refusals

A refusal is the diff declining to guess. Three kinds exist:

Kind	Raised when	Fix
data_loss	A live column has no matching field and drops are not opted in	Restore the field, or re-generate with drops allowed and confirm high risk
needs_explicit_migration	An incompatible type change, or a required: non-nullable field added with no default	Declare an expand-contract migration script, or give the field a default
ambiguous_change	Rename detection matched more than one candidate	Declare the rename explicitly in a migration script

Only these type changes apply inline. Everything else — including every narrowing — is refused:

From	To
string	text
int	bigint
float	decimal
date	datetime

Rename detection only fires when an entity has both unmatched fields and unmatched live columns, and only pairs them on an exact shape match: same canonical type, same DDL-level nullability, same default presence (not value), same uniqueness. More than one candidate refuses rather than moving data between columns.

Caution

A diff with refusals produces a plan with zero steps that is never persisted, even though the CLI prints a plan id and POST /admin/data/plans returns 200. Branch on the refusals list, not on the status code — reading that plan id back returns “plan not found”. Applying a plan that still carries refusals is a hard error.

What the diff never touches

Thing	Behaviour
Foreign keys	Never emitted. references: drives joins and eager loading in the compiler; a database provisioned by bootstrap or plan apply has no declared FK constraints
CHECK constraints	Discovered, never compared
Enum types	No CREATE TYPE is emitted by any migration path. A field mapping to a database enum needs an init script
Unknown live tables and indexes	Reported as warnings. There is no drop-table or drop-index op; warnings never block apply
Audit and history side tables	Created in one shot when missing, then left alone. They are never column-diffed — their shape is engine-owned
System tables	Filtered out of the live scan by the data_ prefix, and injected into the definition set only during bootstrap
Entities whose materialization is not table	Skipped entirely — no ops, no warnings. Check this first when a table is unexpectedly missing
Purely virtual computed fields	Excluded from the column diff. Array fields and materialized computed fields are always treated as nullable whatever nullable: says — the dialects skip NOT NULL on both

Steps and execution

Step kind	Semantics
ddl	One statement, executed directly. No outer transaction
dml	Same, for data statements
validation	Runs a read query and compares the first column of the first row to the expected value: 0, any integer, any, or informational (always passes)
raw_script	An author-supplied body, split at top-level ;. Runs inside one transaction when the step carries transactional, otherwise each statement runs bare — see the caution below for which paths carry it
manual_gate	Halts the run for operator confirmation. Honoured by the executor, but no planner path emits one
seed	Skipped by the executor; seed rows are written by the seed runner

Statement splitting understands single quotes with '' escapes, double-quoted identifiers, line and block comments, and dollar-quoting ($$, $tag$). Block comments do not nest. Anything more exotic belongs in one statement per file.

Caution

transactional survives only where the step reaches the executor in memory — the datastore init pass. data_applied_changes has no column for it, so a migrations[].scripts[] step loses the flag when plan apply rehydrates the plan from the ledger: every rehydrated raw_script step runs each statement bare, with no enclosing transaction, whatever the definition declared. Write migration script bodies so a mid-script failure is recoverable on its own — the flag will not roll them back.

Caution

drop_column and rename_column statements are built directly from the entity name rather than through the dialect, so they are emitted unqualified and unquoted and resolve through the connection’s search_path — a different namespace from the rest of the plan on a schema-qualified deployment. Read those steps in the plan before applying them.

The row count that precedes a DROP COLUMN is informational: the executor records it and proceeds. It shows the operator what is at risk; it does not stop them. The NULL count before SET NOT NULL is the opposite — it expects 0 and fails the plan when it is not met.

Failure and partial application

Steps run in order and the run halts on the first failure. That step flips to failed, the plan flips to failed, every later step stays pending, and the error comes back with a partial result. Steps already applied stay applied.

Plan status	Meaning
pending	Generated, not yet run
running	Apply in progress
applied	Every step succeeded
failed	A step failed; earlier steps are applied
halted	Stopped at a manual gate

Step status	Meaning
pending	Not reached
running	In flight
applied	Succeeded
failed	Raised an error, or a validation mismatched
skipped	Fast-forwarded, or suppressed by a dry run

There is no rollback. The executor deliberately runs each step without an outer transaction, because wrapping the plan would roll back steps the ledger has already recorded as applied. Recovery is fix-and-re-run: generated DDL is IF NOT EXISTS, and the ledger records exactly what ran. migrations[].down is parsed and stored, and never executed by anything.

Note

Resuming mid-plan is a Go-API option only — no CLI flag and no HTTP field reaches it, and it skips on step order alone without consulting recorded status. In practice, re-generate against the current live state and apply the new plan.

--dry-run on apply runs validation steps and records every DDL, DML and raw-script step as skipped with a dry-run note. It takes no lock.

Ordering and coordination

A migration verb targets exactly one (tenant, datastore) pair. There is no fleet sweep — a deployment iterates tenants itself. Per tenant, the order is fixed:

1. Provision the physical database and grant the pool’s role rights on it. External.
2. data bootstrap for that tenant’s datastore, once. It creates the namespace, the entity tables and the ledger.
3. data plan generate + data plan apply for every later definition change.
4. The first request for that (tenant, datastore) pair builds the engine and runs the datastore init scripts once.

Scope planes (shared, superadmin) are bootstrapped once for the deployment, not per tenant.

One coarse lock guards each (tenant, datastore), keyed tenant:<key>:datastore:<name>, held as a row in data_locks in the database being changed:

Property	Value
Scope	One lock per (tenant, datastore) — migration, init scripts, seeds and retention all contend for it
Default lease	5 minutes, extended by a heartbeat at one third of the lease when the lease exceeds 30s
Crash behaviour	The lease expires and the next caller steals the row
Recorded	Holder, host, operation kind, plan id, notes, acquired and expiry timestamps
Not taken by	Dry runs, and the bootstrap apply (the table does not exist yet)

Operation kind is recorded for forensics only — it is not part of the key, so retention cannot run while a migration runs on the same datastore. The init runner treats a held lock as benign: it returns with zero scripts applied and moves on.

Caution

Force-releasing a lock deletes the row without checking the holder or the expiry. Releasing one mid-run lets two migrations race on the same datastore. The deployable service gates lock release on the superadmin role for that reason — see IAM.

The namespace must be set in three places

Ledger, lock and plan SQL run over the raw database handle, which bypasses the per-checkout search_path a shared-endpoint pool relies on. The tenant’s physical schema — the pool’s schema: property, falling back to the composed schema’s name — has to reach all three of: the dialect (so DDL is emitted qualified), the discovery inputs (so introspection reads the right namespace), and the ledger and lock manager (so their own rows land in the right namespace). Setting only some of them is the classic phantom-diff and wrong-namespace-ledger failure. See Datastores and database pools for how the value is resolved.

Raw SQL inside a plan

migrations[].scripts[] attaches author-written SQL to a version label. The scripts become raw_script steps appended after the generated DDL, in migrations[] order and then declaration order. To run SQL before the generated DDL, split the work across two migration entries.

migrations:
  - name: add-order-audit
    version: "2026.07.01"
    description: "trigger for the orders audit log"
    scripts:
      - path: sql/triggers/orders_audit.sql
        kind: ddl
      - path: sql/indexes/orders_concurrent.sql
        kind: ddl
        transactional: false

Key	Type	Default	Notes
name	string	—	Migration name
version	string	—	Version label. Becomes the entity column of every ledger row for these scripts
description	string	—	Copied to each emitted step’s notes in the generate output. The step ledger has no notes column, so plan show does not replay it
scripts[].path	string	—	Required. Resolved at load time against the definition assets; a missing file is a hard load error
scripts[].description	string	—	Operator-facing context
scripts[].kind	string	ddl	ddl or dml. Anything else is a load error
scripts[].transactional	bool	true	Parsed and carried onto the step, but dropped when plan apply rehydrates from the ledger — see the caution under Steps and execution. Honoured on the datastore init path
up, down, risk	—	—	Parsed and stored; no code path reads them

Script bodies are rendered as templates before execution. {{tenant}} is the tenant key and {{schema}} is the physical namespace to qualify objects with; conditionals and filters work, not just variable substitution.

Caution

An undefined variable renders as the empty string and raises no error. A typo like {{schmea}}.orders becomes .orders, and a mistyped variable inside a WHERE clause produces valid SQL with the wrong meaning. Only tenant and schema are bound; check every {{…}} against those two names before committing a script.

Caution

A script’s ledger identity is the checksum of the unrendered body, so one file has one identity across every tenant. Changing only a binding — a different tenant’s schema name — does not change the checksum, so the script does not re-run. Changing the file body does. The ledger stores the rendered SQL, so the recorded statement and the idempotency key describe different strings.

Datastore-level init.scripts[] use the same object shape but run on the first engine build rather than inside a plan, and are documented with the datastore surface — see Init scripts. Their plan row is recorded with a risk of high, but nothing enforces that: the init runner applies its steps with the default confirmation, so the label is a record, not a gate.

Drift detection

Drift is the read-only question “did someone ALTER out of band, or did a migration fail silently?”. The engine handle exposes Drift(ctx, DriftOptions{…}), which discovers the live database, compares it to the composed definitions, and returns the diff without executing anything.

Option	Default	Effect
SystemTablePrefixes	data_, _pg_, _information_, pg_, information_	Tables excluded from the live scan
IncludeAuditTables	false	Whether _audit / _history side tables take part
AllowDrops	false	Whether unmatched live columns surface as ops instead of refusals

An empty diff means the live schema matches the definitions. A backend whose provider exposes no discovery implementation returns an error instead; all in-tree backends implement it.

data.svc has no drift verb. The operator equivalent is data plan generate, which performs the same discover-and-compare and executes no DDL — it differs only in that it persists a plan row you can then choose to apply or ignore.

After a definition change

Two independent things have to happen, in this order, and neither implies the other.

1. Migrate the database — data plan generate then data plan apply (or data bootstrap on a fresh tenant). Skip this and queries against the new shape fail with “relation does not exist” or “column does not exist”.
2. Rebuild the engine — the process caches a compiled schema, pool and hook chain per (tenant, datastore). Until it rebuilds, writes still use the old shape: a create succeeds and silently omits the new column. Restart the process, refresh the tenant config, or clear the cache in place.

curl -X DELETE http://127.0.0.1:8099/cache \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Customer: acme' -H 'X-Product: shop' -H 'X-Env: prod' -H 'X-Tenant: eu1'

Clearing the cache drops every cached schema, pool, engine and action registry for the tenant and the next request rebuilds from source. It rebuilds the engine; it does not touch the database.

CLI reference

Every verb boots the same configuration file as the server, resolves the tenant, and exits. It never opens the listener.

Command	Flags	Does
data bootstrap	--dry-run, --scope shared|superadmin	Create the namespace, entity tables, indexes and ledger on a fresh database
data plan generate	--allow-drops	Discover, diff, plan, persist. No DDL
data plan show <plan-id>	—	Render a persisted plan and its steps
data plan apply <plan-id>	--confirm-risk low|medium|high, --dry-run	Execute the plan under the datastore lock
data applied list	--plan-id	Read the step-level ledger
data init list	—	Declared init scripts and their applied state
data init run	—	Force an init-script pass now; ledger idempotency still applies
data seed run	--name, --env, --dry-run, --force, --triggered-by	Apply declared seeds through the engine’s own write path
data seed list	—	Declared seeds and their applied state
data locks list	--include-expired	Show held coordination locks
data locks release <resource>	—	Force-release a stuck lock

--tenant (required), --datastore (empty selects the schema default) and --output table|json are shared by every verb. data plan generate --source is registered but never read — the source is always the definitions.

Note

data plan resume does not exist, and no --resume-from flag is registered, despite a halted plan’s message suggesting one.

In --output json, Kind and Risk serialise as integers rather than names: 0 is ddl and low respectively.

HTTP reference

The admin surface mounts at /admin/data/* and the cross-tenant surface at /superadmin/*; behind the gateway both sit under /data/. Every route needs a bearer token and the four CPET headers — see the Data API reference for the request conventions.

Route	Body	Gate
POST /admin/data/plans	{"datastore": "", "allow_drops": false} — both optional	Generate
GET /admin/data/plans/:id	?datastore=	Read
POST /admin/data/plans/:id/apply	{"datastore": "", "confirm_max_risk": "low", "dry_run": false}	Apply
GET /admin/data/applied	?plan_id= required, ?datastore= optional	Read
POST /admin/data/seeds/runs	{"datastore","names","environment","dry_run","force"} — runs synchronously	Seed
GET /admin/data/seeds	—	Read
GET /admin/data/locks	?datastore=, ?prefix=, ?op_kind=, ?include_expired=	Read
DELETE /admin/data/locks/:resource	—	Locks
DELETE /cache	—	Token only

Generate returns {plan_id, source, max_risk, warnings?, refusals?, steps[]}. Apply returns 200 when the status is applied and 207 Multi-Status otherwise — including when a partial result accompanies an error — with {plan_id, status, steps_applied, steps_failed, steps_skipped, failed_step?, halted_step?, error?}. An invalid confirm_max_risk is a 400. Errors on this surface are a flat {"error": "<message>"}, not the typed envelope the data plane uses.

Request field names are snake_case. A camelCase key is not read: it silently leaves drops disallowed and the confirmed risk at low.

The superadmin mirror repeats the plan, applied, seed and lock routes under /superadmin/tenant/:tenant/…, plus GET /superadmin/tenant to list the tenants this process holds resources for and DELETE /superadmin/tenant/:tenant/cache. :tenant is the bare tenant name — customer, product and environment come from the caller’s own tenancy context, and a value containing the key separator is a 400. A control plane governs exactly its siblings; a cross-product target is unrepresentable.

Each operation class carries its own gate — read, generate, apply, seed, retention, locks, maintenance — so whether a tenant admin may execute DDL is a per-deployment decision. Any gate left unset resolves to “admin”, so a partial policy is never accidentally open, and /superadmin/* is superadmin-only regardless of the policy. The deployable service ships with generate, read and seed at admin, and apply, retention and lock release raised to superadmin.

Programmatic entry points

A host service reaches the pipeline through an admin provider it registers at boot; handler packages never import a host’s boot package. An unregistered host fails with a no-provider-registered error naming the missing boot call, rather than a nil dereference.

Entry point	Purpose
migrate.Generate(ctx, GenerateInputs, GenerateOptions)	The whole discover → compare → plan → persist sequence
migrate.Compare(definitions, live, DiffOptions)	Diff only. DiffOptions.Dialect is required for physical-type comparison
migrate.PlanFromDiff(diff, PlanOptions)	Lower a diff to SQL steps
migrate.RehydratePlan(planRecord, stepRecords)	Rebuild an in-memory plan from ledger rows
migrate.NewExecutor(db, ledger).Apply(ctx, plan, ApplyOptions)	Execute, with MaxRiskConfirmed, DryRun, HaltOnManualGate
migrate.NewSQLLedger(db).InSchema(ns)	The persisted plan and step ledger
migrate.NewSQLLockManager(db).InSchema(ns) + migrate.ResourceKey(tenant, datastore)	Coordination locks and their canonical key
migrate.EnsureNamespace(ctx, db, dialect)	CREATE SCHEMA IF NOT EXISTS, from the dialect. A no-op on schema-less backends
migrate.SystemEntities()	The six engine-owned entities, in emission order — bootstrap only
migrate.NewInitRunner(datastore, executor, ledger).Run(ctx)	The init-script pass
migrate.NewSeeder(schema, writer, ledger).Run(ctx, opts)	Seeds, written through the engine’s hook chain
migrate.RawScriptBindingsWithSchema(tenant, schema)	The {{tenant}} / {{schema}} bindings for raw scripts
Drift(ctx, DriftOptions{…}) on the engine handle	Read-only drift comparison
Migrate(ctx) / MigrateDetailed(ctx, onError) on the engine handle	The zero-config sweep described below

GenerateInputs carries the definitions, the discovery implementation, the dialect, the database handle, the ledger, the namespace and the raw-script bindings. GenerateOptions carries the source label, AllowDrops, the trigger labels and Persist.

Caution

Persist: false alone does not disable persistence — a non-empty Trigger or TriggeredBy turns it back on. Bootstrap only works because it passes Persist: false and leaves both label fields empty; writing to a plan table that does not exist yet would fail.

The zero-config sweep

The engine handle also exposes a blind, non-introspecting migrate: it emits CREATE … IF NOT EXISTS for every entity and reports per-object failures, continuing past each so one pass shows every problem. Emission order is the entity map’s iteration order, so it is not stable between runs. It has no plan, no review, no ledger and no rollback, and it is the only path that emits foreign-key constraints. Nothing in data.svc calls it — it exists for tests and embedded single-tenant use. Production DDL goes through the plan pipeline.