Tenancy and isolation

One data.svc process serves every tenant of every product that reaches it. Nothing per-tenant exists at boot: the schema, the connection pool, the physical namespace, the script runtime and the engine are all built on the first request that names a given (tenant, datastore) pair, cached, and torn down as one unit.

The CPET key

Every request carries four headers. The chassis tenant middleware reads them, joins them with : into one key, and gates on tenant existence before any handler runs.

Header	Segment	Example
X-Customer	customer	acme
X-Product	product	erp
X-Env	environment	prod
X-Tenant	tenant — the segment only, never the joined key	eu1

X-Customer: acme  +  X-Product: erp  +  X-Env: prod  +  X-Tenant: eu1
        └── tenant key  acme:erp:prod:eu1

All four must be present and non-empty. Any one missing is 400 invalid tenant — the body is that fixed string for every cause; only the log line names which segment was absent. Just /data/health and /data/ready bypass the middleware; the /data/anon/* mirrors do not, because /anon drops the token, not the tenant identity.

The key is what selects the tenant’s definitions, its connection pool, its physical PostgreSQL namespace and its engine. It is also the value row-level security reads as user.tenant_id — the full four-part string, not the tenant slug. See Row-level security.

Caution

A verified token’s tenant claim can never override the headers. If the claim disagrees with the header CPET the request is rejected 401, with the body Unauthorized — the reason token tenant does not match request tenant and both tenant keys appear only in the log. There is no superadmin exception — an operator reaches tenant data through impersonation, whose token already names the target user. The headers name the resource; the token says who you are.

Tenant lookup and the bare-name fallback

The existence gate tries the full joined key first, then falls back to the bare X-Tenant value. The fallback exists for config sources keyed by short names — the local-development convention where one tenant binding at the root path answers for any key. Production bindings should be keyed by the full customer:product:env:tenant.

Note

GET /data/checktenant resolves the full four-part key only — it has no bare-name fallback. A deployment that relies on the fallback gets 400 from the probe while ordinary requests succeed. Fix the binding path rather than the probe.

How a request resolves to a datastore

The datastore is chosen by ?datastore=<name>, and an absent or empty value resolves to the schema’s default-datastore. From there:

Step	Resolved from	Fails with
1. Tenant key	The four headers, joined	400 invalid tenant
2. Product bucket	X-Product; registered on first sighting	no manager registered for product "<p>" → 500
3. Tenant scope	The tenancy manager for that product	resolve tenant scope → 500
4. Definitions	The tenant’s schema source, then the layer fold	build schema → 500
5. Datastore declaration	The named datastore in the composed schema, else its default	schema has no datastore "<n>" (default=…, available=…)
6. Backend type	The schema’s type: when set, otherwise inferred from the bound pool	—
7. Physical namespace	The pool’s schema:, else the tenant’s own name	isolation: shared with no namespace
8. Connection pool	The tenant’s dbpools.<name> entry	no v2 pool factory for DBPool type "<t>", pool ping: …
9. Tenant overlay	The tenant_extensions table, best-effort	—
10. Engine	Compiler, security stack, hook chain, script runtime, registries	build maya, backend.CompilerForDatastore → 500

Steps 4 through 10 run once per (tenant, datastore) pair. Every failure in them surfaces as 500 with code v2.engine_unavailable; the wrapped message names the failing step. The pool and datastore keys themselves are documented on Datastores and database pools.

Caching and invalidation

Each tenant owns a keyed resource scope with one sub-key per datastore. Concurrent first-touches for the same key collapse to a single build behind a per-key lock.

Cached per (tenant, datastore)	Cached per process
Composed schema and its parse	Rate limiter and its buckets
Connection pool (or the shared-endpoint router)	Scripted-endpoint response cache
Script runtime	Registered API providers
Hook chain and engine	—
Scripted-endpoint and action registries	—
Materialized-refresh runner (only when the schema declares cross-entity triggers)	—

The first request for a pair pays for all of it — definition fetch, parse, layer fold, schema compile, pool open, and any datastore init scripts.

Caution

The chassis per-request context timeout is timeouts.defaultcontexttimeout, and it defaults to 1 second. A cold build routinely exceeds that, and the request dies with context-middleware-timed-out while the build itself continues. Raise it — the verified test harness uses 30s. A malformed duration silently keeps the previous value.

loadtenants: takes a list of tenant keys to pre-resolve at boot so their configuration is resident before the first request. It warms configuration only, not engines or pools, and each failure is logged and skipped rather than failing boot.

When the cache is dropped

Trigger	Effect
Tenant configuration change	Drains the tenant’s whole scope; the meta client and config monitor are re-ensured
Tenant removal	Drains the scope, stops the config monitor, sweeps orphaned schema and pool entries
DELETE /data/cache	Drains the calling tenant’s scope
DELETE /data/superadmin/tenant/<tenant>/cache	Drains a sibling tenant’s scope; superadmin only
Process shutdown	Drains every tenant, then the scope-plane engines

A drain is all-or-nothing for the tenant. On a configuration change the service does not work out which datastore changed — it drops every datastore that tenant holds. Rebuild cost is paid lazily, and only for the datastores that are touched again.

Teardown order inside a scope is fixed: drain the engine (bounded by the tenancy drain timeout) → cancel the materialized-refresh runner → close the script runtime → close the pool (refcount-safe on a shared endpoint) → drop the parallel pool-registry entry → clear the schema parse cache and the init-once gate.

Caution

Tearing down a materialized-refresh runner cancels its loop and then waits for the goroutine, with no timeout — losing in-flight refreshes was judged worse than a slow shutdown. A hung shutdown is that queue, first suspect.

Physical isolation

Isolation is delivered by three mechanisms that must all agree: every compiled statement is qualified with the tenant’s PostgreSQL schema, every connection checkout sets app.tenant_id — and, under isolation: shared, also flips search_path — and row-level security predicates run on top. The first is the guarantee; the other two are defence in depth.

Three topologies are reachable from tenant configuration.

Topology	dbpools.<name> per tenant	Requires
Database per tenant	Distinct dbserver: and/or database:; isolation: omitted	Nothing further — each tenant gets a private pool. schema: optional
Schema per tenant, one database, one pool per tenant	Same dbserver:/database:, distinct schema:; isolation: omitted	A distinct schema: per tenant
Schema per tenant, one database, one shared pool	Same dbserver:/database:/dbuser:, distinct schema:, isolation: shared	A distinct schema: per tenant, one database role for all of them, and discrete connection fields — not dsn:

The default is dedicated: one private pool per (CPET, datastore), pinged once after creation so credential and network errors surface at build time. shared puts many tenants on one reference-counted pool per physical endpoint (<backend>://<host>:<port>/<database>), with a per-checkout namespace flip; there is no probe ping, so those errors surface on the first query instead. The engine, schema, hook chain and script runtime stay per tenant in both modes — only the physical pool is shared.

Caution

isolation: shared requires a per-tenant namespace. If none resolves, the engine build fails with DBPool declares isolation: shared but no schema: property — shared isolation requires a per-tenant PG schema, and every request for that tenant returns 500 v2.engine_unavailable.

With schema: omitted the namespace falls back to the tenant’s own name, which is usually right but ties the physical namespace to the tenant name — a rename moves it and orphans the tables. Declare schema: explicitly on any pool you intend to share.

Caution

Declaring dbpools: once at a shared prefix config node and letting tenants inherit it gives every tenant the same explicit schema: — one shared set of tables, with no error anywhere. Restate schema: at each tenant leaf.

Caution

A DSN-only pool cannot produce an endpoint key, so isolation: shared on it silently falls back to a dedicated pool. Nothing warns. A shared-endpoint deployment must use the discrete dbserver:/dbuser:/database: fields.

Two consequences of the endpoint key excluding schema and user: all tenants on a shared endpoint connect as the same database role, and the base pool is sized by the first tenant’s maxconnections. Size a shared endpoint for the whole fleet. Pool mechanics, session reset and the per-checkout statements are covered on Datastores and database pools.

Caution

On a shared pool the per-checkout flip does not cover everything. The migration ledger, lock manager, retention eraser, archive handoff queue, materialized-refresh task table and init-script runner all run raw SQL over the pool’s base connection, which never passes through checkout. Each is explicitly namespace-qualified for that reason. Qualification, not the flip, is what keeps shared tenants apart.

Isolation planes

An entity’s scope: decides which engine and which database its rows live in — tenant (the default), shared, or superadmin. A composed schema is split per plane before any engine is built, so a tenant query physically cannot name a shared or superadmin entity: it is absent from that engine’s schema. Cross-plane relations are dropped. The declaration side is on Entities.

Operationally, a plane engine is a peer of the tenant engines, not a tenant: one per (product, plane, datastore), built lazily over the plane’s own reserved configuration binding, resolved under a tenant key whose four segments are all the plane name (shared:shared:shared:shared). The tenancy drain does not reach them, so they are torn down separately at shutdown.

Caution

If an entity declares a plane whose reserved binding is absent, the build fails with scope is not deployed (no reserved hive entry). This is deliberate: the service never serves a scoped entity from the wrong plane, so the failure is loud rather than a silent fallback to the tenant’s own database.

Cross-tenant operations

/data/superadmin/tenant/* mirrors the whole admin surface with the tenant taken from the path instead of the token. The :tenant segment is the bare tenant slot; customer, product and environment come from the caller’s own headers, and the target is composed as caller-CPET plus the URL tenant. A value containing : is rejected 400 v2.invalid_tenant. A cross-product or cross-environment target is therefore unrepresentable: a superadmin plane governs only its sibling tenants inside its own product and environment.

Route	Purpose
GET /data/superadmin/tenant	Tenant keys this process holds live resources for
DELETE /data/superadmin/tenant/<tenant>/cache	Drain a sibling tenant’s scope
POST /data/superadmin/tenant/<tenant>/plans	Generate a migration plan for a sibling — the onboarding path
POST /data/superadmin/tenant/<tenant>/plans/<id>/apply	Apply it

Because this host runs shared endpoint pools, three admin operation classes are moved off the default admin gate onto superadmin: plan apply (the only route that executes DDL), retention (deletes, anonymises and archives tenant data, plus the ETL handoff), and lock force-release. Read, generate, seed and maintenance stay on admin. A tenant admin can inspect a plan and generate a diff; only a platform operator can execute one. Other hosts of the same admin surface keep everything on admin — this is a per-host posture.

Seeds, plan apply, retention and init scripts all take the same per-(tenant, datastore) lock, so two replicas cannot race them against one datastore.

What a caller sees

Situation	Result
A header missing or empty	400 invalid tenant — before any handler
Tenant not in the configuration source	400 invalid tenant
Configuration not yet loaded	400 invalid tenant
Unknown product	400 invalid tenant — the product is part of the joined key the existence gate resolves, unless the bare-name fallback matches on X-Tenant alone
Product known, tenant scope unresolvable	500 v2.engine_unavailable, log reason no manager registered for product "<p>"
Datastore name not in the composed schema	500 v2.engine_unavailable, schema has no datastore "<n>" (default=…, available=…)
Pool unreachable or bad credentials (dedicated)	500 v2.engine_unavailable, pool ping: …
Pool unreachable or bad credentials (shared)	500 on the first query, not at build
Token minted for another tenant	401 Unauthorized, log reason token tenant does not match request tenant
Cold build slower than the request timeout	context-middleware-timed-out

A product name the process has never seen is registered on first sighting as an empty bucket — no pool, no schema, no script runtime. Nothing is built until a real tenant scope resolves, which needs a resolvable tenant configuration, so a spoofed X-Product costs at most an idle map entry. This lazy registration is what lets one process host many products; without it, every product except a statically registered one returns 500 on its first request.

Checking a tenant is servable

/data/ready and /data/health are both flipped true at boot — after routes register, before the listener is even created, and before any tenant schema, pool or engine exists. A green /data/ready says the process is listening. It says nothing about whether any tenant can be served.

curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/data/ready

curl -s https://api.example.com/data/checktenant -H 'X-Customer: acme' -H 'X-Product: erp' -H 'X-Env: prod' -H 'X-Tenant: eu1'

/data/checktenant returns {"success":true} when that CPET resolves to a loaded tenant configuration, and 400 otherwise. It confirms configuration, not the database: it does not build an engine, open a pool, or run a query.

Caution

Two per-tenant failures are invisible to every probe. A datastore init script that fails is logged at error level and the engine still serves traffic. A materialized-refresh runner that fails to install is logged and the engine continues — mutations succeed, refresh tasks are never enqueued, and cross-entity computed fields go permanently stale with no request-level error. Check the logs, data init list, and GET /data/admin/data/materialize/pending.

The full endpoint list, headers and response envelope are in the Data API reference.