The chassis token middleware, before any handler ran
the literal string Unauthorized, no code
text/plain
The tenant middleware, before auth ran
the literal string invalid tenant, HTTP 400, no code
application/text
A plain-text Unauthorized on /rest or /admin is the single most common failure on the entity
surface and it carries no diagnostic at all — jump to
The token is rejected by another service.
engine_unavailable always reads could not reach datastore, whatever actually failed. The pool,
config and vault error text only exists in the server log.
The service will not start, or starts and serves nothing
400 + plain-text invalid tenant on every route except /ready and /health
The four CPET headers are missing, incomplete, or neither the full CPET join nor the bare X-Tenant value resolves to a configured tenant node
Re-send with all four headers
Send X-Customer, X-Product, X-Env, X-Tenant; add the node to the tenant hive or the config server
400 invalid tenant with all four headers set
The header is X-Env, not X-Environment
Print the request headers
Rename the header
500 engine_unavailable, could not reach datastore
The CPET passed the existence gate but no engine could be built: no datastore/dbpool for that CPET, an unreachable database, bad credentials, or a config refresh in flight
Server log at info names the failing step
Declare the CPET’s datastores and dbpools, provision it, fix the credential
500 engine_unavailable for a product you just added
An unknown product does not fail at the middleware. Product registration is a cheap empty bucket and a missing tenant section resolves as empty, so the first thing that actually fails is the pool build
Log shows the pool or datastore resolution failing, not a tenancy error
Add the tenant node for that full CPET. There is no product allow-list to edit
Log: no v2 pool factory for DBPool type "<t>"
The dbpool declares a backend other than PostgreSQL
Read the dbpool’s type
Only postgres has a pool factory here
Log: pool ping: … authentication failure with a correct-looking password
A dbpool password is tried against vault first and falls back to being used literally. With no vault client registered, a vault secret name is sent as the password
Check whether vault.url is set
Set vault.url, or inline the literal password
Log: no DStore config for "<tenant>":"<ds>"
The tenant node has no datastore by that name
—
Add the datastore, or send ?datastore=<name>
403 v2.no_tenant on /rest or /schema
The same missing-header condition the auth surface reports as 400 or 401
—
Send the CPET headers
A request for a CPET that does not exist still reaches the engine
The middleware’s existence gate is a two-key probe: the full CPET, then the bareX-Tenant value. A hive node named for the bare tenant satisfies the gate even when the full CPET has no node
Compare the node path against the full four-segment key
Key production nodes by the full CPET
Header resolution and the per-CPET scope model are described in Tenancy.
Login is refused for a credential you know is valid
Wrong password, no such identity, an inactive identity, or an identity row with no password hash. The four are deliberately indistinguishable, and the miss path runs a dummy argon2 verify so timing does not leak either
Read the identity row over /rest/users with an admin token
Reset the password, activate the account, or check that the row carries a hash
400 unknown_realm
The realm named in the body is not in this CPET’s composed behavior config
List the realms the tenant actually has
Fix the realm value, or deliver the product’s realm declarations to this CPET (meta.url / meta.localdir)
403 provider_not_allowed, realm <name> does not allow password login
The realm exists but does not list password among its providers
Read the realm’s providers list
Add password, or use the realm that has it
400 bad_identity_type
identity_type names a column the realm’s entity does not declare as a login identifier. The message lists the ones it does
—
Use a declared identifier, or declare the column on the entity
403 hook_denied with an unfamiliar message
A product auth hook aborted the flow. The message is hook-authored and is not a stable string
Search the product’s hook definitions for the message
Fix or remove the hook
429 login_abuse + Retry-After
The per-(tenant, caller IP) login bucket is exhausted (capacity 30, refill 0.5/s)
—
Back off. The limiter is in-memory per process, so N replicas give N× the limit, and there is no config key to tune it
409 session_limit
A managed CPET reached session.max_concurrent_sessions and its session.on_limit is reject
Read the tenant’s session: block
Set on_limit: evict_oldest, or raise the cap
503 policy_unavailable
The tenant’s session policy could not be resolved, and the service refuses to guess stateless vs managed
Log names the behavior-composition failure
Fix the behavior layer; a downgrade to standard would silently weaken a managed tenant
500 session_service_unavailable
The CPET is configured for managed sessions but no session service is wired into this process
—
Deployment wiring problem — it refuses loudly rather than issuing a stateless token
500 issue_failed, could not issue refresh token after an upgrade
The live refresh-token table predates the binary’s schema. The access token minted; the refresh insert hit a missing column
Compare the table against a generated plan
iam.svc data plan generate → show → apply per tenant. Refresh tokens issued before the migration 401 once
500 sign_failed
The tenant’s signing key could not sign the claim set
Log carries the real error
Bad bring-your-own PEM, a missing signing-key table, or an unreachable vault
500 user_query_failed, lookup failed
The identity lookup query itself errored — distinct from returning no rows
—
Missing identity table or an engine fault; run the migration
503 superadmin_not_deployed on a login with realm: superadmin
The control plane for this product and environment is not deployed
—
Add the <customer>:<product>:<environment>:superadmin node and bootstrap it
Only /auth/login returns a named 429 code. Every other rate-limited route — MFA verify, password
reset, magic link, POST /auth/token, API-key issue — returns the limiter’s own 429 body with no
code. POST /superadmin/impersonate is wrapped under a profile the default limiter does not
declare; do not assume it is limited.
The chassis token middleware rejected the credential before the handler. Signature failure, unknown key id, no configured issuer URL, a JWKS fetch timeout, or a tenant claim that disagrees with the request headers — all collapse to this one body
Read the receiving service’s log
Work through the rows below
Log: jwks: no configured URL for issuer "iam"
The relying service has no jwks.issuers.iam entry, or it was nested under hives:
jwks.issuers must be top level in the boot document, a sibling of host and port
Set it to the externally reachable base URL of iam.svc
A token iam.svc just minted 401s on its own/rest or /admin
The same rule applies to itself — its entity surface goes through the same JWKS path
Same log line
iam.svc must set jwks.issuers.iam to its own reachable URL
401 tenant_mismatch
The token’s tenant claim names a different CPET than the request’s headers. Headers are authoritative and name the resource; the claim names the caller
Decode the claim and compare against X-Tenant
Re-authenticate against the target tenant. There is no superadmin exception — a control-plane token is refused on a tenant surface exactly like any other mismatch
401 token_expired on a token that looks fresh
exp/nbf places it outside its window — or the token carries no exp at all, which is treated as forgery
Decode the claims
Mint a new token; do not strip exp
401 auth_required where you expected invalid_token
The Authorization header did not match Bearer or ApiKey — both are prefix-matched, case-sensitive, single space. An unrecognised scheme is treated as no credential and passes the middleware with no principal
Byte-compare the header
Fix the scheme and spacing
A rotated or revoked key keeps verifying during an issuer outage
JWKS resolution prefers availability: cached for 5 minutes, background-refreshed to 24 hours, and past 24 hours a failed refetch still honours the cached key
—
Treat key rotation as eventually consistent; revoke sessions, do not rely on key removal
A receiving service accepts expired tokens indefinitely
It called the signature-only verification entry point, which does not check exp/nbf, and never added its own timeliness check
Read the relying service’s verification call
Use the entry point that validates temporal claims, or enforce exp yourself
Log banner: SECURITY: jwks.disabled=true — this service is accepting tokens WITHOUT signature verification
The signature bypass is on. Every identity claim is attacker-controlled; only header-authoritative tenancy still holds
grep jwks .iam.yaml
Remove jwks.disabled outside development
GET /.well-known/jwks.json returns the wrong keys
The key set is per tenant and requires the CPET headers. Tenant A’s set never contains tenant B’s keys
This CPET uses managed server-side sessions; refresh tokens do not exist for it. Note the status — SDKs that treat 403 as permanently forbidden will not fall back
Read the tenant’s session.preset
Exchange the opaque reference at POST /internal/session/exchange instead
401 invalid_session, session invalid or expired at exchange
Unknown, expired (idle or absolute) or revoked — deliberately one answer for all three, so it is not an oracle
Check session.idle_timeout_s and session.absolute_expiry_s
Log in again
404 unknown_session on POST /internal/session/rotate
No managed session with that control-plane id. Rotate returns 404 where exchange returns 401 for the same condition
—
The id used by revoke, rotate and list is the control-plane row id, never the opaque session_id credential
A browser never sends the session cookie
The kis_session cookie is set HttpOnly, Secure, SameSite=Lax. Over plain HTTP a browser drops it
Watch the Set-Cookie header
Serve over TLS in development, or read session_id from the JSON body
Login on a managed CPET still returned token and refresh_token
The managed branch is wired into password login only. MFA verify, magic link, WebAuthn, OAuth and refresh still mint stateless tokens on a managed CPET
Compare the login response shape across flows
Route managed tenants through password login until the other flows carry the branch
Tokens minted under preset: fedramp cannot be verified anywhere
The preset parses and selects the v3 crypto suite, but v3 verification is not wired. Every token this service mints is PASETO v4.public (Ed25519) regardless of crypto_suite
—
Use standard or hipaa
Session models, timeouts and revocation live in Sessions.
503 webauthn_disabled, WebAuthn not configured, on all four /auth/webauthn/* routes
The WebAuthn implementation is never enabled in the deployable binary. There is no configuration key that turns it on
None. Do not build a passkey flow against this deployment
404 unknown_provider, OAuth provider not configured, on /auth/oauth/:provider/start and /callback
No OAuth provider is registered in the deployable binary, so every provider name is unknown. Declaring type: oauth in a provider declaration wires nothing — a provider declaration carries only name, type and template, no credentials, and nothing reads type or template at request time
None. The route answers 404 for every name
You cannot find an OTP login endpoint
There is no OTP provider — no SMS, email or WhatsApp one-time-code login. A user_otp table exists in the schema and no code reads it
404 v2.entity_not_found for an entity you declared
The entity is not in this CPET’s composed schema — the product or tenant layer that declares it was not delivered, or a feature gate removed it
GET /schema/entities for that CPET
Deliver the layer; check realmconfig.enable.agents/bots/delegations — setting one false removes the entity from the schema entirely
404 on /admin/realm or /admin/tenant
Only two admin aliases are mounted: /admin/user and /admin/role. Realms are configuration, not rows, so the realm entity was removed
—
Use /rest/<entity> or the control-plane routes
404 on /schema/<entity>
The path is /schema/entities/:entity
—
Correct the path
A raw database or engine error in the error field
The entity surface returns engine error text verbatim and classifies by status: 403 when an access rule denied, 422 on validation, 404 when the row is absent, 500 otherwise
The status carries the classification, not the code
Read the status first; do not parse the message
An entity is readable with no credential at all
The full anonymous mirror /anon/rest/* is mounted. Those requests reach the engine with no user id and no roles — only each entity’s own access: rules stop them
curl the entity under /anon/rest/ with no Authorization header
Give every extension entity explicit access: rules
A non-admin can write through /admin/user or /admin/role
Those mounts declare an admin rule, but no route authorizer is registered in this service, so the route-level check never fires. Access control there rests entirely on the users and role entities’ own access rules
403 v2.requires_superadmin on plan apply, seed run, lock release
Working as intended. Apply, Seed, Retention, Locks and Maintenance are raised to superadmin on this service because the DDL touches signing-key and credential tables. Read and plan-generate stay at admin
—
Use a control-plane token, or run the CLI out of band
schema "…" does not exist after a bootstrap
data bootstrap creates the PostgreSQL schema, never the database
Connect to the database directly
Create the database out of band; it is infrastructure, not this service’s job
Tables landed in a schema you did not expect
When a dbpool omits schema:, the physical schema is the tenant slot name, not the service name
Read the dbpool declaration
Declare schema: explicitly
A plan refuses to apply
The plan’s maximum risk tier exceeds --confirm-risk (default low)
iam.svc data plan show <plan-id>
Re-run with --confirm-risk medium or high after reading the ops
invalid risk level "…" (use low / medium / high)
Typo in --confirm-risk
—
Use one of the three words
A migration hangs behind a lock
A previous run died holding a coordination lock
GET /admin/data/locks
Force-release with DELETE /admin/data/locks/:resource (superadmin)
/admin/data/retention/* and /admin/data/materialize/* always fail
The routes are mounted by the shared admin router but no provider backs them here
—
Not supported on this service
One tenant migrated, the rest did not
Every data verb takes exactly one --tenant. There is no all-tenants pass
—
Loop over your tenant inventory, and migrate the control plane separately with --tenant <customer>:<product>:<environment>:superadmin
config not ready: <why> from a CLI command
The config client did not finish its first load inside 15 seconds
Pass all four segments. Without --tenant the CLI targets default:iam:dev:default
Help text tells you to run iam migrate first
There is no such command
—
The verb is iam.svc data bootstrap
data seed run does nothing
Two different seed verbs exist: iam.svc seed creates privileged identities, iam.svc data seed run applies schema-declared reference data — of which this service declares none
Three causes, one answer: bootstrap.token is unset (the endpoint is disabled entirely), the token did not match, or a superadmin already exists
Check whether bootstrap.token is configured
Set bootstrap.token, or create the first operator with iam.svc seed
503 superadmin_not_deployed
The control plane for this product and environment has no hive binding, or its schema was never bootstrapped. The plane is per customer, product and environment — not one per deployment
Look for a node whose tenant slot is superadmin
Add the node, then iam.svc data bootstrap --tenant <customer>:<product>:<environment>:superadmin
400 reserved_or_invalid_name registering a tenant
The slug is empty, is not a valid four-segment key, or uses superadmin or shared in any segment
—
Those two names are reserved everywhere, in every deployment
403 not_an_operator on impersonate
A tenant identity, or an operator of another product’s plane, called a control-plane operation
—
Use the right plane’s operator token
403 role_required
The operator holds neither impersonate-readonly nor impersonate. Neither is implied by any other role
Read the operator’s roles
Grant the role explicitly; read-write requires the stronger one
403 not_granted, you do not manage that tenant
No grant over the target tenant — or the tenant does not exist, deliberately conflated so the endpoint cannot enumerate tenants
—
Add the grant, and verify the tenant separately
403 impersonation_not_allowed
The target user has not opted in. An operator cannot write that field, so cannot self-authorise
Read the user row
The user or their tenant admin must opt in
400 reason_required
reason is mandatory — the audit record cannot be reconstructed later
—
Supply it
GET /superadmin/tenant is missing tenants you configured
It lists tenants touched in this process, so the answer differs per replica
Compare against the config hive
The hive or config server is the authoritative inventory
In-flight requests are not drained — only the listener closes. Drain connections upstream before stopping an instance
ready:true while every request 500s
Both probes are process-liveness flags. Neither consults the database, the config client or any tenant
Password reset and magic link return 200 but nothing is delivered
notify.url is unset, so the sink is a no-op and codes are dropped. The endpoints still succeed; the gap appears as sink_error in the flow’s after-events
compliances: encrypted fields are readable plaintext in the database
Field encryption is installed only when a vault client is registered. With vault.url empty the hooks are simply absent — no error. Turning it on later needs a data migration; existing plaintext rows stay plaintext until rewritten
No metrics and no traces despite the config
Telemetry is never initialised on this boot path. telemetry.enabled and enablecache are inert keys in the shipped profiles, and the metric names in the source are recorded by nothing. Observability here is structured logs
A JWKS response or a credential list looks truncated
Key sets cap at 100 entries and WebAuthn credentials at 50, silently
A 403 after a config change took a whole tenant offline
A malformed behavior file, a malformed tenant overlay row, or a reserved claim name under token.claims/token.claims_from refuses the engine build for that CPET. Reserved names: iss, sub, tenant, realm, roles, iat, nbf, exp, type, scope, act
A revoked user still has a working access token
Stateless tokens are not revocable — they live out their TTL (15 minutes by default). Only managed sessions revoke instantly
