AI Gateway Operations

Cost accounting

Every request emits one accounting record, whether it hit an upstream or was served from cache:

{
  "request_id": "01JB7CY1Q6R0GK09RZAVDQGF11",
  "tenant_id": "acme:prod:support-portal:main",
  "product_id": "support-portal",
  "environment_id": "prod",
  "customer_id": "acme",
  "block": "support-bot",
  "purpose": "triage",
  "prompt": "support-triage-system",
  "prompt_version": "3f9a1c0d2b7e",
  "api_dialect": "openai",
  "alias": "support.triage.v1",
  "upstream_provider": "vllm_internal",
  "upstream_model": "qwen2.5-7b-support-v2",
  "fallback_attempts": 0,
  "cache_mode": "miss",
  "input_tokens": 4200,
  "output_tokens": 380,
  "cache_read_tokens": 0
}

tenant_id is the tenant’s full key, customer:env:product:tenant, and customer_id, environment_id and product_id are its parts. prompt is the prompt rendered into the request, and prompt_version the stamp of the folder it was read from (the product’s ai/prompts, or the platform’s prompt directory): a digest of the folder’s files, so two records with the same stamp ran the same prompt text, and the product’s git history says which commit it was. ab_variant names the variant when an A/B split chose one.

Because the record carries tenant, product, environment, block, purpose, alias and prompt version, spend can be sliced any of those ways without instrumenting the calling application. “Which feature costs the most”, “what did this customer cost us this month” and “did the prompt get more expensive after last week’s change” are all queries over the same table.

With accounting.file set, records are appended to that JSONL file; unset, they go to the service log.

fallback_attempts is the index of the entry that actually served the request, a non-zero value on steady-state traffic means the primary is failing quietly, worth an alert, because the request still succeeded and nothing else will tell you.

Audit and replay

Every request and response is written to an audit trail with PII redaction applied on the way in. Retention is tiered:

Tier	Holds	Typical retention
Hot	Recent traffic, queryable interactively	7 days
Warm	Full records, queryable on demand	90 days
Cold	Archived, compliance retrieval	7 years

Records are replayable. Given a request ID the gateway can re-issue the exact upstream call, same alias version, same prompt version, same parameters, which is what makes “this answer was wrong three weeks ago” a debuggable claim rather than an anecdote. Replay is also how an eval suite re-scores historical traffic against a candidate prompt before it ships.

Streaming responses are recorded with their mid-stream events, so a replay of a streamed call reproduces the stream rather than just the final text.

An audit entry is read with GET /admin/v1/audit/{request_id}: the request’s tenant’s entries only, to its administrator (see API). Entries hold complete prompts and completions, which is why an ordinary user of the tenant cannot read them.

Failure modes

An upstream is unavailable. The alias’s fallback chain absorbs it. Transient failures are retried with backoff before falling through; non-retryable ones fall through immediately. Watch fallback_attempts, a chain silently serving from entry 2 looks healthy from the caller’s side.

All upstreams in a chain are unavailable. The request fails with a structured error naming every attempt. There is no silent degradation to a lower-quality answer.

The gateway itself is unavailable. Model traffic stops platform-wide. This is the tradeoff of routing everything through one service, and it is why the gateway is deployed with more replicas than the blocks that depend on it. It holds no state that a restart loses: the platform’s aliases and prompts reload from the operator’s directories and each tenant’s from its product, and budget counters reconcile from durable storage.

A product file is broken. A file in a tenant’s ai/models or ai/prompts that does not parse or validate is refused: the gateway logs the tenant, the file and the problem, and the tenant keeps serving the aliases and prompts it had. Other tenants are unaffected. Fix the file and the next change loads.

A tenant’s product has not loaded. Until the first load of a listed tenant succeeds, its calls are answered 503 with ai-gateway-tenant-not-loaded and Retry-After, and the gateway retries the load. Other tenants are served meanwhile.

A tenant is not listed. A tenant the gateway’s configuration does not list is refused with 400 and ai-gateway-tenant-not-served; add it to the configuration’s tenants:.

A budget is exhausted. Requests are rejected with 429 and a structured error naming the scope that was hit. Nothing is sent upstream, so an exhausted budget costs nothing. If the soft threshold fired first, the warning events preceding it are in the audit trail.

Health and readiness

The gateway boots once its configuration and the platform’s directories have loaded: an ai-gateway: key it does not read, or a configured registry.path or prompts.path that does not load, fails boot and names the problem. One tenant’s product that does not load does not make the gateway unready: the others keep being served.

Each tenant load logs a line naming the tenant, its prompt, alias and provider counts and the stamp it loaded (ai: tenant <key> serves …), or the file that failed. Useful checks, with a tenant’s headers and its administrator’s token:

• GET /v1/models: the aliases the tenant resolves, the platform’s and its product’s
• GET /admin/v1/stats: the stamp of the tenant’s loaded product, and its alias, prompt and provider counts
• GET /admin/v1/budgets/{tenant}: the tenant’s configured ceilings

What to watch

Signal	Why it matters
fallback_attempts > 0 rate	Primary upstream degrading, invisible to callers
Cache hit rate by alias	A cache mode mismatched to the traffic shape
Soft-budget warnings	Tenant about to start getting 429s
p99 latency by upstream	One provider dragging a fallback chain
Rejected requests by scope	Which budget is actually binding
Tenant load failures in the log	A product change the gateway refused; the tenant is serving its previous files

Cache hit rate is the one worth watching per alias rather than in aggregate, a platform-wide hit rate averages a long-stable-prompt alias hitting 90% with a high-variance one hitting nothing, and tells you about neither.