Operations

Cost accounting

Every request emits one accounting record, whether it hit an upstream or was served from cache:

{
  "request_id": "01JB7CY1Q6R0GK09RZAVDQGF11",
  "tenant_id": "acme",
  "product_id": "support-portal",
  "environment_id": "prod",
  "block": "support-bot",
  "purpose": "triage",
  "alias": "support.triage.v1",
  "prompt_version": "support/triage-system-v1",
  "upstream_provider": "openai_compatible",
  "upstream_model": "qwen2.5-7b-support-v2",
  "ab_variant": "v1",
  "fallback_attempts": 0,
  "cache_mode": "miss",
  "input_tokens": 4200,
  "output_tokens": 380,
  "cache_read_tokens": 0
}

Because the record carries tenant, product, environment, block, purpose, alias and prompt version, spend can be sliced any of those ways without instrumenting the calling application. “Which feature costs the most”, “what did this customer cost us this month” and “did prompt v4 get more expensive than v3” are all queries over the same table.

fallback_attempts is the index of the entry that actually served the request. A non-zero value on steady-state traffic means the primary is failing quietly — worth an alert, because the request still succeeded and nothing else will tell you.

Audit and replay

Every request and response is written to an audit trail with PII redaction applied on the way in. Retention is tiered:

Tier	Holds	Typical retention
Hot	Recent traffic, queryable interactively	7 days
Warm	Full records, queryable on demand	90 days
Cold	Archived, compliance retrieval	7 years

Records are replayable. Given a request ID the gateway can re-issue the exact upstream call — same alias version, same prompt version, same parameters — which is what makes “this answer was wrong three weeks ago” a debuggable claim rather than an anecdote. Replay is also how an eval suite re-scores historical traffic against a candidate prompt before it ships.

Streaming responses are recorded with their mid-stream events, so a replay of a streamed call reproduces the stream rather than just the final text.

Failure modes

An upstream is unavailable. The alias’s fallback chain absorbs it. Transient failures are retried with backoff before falling through; non-retryable ones fall through immediately. Watch fallback_attempts — a chain silently serving from entry 2 looks healthy from the caller’s side.

All upstreams in a chain are unavailable. The request fails with a structured error naming every attempt. There is no silent degradation to a lower-quality answer.

The gateway itself is unavailable. Model traffic stops platform-wide. This is the tradeoff of routing everything through one service, and it is why the gateway is deployed with more replicas than the blocks that depend on it. It holds no state that a restart loses — aliases and prompts reload from the registry, budget counters reconcile from durable storage.

A budget is exhausted. Requests are rejected with 429 and a structured error naming the scope that was hit. Nothing is sent upstream, so an exhausted budget costs nothing. If the soft threshold fired first, the warning events preceding it are in the audit trail.

Health and readiness

The gateway is ready once it has loaded the alias registry and reached at least one configured upstream. Useful checks:

GET /v1/models — lists resolvable aliases; an empty list means the registry did not load
GET /admin/v1/budgets — current consumption against every scope
Provider reachability — surfaced per upstream, since a single unreachable provider does not make the service unready when fallbacks exist

What to watch

Signal	Why it matters
`fallback_attempts > 0` rate	Primary upstream degrading, invisible to callers
Cache hit rate by alias	A cache mode mismatched to the traffic shape
Soft-budget warnings	Tenant about to start getting `429`s
p99 latency by upstream	One provider dragging a fallback chain
Rejected requests by scope	Which budget is actually binding

Cache hit rate is the one worth watching per alias rather than in aggregate. A platform-wide hit rate averages a long-stable-prompt alias hitting 90% with a high-variance one hitting nothing, and tells you about neither.