AI Gateway Core Concepts

Five ideas carry most of the AI Gateway: aliases, prompts, caches, fallback and budgets. Everything else is plumbing around them.

Aliases

An alias is a logical model name bound to a concrete upstream, its parameters, a prompt, a fallback chain, a cache policy and a budget. Your code names the alias; the alias names everything else.

A product keeps its aliases in its ai/models folder:

kind: model-alias
metadata:
  name: support.classifier.v3
  version: 3
spec:
  upstream:
    provider: anthropic
    model: claude-haiku-4-5
    parameters:
      max_tokens: 8000
      temperature: 0.3
  prompt:
    registry_ref: support-classify-system
  fallback:
    - provider: anthropic
      model: claude-sonnet-4-6
    - provider: vllm_internal
      model: qwen2.5-7b-support-v2
  cache:
    enabled: true
    mode: prefix
    ttl: 24h
  budget:
    cost_ceiling_usd_per_tenant_per_day: 50
    rate_limit_rpm: 1000

provider names one of the gateway’s upstream providers, or one of the tenant’s own (see Where definitions come from). A product alias names providers only: it does not say where the gateway connects or which key it presents.

Alias files are versioned by the product’s git history. The gateway watches ai/models, so a merged change is served without restarting anything. See Configuration for every field.

Prompts

Prompts are separate files from aliases, kept in the product’s ai/prompts folder and referenced by name:

prompts:
  - name: support-classify-system
    description: Intent classifier for inbound support messages.
    tags: [support]
    system-prompt: |
      You classify inbound support messages into the intent taxonomy below.
      Answer with the intent name only.

system-prompt and user-prompt are Liquid templates; few-shot prompts add worked samples. A call names a prompt in its body ("prompt": "<name>", with "parameters": {...}), or runs the prompt its alias names with spec.prompt.registry_ref.

Keeping prompts out of application code is what makes prompt changes reviewable, A/B testable and attributable. Every accounting record carries the prompt’s name and a stamp of the folder version it was rendered from, so a quality regression can be traced to the exact revision that caused it. See Authoring Prompts for the format and how a prompt is rendered into the request.

Where definitions come from

A tenant sees aliases, prompts and providers from three places:

Layer	Holds	Kept in
Platform	The platform’s aliases and prompts	The operator’s directories, `ai-gateway.registry.path` and `ai-gateway.prompts.path`
Product	The tenant’s aliases and prompts	The tenant’s product: `ai/models` and `ai/prompts`, read through meta.svc
Tenant configuration	The tenant’s own upstream providers	The tenant’s configuration: `ai-gateway.upstream_providers`

Platform names are reserved. A name resolves to the platform’s alias or prompt first, then to the product’s. A product alias or prompt that takes a platform name is not served (the platform’s is), and every load of that product logs a warning naming the file.

A tenant’s own providers replace the gateway’s of the same name, for that tenant only. A tenant whose traffic must stay in a region, or that brings its own provider account, gets a provider entry in its configuration under the name the aliases already use. Platform and product aliases then reach that tenant’s endpoint without changing an alias file.

Each tenant sees its own product only. Two products may each declare an alias or a prompt of the same name; each tenant runs its own, and neither can name the other’s.

Caching

Three layers, checked in order. Each is cheaper than the call it avoids.

Layer	Matches on	Typical use
Exact	Byte-identical request	Repeated identical prompts
Semantic	Embedding similarity above a threshold	Paraphrases of the same question
Prefix	Shared leading tokens, provider-side	Long, stable system prompts

Cache mode is set per alias. Cache entries are partitioned by the tenant’s full key, so one tenant is never served another’s cached answer. Every accounting record reports which layer served the request (miss, exact, semantic or prefix), so hit rates are measurable per alias rather than guessed at.

Fallback chains

Each alias carries an ordered list of upstreams. When one fails the gateway moves to the next, and the failure class decides whether it retries first:

Transient (timeout, 503, upstream rate-limit): retry the same upstream with backoff, then fall through
Non-retryable (400, 401, 403): fall through immediately
Content-filter rejection: fall through immediately, since a different model may not refuse

Every attempt is recorded with provider, model, latency and error, so a chain that is quietly serving most of its traffic from the second entry is visible rather than invisible.

A/B routing

An alias can split traffic across variants:

ab_routing:
  - weight: 90
    ref: support.classifier.v3
  - weight: 10
    ref: support.classifier.v4-experimental

Each ref is an alias in the tenant’s view. Routing is sticky per request-ID hash, so the same input always lands on the same variant and results stay reproducible. Two variants that name different prompts with spec.prompt.registry_ref A/B test the prompts.

A/B routing and fallback are deliberately different mechanisms: A/B is an intentional split for measurement, fallback is an unintended diversion caused by failure. They compose, A/B picks the variant, and that variant’s own fallback chain handles its failures. Accounting records carry the variant tag so cost and quality can be compared directly.

Budgets

Budgets are enforced at three scopes, all before the upstream call, and counted per tenant (its full key):

Per-alias: this alias may spend $50/day per tenant at 1000 RPM
Per-tenant-per-block: this tenant’s bot may spend $200/day across every alias it uses
Per-tenant globally: this tenant may spend $5000/day across the whole platform

Each has a soft and a hard threshold, defaulting to 80% and 100%. Crossing the soft threshold still serves the request but emits a warning event; crossing the hard one rejects with 429. The gap is deliberate. It gives a tenant time to react before anything breaks.

Priority lanes

When a tenant nears its ceiling, the gateway sheds cheap traffic first. X-Kis-Priority takes critical (served until the hard budget), normal (the default), or background (deprioritised first, batch distillation, backfills and similar).

Authoring Prompts: the prompt format, calling a prompt, rendering
Configuration: the service configuration, tenants, aliases
Operations: accounting, audit and replay