Core Concepts

Five ideas carry most of the AI Gateway: aliases, prompts, caches, fallback and budgets. Everything else is plumbing around them.

Aliases

An alias is a logical model name bound to a concrete upstream, its parameters, a prompt, a fallback chain, a cache policy and a budget. Your code names the alias; the alias names everything else.

kind: model-alias
metadata:
  name: support.classifier.v3
  tenant: acme
  version: 3
spec:
  upstream:
    provider: anthropic
    model: claude-haiku-4-5
    parameters:
      max_tokens: 8000
      temperature: 0.3
  prompt:
    registry_ref: support/classify-system-v3
  fallback:
    - provider: anthropic
      model: claude-sonnet-4-6
    - provider: openai_compatible
      endpoint: vllm-internal
      model: qwen2.5-7b-support-v2
  cache:
    enabled: true
    mode: prefix
    ttl: 24h
  budget:
    cost_ceiling_usd_per_tenant_per_day: 50
    rate_limit_rpm: 1000

Aliases are versioned and signed. The gateway reloads them on a periodic cycle, so publishing a new version changes behaviour without restarting anything.

Platform aliases are additive, never overridden. kis.ai publishes a catalogue of aliases; tenants define their own alongside them. A tenant alias extends the catalogue — it cannot shadow or redefine a platform one. The same additive-inheritance rule the rest of the platform’s schemas use.

The prompt registry

Prompts are separate artifacts from aliases, versioned and referenced by ID:

kind: prompt
metadata:
  name: support/classify-system-v3
  version: 3
spec:
  system: |
    You classify inbound support messages into the intent taxonomy below.
    Answer with the intent name only.
  variables:
    - name: max_summary_tokens
      type: integer
      default: 500

Keeping prompts out of application code is what makes prompt changes reviewable, A/B testable and attributable — every accounting record carries the prompt version that produced it, so a quality regression can be traced to the exact revision that caused it.

Caching

Three layers, checked in order. Each is cheaper than the call it avoids.

Layer	Matches on	Typical use
Exact	Byte-identical request	Repeated identical prompts
Semantic	Embedding similarity above a threshold	Paraphrases of the same question
Prefix	Shared leading tokens, provider-side	Long, stable system prompts

Cache mode is set per alias. Every accounting record reports which layer served the request (miss, exact, semantic or prefix), so hit rates are measurable per alias rather than guessed at.

Fallback chains

Each alias carries an ordered list of upstreams. When one fails the gateway moves to the next, and the failure class decides whether it retries first:

• Transient (timeout, 503, upstream rate-limit) — retry the same upstream with backoff, then fall through
• Non-retryable (400, 401, 403) — fall through immediately
• Content-filter rejection — fall through immediately, since a different model may not refuse

Every attempt is recorded with provider, model, latency and error, so a chain that is quietly serving most of its traffic from the second entry is visible rather than invisible.

A/B routing

An alias can split traffic across variants:

ab_routing:
  - weight: 90
    ref: support.classifier.v3
  - weight: 10
    ref: support.classifier.v4-experimental

Routing is sticky per request-ID hash, so the same input always lands on the same variant and results stay reproducible.

A/B routing and fallback are deliberately different mechanisms: A/B is an intentional split for measurement, fallback is an unintended diversion caused by failure. They compose — A/B picks the variant, and that variant’s own fallback chain handles its failures. Accounting records carry the variant tag so cost and quality can be compared directly.

Budgets

Budgets are enforced at three scopes, all before the upstream call:

• Per-alias — this alias may spend $50/day per tenant at 1000 RPM
• Per-tenant-per-block — this tenant’s bot may spend $200/day across every alias it uses
• Per-tenant globally — this tenant may spend $5000/day across the whole platform

Each has a soft and a hard threshold, defaulting to 80% and 100%. Crossing the soft threshold still serves the request but emits a warning event; crossing the hard one rejects with 429. The gap is deliberate — it gives a tenant time to react before anything breaks.

Priority lanes

When a tenant nears its ceiling, the gateway sheds cheap traffic first. X-Kis-Priority takes critical (served until the hard budget), normal (the default), or background (deprioritised first — batch distillation, backfills and similar).

Next

• Configuration — authoring aliases and prompts
• Operations — accounting, audit and replay