AI Gateway

AI Gateway (ai-gateway) is the one place every model call on the platform goes through. Application code never holds a provider API key, never names a vendor model, and never implements its own retry or budget logic. It calls the gateway with a logical model name, and the gateway decides what that resolves to.

That indirection is what the gateway buys you. Swapping a frontier model for a fine-tuned 7B, adding a fallback, capping a tenant’s daily spend, or A/B testing a new prompt are changes to a file in the product’s ai/models or ai/prompts, or to the gateway’s configuration, and the gateway serves them without a restart. No redeploy of the calling application.

What it gives you

Capability	What it means in practice
Provider abstraction	One API across every supported provider (see below)
Model aliasing	Code references support.classifier.v3, not claude-haiku-4-5
Fallback chains	Primary times out, the next upstream serves the request
Budgets	Per-alias, per-tenant-per-block and per-tenant daily cost and rate ceilings
Response caching	Exact, semantic and provider-side prefix caches
Prompts	Liquid prompts in the product’s ai/prompts, called by name with parameters
Per-tenant definitions	Each tenant’s aliases and prompts come from its own product; its own providers from its configuration
Cost accounting	Every call attributed to tenant, product, environment, block and purpose
Audit and replay	Full request/response trail with redaction, replayable

Providers and modalities

Provider	Reached as
Anthropic	Native adapter
OpenAI	Native adapter; also covers Azure, Together and Mistral by configuration
Gemini, Bedrock, Cohere	Native adapters
vLLM, Ollama, llama.cpp, Groq, xAI	Named entry points over the OpenAI wire format
TEI	Sentence-transformers / BGE, including native reranking
Deepgram, whisper.cpp	Speech to text
Stability	Image generation

Adapters are written directly against each provider’s HTTP API rather than wrapping vendor SDKs, which is why adding a provider does not add a dependency tree to every block that calls one.

Two details worth knowing. llama.cpp gives exact local token counts through its native tokenize endpoint, so cost accounting for self-hosted models is measured rather than estimated. vLLM supports guided decoding, which is the reliable way to get schema-valid structured output out of an open-weight model.

Beyond chat and embeddings the gateway covers image generation, speech-to-text and text-to-speech through the same aliasing, budget and accounting machinery. A modality is not a separate integration path.

Two dialects, one endpoint

The gateway serves the OpenAI and Anthropic APIs simultaneously on the same port, so you can point an existing SDK at it without rewriting your integration:

POST /v1/chat/completions      # OpenAI dialect, sync or SSE
POST /v1/embeddings            # OpenAI dialect
GET  /v1/models                # lists the aliases you can use
POST /v1/messages              # Anthropic Messages dialect, sync or SSE

Internally every request is normalised to one canonical representation, processed, and re-encoded in whichever dialect the caller used. The client dialect is independent of the upstream dialect: an Anthropic-SDK caller can be served by a vLLM upstream, and an OpenAI-SDK caller by Anthropic, because nothing downstream of the codec knows which wire format the request arrived in.

A request end to end

your code ──▶ POST /v1/chat/completions  { "model": "support.classifier.v3"... }
                        │
                        ▼
     resolve alias ──▶ apply A/B split ──▶ render prompt
                        │
                        ▼
              cache hit? ──yes──▶ return, no upstream call, no cost
                        │ no
                        ▼
              check budget
                        │
                        ▼
              call upstream (fall through the chain on failure)
                        │
                        ▼
              meter tokens + cost ──▶ write audit record ──▶ respond

Budget checks happen before the upstream call. A request that would breach a hard ceiling is rejected with 429 and a structured error naming the budget it hit, so it costs nothing.

Calling it

Every request carries a bearer token and the platform’s tenancy headers. They are what makes per-tenant budgets, isolation and cost attribution work:

POST /v1/chat/completions HTTP/1.1
Authorization:    Bearer <token>
X-Customer:       acme
X-Env:            prod
X-Product:        support-portal
X-Tenant:         main
X-Kis-Block:      support-bot
X-Kis-Purpose:    triage
X-Kis-Request-ID: 01JB7CY1Q6R0GK09RZAVDQGF11
Content-Type:     application/json

{
  "model": "support.classifier.v3",
  "messages": [{ "role": "user", "content": "where is my order?" }]
}

X-Customer, X-Env, X-Product and X-Tenant name the tenant, and a user token’s tenant claim must name the same one; the gateway serves only the tenants its configuration lists. X-Kis-Block, X-Kis-Purpose and X-Kis-Request-ID are mandatory. X-Kis-Tenant, X-Kis-Customer, X-Kis-Product and X-Kis-Environment are optional restatements: when present they must name the same tenant. See API.

A call can also name one of the product’s prompts and its parameters:

{
  "model": "chat.default",
  "prompt": "email-template-gen",
  "parameters": { "templatecount": 5, "your-brand": "kis.ai", "charactercount": 150 }
}

What it is not

The AI Gateway handles model traffic only: completions, embeddings, batch jobs and structured-output calls. It is not your network gateway, and it is not an agent runtime; for running multi-step work see AI Flow.

Next

• Core Concepts: aliases, prompts, caching, fallback and A/B routing
• Configuration: the service configuration, tenants, alias files, budgets
• Authoring Prompts: prompt files in ai/prompts and calling them
• Operations: deploying, accounting, audit and replay
• API: endpoints, headers, admin routes and error codes