AI Gateway

AI Gateway (ai-gateway) is the one place every model call on the platform goes through. Application code never holds a provider API key, never names a vendor model, and never implements its own retry or budget logic. It calls the gateway with a logical model name, and the gateway decides what that resolves to.

That indirection is what the gateway buys you. Swapping a frontier model for a fine-tuned 7B, adding a fallback, capping a tenant’s daily spend, or A/B testing a new prompt are all configuration changes at the gateway. No redeploy of the calling application.

What it gives you

Capability	What it means in practice
Provider abstraction	One API across every supported provider (see below)
Model aliasing	Code references support.classifier.v3, not claude-haiku-4-5
Fallback chains	Primary times out, the next upstream serves the request
Budgets	Per-alias, per-tenant-per-block and per-tenant daily cost and rate ceilings
Response caching	Exact, semantic and provider-side prefix caches
Prompt registry	Versioned prompts referenced by ID, with A/B routing
Cost accounting	Every call attributed to tenant, product, environment, block and purpose
Audit and replay	Full request/response trail with redaction, replayable

Providers and modalities

Provider	Reached as
Anthropic	Native adapter
OpenAI	Native adapter; also covers Azure, Together and Mistral by configuration
Gemini, Bedrock, Cohere	Native adapters
vLLM, Ollama, llama.cpp, Groq, xAI	Named entry points over the OpenAI wire format
TEI	Sentence-transformers / BGE, including native reranking
Deepgram, whisper.cpp	Speech to text
Stability	Image generation

Adapters are written directly against each provider’s HTTP API rather than wrapping vendor SDKs, which is why adding a provider does not add a dependency tree to every block that calls one.

Two details worth knowing. llama.cpp gives exact local token counts through its native tokenize endpoint, so cost accounting for self-hosted models is measured rather than estimated. vLLM supports guided decoding, which is the reliable way to get schema-valid structured output out of an open-weight model.

Beyond chat and embeddings the gateway covers image generation, speech-to-text and text-to-speech through the same aliasing, budget and accounting machinery. A modality is not a separate integration path.

Two dialects, one endpoint

The gateway serves the OpenAI and Anthropic APIs simultaneously on the same port, so you can point an existing SDK at it without rewriting your integration:

POST /v1/chat/completions      # OpenAI dialect, sync or SSE
POST /v1/embeddings            # OpenAI dialect
GET  /v1/models                # lists the aliases you can use
POST /v1/messages              # Anthropic Messages dialect, sync or SSE

Internally every request is normalised to one canonical representation, processed, and re-encoded in whichever dialect the caller used. The client dialect is independent of the upstream dialect: an Anthropic-SDK caller can be served by a vLLM upstream, and an OpenAI-SDK caller by Anthropic, because nothing downstream of the codec knows which wire format the request arrived in.

A request end to end

your code ──▶ POST /v1/chat/completions  { "model": "support.classifier.v3"... }
                        │
                        ▼
     resolve alias ──▶ apply A/B split ──▶ load prompt ──▶ check budget
                        │
                        ▼
              cache hit? ──yes──▶ return, no upstream call, no cost
                        │ no
                        ▼
              call upstream (fall through the chain on failure)
                        │
                        ▼
              meter tokens + cost ──▶ write audit record ──▶ respond

Budget checks happen before the upstream call. A request that would breach a hard ceiling is rejected with 429 and a structured error naming the budget it hit, so it costs nothing.

Calling it

Every request carries the platform’s tenancy headers. They are what makes per-tenant budgets, isolation and cost attribution work:

POST /v1/chat/completions HTTP/1.1
X-Kis-Tenant:       acme
X-Kis-Product:      support-portal
X-Kis-Environment:  prod
X-Kis-Block:        support-bot
X-Kis-Priority:     normal
Content-Type:       application/json

{
  "model": "support.classifier.v3",
  "messages": [{ "role": "user", "content": "where is my order?" }]
}

X-Kis-Tenant, X-Kis-Product, X-Kis-Environment and X-Kis-Block are mandatory. X-Kis-Customer and X-Kis-Priority are optional.

What it is not

The AI Gateway handles model traffic only: completions, embeddings, batch jobs and structured-output calls. It is not your network gateway, and it is not an agent runtime; for running multi-step work see AI Flow.

Next

• Core Concepts: aliases, prompts, caching, fallback and A/B routing
• Configuration: writing aliases and prompts, budgets, tenancy
• Operations: deploying, accounting, audit and replay