AI Gateway Configuration

The AI Gateway has three configuration surfaces:

Surface	Holds	Changed by
Service configuration (ai-gateway: block)	Upstream providers, the platform’s alias and prompt directories, caches, budgets, sessions, accounting and audit sinks, the secret store	The operator. A change rebuilds the gateway without a restart
The gateway’s tenants	Which tenants it serves, each tenant’s product (its meta source) and the tenant’s own providers	The operator, in the gateway’s configuration
Product files	A tenant’s model aliases (ai/models) and prompts (ai/prompts)	The product’s developers, through git. Watched: an edit is served without a restart

Service configuration

ai-gateway:
  allow_direct_refs: false
  enforce_tenant_match: true

  upstream_providers:
    anthropic:
      kind: anthropic
      endpoint: https://api.anthropic.com
      api_key_ref: vault:anthropic-api-key
      timeout: 60s
      max_concurrent: 200
    openai:
      kind: openai
      endpoint: https://api.openai.com/v1
      api_key_ref: vault:openai-api-key
      timeout: 60s
      max_concurrent: 200
    vllm_internal:
      kind: vllm
      endpoint: https://vllm.svc.cluster.local/v1
      timeout: 30s
      max_concurrent: 500

  registry:
    path: /etc/ai-gateway/aliases   # the platform's aliases
    cache_ttl: 10s                  # how often the directory is re-read
  prompts:
    path: /etc/ai-gateway/prompts   # the platform's prompts
    cache_ttl: 10s

  cache:
    exact:
      enabled: true
      default_ttl: 1h
    semantic:
      enabled: true
      embed_alias: embed.default    # an alias that resolves to an embedding model
      threshold: 0.92               # cosine similarity required for a semantic hit

  budget:
    default_soft_threshold: 0.8
    tenant_block_ceiling_usd_per_day: 200
    tenant_global_ceiling_usd_per_day: 5000

  session:
    idle_ttl: 30m

  accounting:
    file: /var/lib/ai-gateway/accounting.jsonl
  audit:
    file: /var/lib/ai-gateway/audit.jsonl

  pricing:
    overrides:
      - { model: "qwen2.5-7b-support-v2", input_per_mtok: 0.10, output_per_mtok: 0.20 }

  vault:
    url: https://vault.internal:8200

The ai-gateway: block is a closed key set: a key the gateway does not read refuses boot, and the error names it. A misspelt key therefore never boots as its default. The block may live in the bootstrap file or the cluster configuration; on a cluster-configuration change the gateway is rebuilt from the new configuration, and a configuration it cannot serve (an unknown key, a platform directory that does not load) changes nothing: the running configuration stays.

Earlier releases named the block loki:. A configuration that still does refuses boot with the loki: block is now ai-gateway:: rename the block, in the service configuration and in every tenant node that has one. The keys under it are unchanged.

Keys

Key	Default	Effect
allow_direct_refs	false	Accept a provider:model reference in model, bypassing aliases
enforce_tenant_match	true	A user token’s tenant claim must name the request’s tenant, else 403. false only logs the mismatch: development only
upstream_providers.<name>.kind	inferred from the name	anthropic, openai, ollama, vllm, groq or xai. Unset, the name decides: one containing anthropic or claude is anthropic, ollama, vllm or groq that kind, xai or grok is xai, and any other is openai. An explicit kind outside the list fails boot
upstream_providers.<name>.endpoint		The provider’s API base URL, including /v1 where the API has one (Ollama: http://localhost:11434/v1)
upstream_providers.<name>.api_key_ref		vault:<name> or openbao:<name> (resolved through the vault block), env:<VAR>, or a literal key
upstream_providers.<name>.timeout	60s	Request timeout
upstream_providers.<name>.max_concurrent	0 (unlimited)	In-flight requests to this provider
registry.path	none	Directory of the platform’s alias files
registry.cache_ttl	10s	Re-read cycle of registry.path
prompts.path	none	Directory of the platform’s prompt files
prompts.cache_ttl	10s	Re-read cycle of prompts.path
cache.exact.enabled	false	The exact response cache, for aliases with cache.mode: exact
cache.exact.default_ttl	24h	Entry lifetime for an alias with no cache.ttl
cache.semantic.enabled	false	The semantic cache, for aliases with cache.mode: semantic. Needs embed_alias
cache.semantic.embed_alias	none	The alias that embeds queries for the semantic cache
cache.semantic.threshold	0.92	Cosine similarity a semantic hit needs
budget.default_soft_threshold	0.8	Fraction of a ceiling at which the soft warning fires
budget.tenant_block_rpm	0 (none)	Requests per minute per tenant and calling block
budget.tenant_block_ceiling_usd_per_day	0 (none)	Daily cost ceiling per tenant and calling block
budget.tenant_global_rpm	0 (none)	Requests per minute per tenant
budget.tenant_global_ceiling_usd_per_day	0 (none)	Daily cost ceiling per tenant
session.idle_ttl	30m	A session (X-Kis-Session) idle longer than this is dropped
session.default_rate_rpm	0 (unlimited)	Requests per minute per session
session.default_concurrency	0 (unlimited)	In-flight requests per session
accounting.file	none	JSONL file of accounting records. Unset, records go to the service log
audit.file	none	JSONL file of audit entries. Unset, entries go to the service log
pricing.overrides[]	none	model, input_per_mtok, output_per_mtok: USD per million tokens, for models without a published price (self-hosted, fine-tunes)
vault.url	none	The secret store vault: and openbao: key references resolve through
vault.file	none	A local vault file, used instead of vault.url on a developer machine
vault.apikey, vault.dc, vault.cluster	none	The secret-store client’s credential, and the datacentre and cluster it is named for
vault.client.transport, vault.client.agent_socket, vault.client.dial_timeout, vault.client.transport_canary		The secret-store client’s transport; see Vault

Caution

Give provider keys as vault: references. The gateway holds every upstream API key so that no calling application has to; a value with no prefix is used as the literal key, which puts the key in this file and defeats the main security property of routing model traffic through one service.

Upstream providers

Each entry under upstream_providers is a provider an alias can name by its key (anthropic, vllm_internal). kind: openai covers the OpenAI API and any server speaking its wire format.

max_concurrent bounds in-flight requests to that upstream. It is the backstop that stops one misbehaving tenant from exhausting a shared provider quota; budgets are the per-tenant control, this is the global one.

Tenants

The gateway serves the tenants its configuration lists, and only those. Each tenant is named by its full key, customer:env:product:tenant. Its tenant node names the product the gateway reads (meta.url, or meta.localdir for a product checkout on disk), and may give the tenant its own providers. In the gateway’s configuration (bootstrap form):

hives:
  cluster:
    - path: ""
      tenants: ["acme:prod:shop:main", "globex:prod:erp:main"]
  tenant:
    - path: "acme:prod:shop:main"
      meta:
        url: http://meta.internal:7100
    - path: "globex:prod:erp:main"
      meta:
        url: http://meta.internal:7100
      ai-gateway:
        upstream_providers:
          anthropic:                  # replaces the gateway's `anthropic` for this tenant only
            kind: anthropic
            endpoint: https://eu.anthropic.example
            api_key_ref: vault:globex-anthropic

A tenant node’s path is the tenant’s full key. For each listed tenant the gateway reads the product’s ai/models and ai/prompts through meta.svc and watches them. A tenant with no meta source is served with the platform’s aliases and prompts only.

Tenant	Answer
Not listed in the configuration	400, ai-gateway-tenant-not-served
Listed, its product not loaded yet	503, ai-gateway-tenant-not-loaded, with Retry-After
Loaded	Served. A later load that fails keeps what the tenant had

A tenant’s own providers

ai-gateway.upstream_providers in a tenant node takes the same keys as the service’s: kind, endpoint, api_key_ref, timeout, max_concurrent. A tenant provider replaces the gateway’s provider of the same name for that tenant only, so platform and product aliases that name it follow the tenant’s endpoint and key. This is where a tenant’s own endpoints live: data residency, or a tenant’s own provider account.

A tenant node’s ai-gateway: block holds upstream_providers and nothing else. Any other key, an endpoint that is not an http(s) URL, or a kind the gateway does not know fails that tenant’s load, and the gateway logs the reason.

Model aliases in a product

A product’s aliases are YAML files in its ai/models folder:

myproduct/
  ai/
    models/          model alias files, at any depth
    prompts/         prompt files: see Authoring Prompts

• Only .yaml files are alias files. A .yml file is skipped, and the gateway logs a warning naming it.
• A file holds one or more alias documents, separated by ---.
• Alias names are unique across the product’s ai/models, and contain no :.
• Unknown keys and wrong types are errors: a misspelt policy never loads as no policy.

Field	Meaning
kind	model-alias, required
metadata.name	The alias name, required
metadata.version	An integer, shown by the admin API
spec.upstream	provider and model (both required), and parameters: max_tokens, temperature, top_p
spec.fallback	Upstreams tried in order when the primary fails, the same shape as spec.upstream
spec.cache	enabled, mode (exact, semantic or prefix), ttl (a duration, 30m)
spec.budget	rate_limit_rpm, cost_ceiling_usd_per_tenant_per_day
spec.sampling	response: n candidates and a strategy (vote, first or longest); shadow: an alias to mirror to and a rate between 0 and 1
spec.prompt.registry_ref	The prompt a call to this alias runs when it names none
spec.ab_routing	weight and ref (an alias) per variant

A product alias names providers: one of the gateway’s, or one of the tenant’s own. It may not declare endpoint, kind or api_key_ref: where the gateway connects, and which key it presents, are the operator’s. metadata.tenant is not set in a product: the product is the scope. Each of these fails the load.

Every file of ai/models and ai/prompts is validated before any is served. A load that fails names the file, and the tenant keeps the aliases and prompts it had.

Writing an alias

Start from the cheapest upstream that can do the job and let fallback cover the rest:

ai/models/support.yaml

kind: model-alias
metadata:
  name: support.triage.v1
  version: 1
spec:
  upstream:
    provider: vllm_internal
    model: qwen2.5-7b-support-v2
    parameters:
      max_tokens: 512
      temperature: 0.2
  prompt:
    registry_ref: support-triage-system
  fallback:
    - provider: anthropic
      model: claude-haiku-4-5
  cache:
    enabled: true
    mode: semantic
    ttl: 6h
  budget:
    cost_ceiling_usd_per_tenant_per_day: 25
    rate_limit_rpm: 600

A few things worth getting right the first time:

Set a budget on every alias. An alias with no ceiling inherits only the tenant-global one, which means a runaway loop in one feature can consume the whole tenant’s daily allowance before anything trips.

Match cache mode to the traffic. exact for genuinely repeated calls, semantic for natural-language questions that get asked many ways, prefix for long stable system prompts where the variable part is short.

Order fallbacks by cost, not by preference. The chain exists to keep you serving during a failure, and the entry most likely to be reached in an incident is the one you should be happy to pay for.

Platform aliases and prompts

registry.path and prompts.path hold the platform’s aliases and prompts: files at any depth, in the product formats, re-read every cache_ttl. They hold platform files only: an alias file whose metadata.tenant names a tenant (anything but empty or kis-platform) is refused. A tenant’s aliases and prompts belong in its product, and its endpoints in its tenant node.

A platform alias may also declare an upstream inline: endpoint (an http(s) URL), with kind and api_key_ref.

Platform names are reserved. A product alias or prompt that takes one is not served (the platform’s is), and every load of that product logs a warning naming the file. A platform change therefore never fails a product’s load.

A configured directory that does not load fails boot. A later re-read that fails keeps the set that loaded last, and the gateway logs the error.

Managing aliases and prompts

Aliases and prompts are files, so they are changed the way the rest of a product is: a commit to the product repository, served through meta.svc. Git is their version history. The gateway records, for every call, the prompt it ran (prompt) and a stamp of the folder version it came from (prompt_version) in the accounting record.

To read what a tenant sees, use the read-only admin routes: GET /admin/v1/aliases, GET /admin/v1/aliases/{name} and GET /admin/v1/prompts/{name}. They need the tenant’s administrator; see API.

Tenancy

The request’s tenant is the one the X-Customer, X-Env, X-Product and X-Tenant headers name, and a user token’s tenant claim must name the same tenant. X-Kis-Block, X-Kis-Purpose and X-Kis-Request-ID are mandatory. X-Kis-Tenant, X-Kis-Customer, X-Kis-Product and X-Kis-Environment are optional; when present they must name the same tenant, else 403.

The full key is the partition key for aliases, prompts, caches, budgets, sessions, batches, the audit trail and the accounting records. See API.

Next

• Authoring Prompts: the prompt files in ai/prompts
• Operations: running the gateway, accounting, audit and replay