Configuration

The AI Gateway has two configuration surfaces. Service configuration is a YAML file read at startup — providers, credentials, storage, retention. Alias and prompt configuration is managed at runtime through the admin API and reloaded on a cycle, so day-to-day changes never require a restart.

Service configuration

gateway:
  bind: 0.0.0.0:8443
  tls:
    cert_ref: vault:ai-gateway/tls-cert
    key_ref: vault:ai-gateway/tls-key

  upstream_providers:
    anthropic:
      endpoint: https://api.anthropic.com
      api_key_ref: vault:anthropic/api-key
      timeout: 60s
      max_concurrent: 200
    openai:
      endpoint: https://api.openai.com
      api_key_ref: vault:openai/api-key
      timeout: 60s
      max_concurrent: 200
    vllm_internal:
      endpoint: https://vllm.svc.cluster.local
      timeout: 30s
      max_concurrent: 500

  registry:
    cache_ttl: 10s          # how often aliases and prompts are reloaded

  cache:
    semantic:
      enabled: true
      threshold: 0.92       # cosine similarity required for a semantic hit

  budget:
    reconciliation: 60s     # how often in-memory counters are flushed to durable storage

  pricing:
    reload: 1h              # provider price tables, used for cost accounting

Caution

Provider credentials are always vault: references, never inline values. The gateway holds every upstream API key so that no calling application has to — putting a literal key in this file defeats the main security property of routing model traffic through one service.

Upstream providers

Each entry under upstream_providers names a provider an alias can target. Supported: anthropic, openai, and openai_compatible for anything speaking the OpenAI wire format — vLLM, Ollama, llama.cpp, Groq, xAI and most self-hosted servers.

max_concurrent bounds in-flight requests to that upstream. It is the backstop that stops one misbehaving tenant from exhausting a shared provider quota; budgets are the per-tenant control, this is the global one.

Admin API

Alias, prompt, budget and experiment management live on a separate port behind mTLS and platform authentication:

POST   /admin/v1/aliases              create an alias
GET    /admin/v1/aliases              list aliases visible to the tenant
GET    /admin/v1/aliases/{name}       fetch one, including resolved inheritance
PUT    /admin/v1/aliases/{name}       publish a new version
DELETE /admin/v1/aliases/{name}       retire a tenant alias

POST   /admin/v1/prompts              create a prompt
GET    /admin/v1/prompts/{name}       fetch a prompt, any version
PUT    /admin/v1/prompts/{name}       publish a new version

GET    /admin/v1/budgets              current consumption against every scope
PUT    /admin/v1/budgets/{scope}      set soft and hard thresholds

POST   /admin/v1/experiments          start an A/B split
GET    /admin/v1/experiments/{id}     per-variant cost and quality

Platform-owned aliases are readable but not writable by a tenant. Tenant aliases live alongside them and are only visible within that tenant.

Writing an alias

Start from the cheapest upstream that can do the job and let fallback cover the rest:

kind: model-alias
metadata:
  name: support.triage.v1
  tenant: acme
  version: 1
spec:
  upstream:
    provider: openai_compatible
    endpoint: vllm_internal
    model: qwen2.5-7b-support-v2
    parameters:
      max_tokens: 512
      temperature: 0.2
  prompt:
    registry_ref: support/triage-system-v1
  fallback:
    - provider: anthropic
      model: claude-haiku-4-5
  cache:
    enabled: true
    mode: semantic
    ttl: 6h
  budget:
    cost_ceiling_usd_per_tenant_per_day: 25
    rate_limit_rpm: 600

A few things worth getting right the first time:

Set a budget on every alias. An alias with no ceiling inherits only the tenant-global one, which means a runaway loop in one feature can consume the whole tenant’s daily allowance before anything trips.

Match cache mode to the traffic. exact for genuinely repeated calls, semantic for natural-language questions that get asked many ways, prefix for long stable system prompts where the variable part is short.

Order fallbacks by cost, not by preference. The chain exists to keep you serving during a failure, and the entry most likely to be reached in an incident is the one you should be happy to pay for.

Tenancy

Every model request must carry X-Kis-Tenant, X-Kis-Product, X-Kis-Environment and X-Kis-Block. X-Kis-Customer and X-Kis-Priority are optional.

These are not bookkeeping. They are the partition key for the alias registry, the cache, the budget counters, the audit trail and the accounting records — a request missing them is rejected rather than served against a default tenant.

Next

• Operations — running the gateway, accounting, audit and replay