Gateway

AI Gateway allows developers to expose multiple models through specific URL paths, simplifying the integration of various AI models into applications. This block ensures seamless communication and scalability by providing multiple proxy LLMs for different use cases. Key features include A/B testing between LLMs, querying multiple LLMs simultaneously, load balancing across LLMs, and selecting the best response from multiple LLMs. Additionally, it secures these LLMs with access rules and supports rate limiting at the user, team, or route level.

Model Providers

Model providers provide access to foundational and private models over REST API. The model providers abstract the detail of model location, security tokens and availability.

Name	Description	Supported Models
openai	Open AI and all the models provided by them	gpt-3.5-turbo, gpt-4-turbo, gpt-4o
anthropic	Anthropic and all the models provided by them	claude-3-opus, claude-3-sonnet, claude-3-haiku
local-llm	Open source LLMs available locally	llama-3-8b, llama-3-8b-128k, llama3-70b, codellama2-34b, codellama2-7b, mistral7b, mixtral8x7b, phi3
load-llm	Proxy LLM which splits the request between multiple LLMs	All the models above
multi-llm	Proxy LLM which queries multiple llms with the same query for critical use cases	All the models above
ab-llm	Proxy LLM which splits queries between multiple models in a given ratio	All the models above
best-llm	Proxy LLM which queries multiple models with same query and gives the better answer scored by different llm	All the models above

Examples

Model providers are defined in the ai/routes/ folder of a product in YAML files under the key model-providers. There can be multiple YAML files in this folder and the AI Gateway service will coalesce all of them and resolve all providers.

Open AI, LLaMa 3 Providers

model-providers:
- name: openai
  type: openai
  model: gpt-4o
  vault-key: llm/open-ai-secure-key
- name: llama3-8b-128k
  type: local-llm
  model: llama3-8b-128k # llama3-8b with 128k context window by Gradient AI

Load-balancing LLM Provider

model-providers:
# the llama providers are not included for brevity
- name: load-know
  type: load-llm
  llms:
    - name: llama3-8b-01
    - name: llama3-8b-02
    - name: llama3-8b-03
  config:
    sticky: false
    strategy: roundrobin

Multi LLM Provider

model-providers:
# the openai and llama providers are not included for brevity
- name: multi-know
  type: multi-llm
  llms:
    - name: openai
    - name: llama3-8b-128k
  config:
    strategy: waitforall

A/B testing with LLM Providers

model-providers:
# the openai and llama providers are not included for brevity
- name: a-80-b-20
  type: ab-llm
  llms:
    - category: a
      name: openai
      ratio: 0.8
    - category: b
      name: llama3-8b-128k
      ratio: 0.2

Best LLM Provider

model-providers:
# the openai and llama providers are not included for brevity
- name: best-llm
  type: best-llm
  llms:
    - name: openai
      ratio: 0.8
    - name: llama3-8b-128k
      ratio: 0.2
  judge-llm:
    - name: mistral
  config:
    strategy: best

Routes

Routes are defined by the developers in the ai/routes/ folder of a product in YAML files under the key routes. There can be multiple YAML files in this folder and the AI Gateway service will coalesce all of them and serve. Routes YAML has three key information for each route:

Model Provider and configuration for that provider
Access Rules on who can access this route
Rate Limits defined at user, group and route level

Examples

Access Open AI gpt-4o

In the below example route the developer has created a new route /ai/tasks which can be accessed to Open AI models without exposing or sharing the token. Here a user is allowed access to make the calls only if they have llm-open-ai-allowed group in their jwt token claims. They have also applied additional usage limits to keep the costs in control.

routes:
- name: tasks
  path: /tasks
  provider: openai
  access: |
    user.groups.includes('llm-open-ai-allowed')
  rate-limits:
    user:
      prompts: 60 / hour
      tokens: 3000 / prompt # or 100000 / hour or 8000 / request
    group:
      rule: |
        user.department
      prompts: 600 / hour
      tokens: 3000 / prompt # or 100000 / hour
    total:
      prompts: 1000 / hour
      tokens: 3000 / prompt

On the above endpoint, a user can call 60 times in an hour, and all users in a given department ‘Sales’ can make 600 requests in an hour collectively, and the entire user base requests are limited at 1000 per hour. Similarly the token limits are applied.

If any of the rate-limit thresholds are violated the gateway returns 429 HTTP status code.

Access local llama3-8b

Here the developer is exposing a local llama3-8b over a route /ai/know

routes:
- name: knowledge
  path: /know
  provider: llama3-8b-128k
  access: |
    user.groups.includes('llm-llama-know-allowed')

Access local llama3-8b (Multi)

Here the developer is exposing a local llama3-8b over a route /ai/know

routes:
- name: multi
  path: /multi-know
  provider: multi-know
  access: |
    user.groups.includes('llm-multi-know-allowed')

Accessing the models

With the model-providers and routes configuration, the AI Gateway block provides unified API access to all models. The format for of response matches the Open AI chat completion response format.

Examples

Here the developer is calling the /tasks route with POST verb with a json payload with the prompt name and parameters required to expand the prompt. Refer to Prompts to learn more about defining prompts.

OpenAI example

Request

POST /ai/tasks

{
  "prompt": "email-template-gen",
  "parameters": {
    "templatecount": 5,
    "your-brand": "kis.ai",
    "charactercount": 150,
    "topic": "hotel management software",
    "brand": "Sun and Sands Resorts",
    "firstname": "Joe",
    "lastname": "Pescano",
    "signature": "Wishing a sunny day, AP"
  }
}

Response

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": "### Email Template 1\nSubject: Enhance Sun and Sands Resorts with kis.ai\n---\nHi Joe Pescano,\n\nDiscover how kis.ai can revolutionize your hotel management software at Sun and Sands Resorts. Let's discuss!\n\nWishing a sunny day,\nAP\n\n### Email Template 2\nSubject: Transform Sun and Sands Resorts with AI\n---\nHi Joe,\n\nBoost efficiency at Sun and Sands Resorts using kis.ai's advanced AI for hotel management software.\n\nWishing a sunny day,\nAP\n\n### Email Template 3\nSubject: Innovate Hotel Management with kis.ai\n---\nHi Joe,\n\nElevate Sun and Sands Resorts' software with kis.ai's AI capabilities. Ready to explore?\n\nWishing a sunny day,\nAP\n\n### Email Template 4\nSubject: AI Solutions for Sun and Sands Resorts\n---\nHi Joe,\n\nImprove Sun and Sands Resorts with kis.ai's hotel management software. Let's connect!\n\nWishing a sunny day,\nAP\n\n### Email Template 5\nSubject: Upgrade Sun and Sands Resorts with kis.ai\n---\nHi Joe,\n\nTransform your hotel management at Sun and Sands Resorts using kis.ai's AI solutions.\n\nWishing a sunny day,\nAP",
        "role": "assistant"
      }
    }
  ],
  "created": 1716301646,
  "id": "chatcmpl-9RKkYAAHZJ4ewBV4rxdCdnVBCav08",
  "model": "gpt-3.5-turbo-0125",
  "object": "chat.completion",
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 184,
    "prompt_tokens": 28,
    "total_tokens": 212
  }
}

Multi LLM Example

Request

{
    "prompt": "test-zero-shot-prompt",
    "parameters": {
        "prompt": "please tell me a good story"
    }
}

Response

{
  "openai": {
    "choices": [
      {
        "finish_reason": "stop",
        "index": 0,
        "logprobs": null,
        "message": {
          "content": "Once upon a time, in a small village nestled between rolling green hills and lush forests, there lived a young girl named Lily. Lily was known throughout the village for her kind heart and gentle spirit. She had a special connection with animals, and they seemed to be drawn to her.\n\nOne day, as Lily was walking through the woods, she heard a faint meowing coming from behind a bush. Curious, she approached and discovered a tiny kitten, all alone and in need of help. Lily gently picked up the kitten and cradled it in her arms. She could see that it was malnourished and scared.\n\nDetermined to help the kitten, Lily brought it back to her home and carefully nursed it back to health. As days passed, the kitten grew stronger and more playful, forming a deep bond with Lily. They became inseparable companions, exploring the woods together and sharing adventures.\n\nWord of Lily's kindness spread throughout the village, and soon people from far and wide would come to seek her help with their own animals in need. Lily was always willing to lend a hand, whether it was rescuing a lost puppy or nursing an injured bird back to health.\n\nAs time went on, the village grew to become a haven for animals of all kinds, thanks to Lily's compassion and dedication. And so, the young girl with the kind heart became known as the Guardian of the Animals, a title she wore with pride and humility.\n\nAnd they all lived happily ever after, surrounded by the love and gratitude of the creatures they had helped.",
          "role": "assistant"
        }
      }
    ],
    "created": 1716303276,
    "id": "chatcmpl-9RLAqNEoMg1dixbIa5T8mu5zLx92O",
    "model": "gpt-3.5-turbo-0125",
    "object": "chat.completion",
    "system_fingerprint": null,
    "usage": {
      "completion_tokens": 314,
      "prompt_tokens": 26,
      "total_tokens": 340
    }
  },
  "test-arka-flow": {
    "choices": [
      {
        "finish_reason": "length",
        "index": 0,
        "logprobs": null,
        "text": " Once upon a time in a faraway land, there was a small, peaceful village nestled among the lush green rolling hills. This village was called Willowbrook, and it was home to a variety of creatures who lived harmoniously together. Among the villagers were rabbits, squirrels, birds, deer, and even a family of bears.\n\nAt the heart of Willowbrook was a magnificent willow tree that stood tall and proud. According to a legend, this was the tree that had given the village its name. It was said that the tree held the spirits of the ancestors of the animals who lived there.\n\nOne sunny day as the village was going about its usual business, there was an unexpected event. A great and powerful storm brewed over the horizon. The animals of Willowbrook grew anxious as the ominous dark clouds rolled in, bringing with them thunder and lightning, heavy rain, and strong winds.\n",
        "role": "assistant"
      }
    ],
    "created": 1004163,
    "id": "cmpl-beff332550344237812c5aa715072c59",
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "object": "text_completion",
    "usage": {
      "completion_tokens": 200,
      "prompt_tokens": 35,
      "total_tokens": 235
    }
  }
}