What is an LLM gateway?

An LLM gateway is a proxy that sits between your application or agent and the model providers it calls. Instead of writing separate code for OpenAI's API, Anthropic's API and Google's API, your app calls the gateway with one format, and the gateway forwards each request to whichever provider and model you've configured.

Most gateways also do a few jobs around the forwarding, like failing over when a provider is down and capping what each team can spend.

Why it exists

A team that calls several model providers directly ends up with a different SDK and key for each one in its code. Moving a feature from GPT to Claude means rewriting the calls.

A gateway puts one interface in front of all of them. LiteLLM's docs describe it this way:

"LiteLLM is an open-source library that gives you a single, unified interface to call 100+ LLMs (OpenAI, Anthropic, Vertex AI, Bedrock, and more) using the OpenAI format."LiteLLM docs, Getting Started

Portkey makes the same pitch at a larger count of models: "Portkey lets you access 1,600+ LLMs via a unified API, so you can focus on building, not managing." OpenRouter frames it as access rather than infrastructure you run yourself:

"OpenRouter gives you access to hundreds of AI models through a single API endpoint. It handles fallbacks automatically and picks the most cost-effective option for each request."OpenRouter docs, Quickstart

What it actually does

One API across providers

The gateway accepts requests in one format, usually the OpenAI chat completions shape, and translates them to whatever each provider expects. LiteLLM's Python SDK and proxy both return that same OpenAI-shaped response regardless of which provider answered, and it maps every provider's errors to OpenAI's exception types so existing error handling keeps working.

Routing and fallbacks

If a model or provider is slow or down, the gateway can retry against a different one without your code noticing. Cloudflare's AI Gateway overview lists this directly among its features:

"you can gather insights on how people are using your application with analytics and logging and then control how your application scales with features such as caching, rate limiting, as well as request retries, model fallback, and more."Cloudflare AI Gateway docs, Overview

Kong's AI Gateway lists the same job as "automatically fail over to a different provider or model when one is slow or unavailable" under load balancing.

Key management

The gateway holds the upstream API keys, and your app or agent authenticates to the gateway with its own token instead. LiteLLM's proxy issues virtual keys for this. Kong's use case table describes the goal as centralizing provider credentials in one place and rotating them without touching every team's configuration.

Rate limits and budgets

Once every call goes through one place, that place can cap it. Kong scopes limits to an "AI Consumer Group" by team or department instead of per person. LiteLLM's virtual keys carry budgets and rate limits per team or user. Portkey's homepage says: "Set budget limits and optimize resource allocation to reduce AI expenses."

Caching

A repeated or near-identical prompt doesn't need a new call to the model. Cloudflare's AI Gateway can "serve requests directly from Cloudflare's cache instead of the original model provider," and Kong's AI Semantic Cache matches prompts that are similar, not just identical.

Logging and observability

Every call through the gateway is a call it can record: tokens, cost, latency and the full prompt and response if you want it. LiteLLM sends that to Langfuse, MLflow, Helicone and others with one callback. Kong tracks "token usage, latency, and costs through audit logs, metrics exporters, and OpenTelemetry."

Guardrails

Some gateways also inspect what goes in and out. Kong's use case table lists blocking disallowed topics, catching jailbreak attempts phrased as natural language, and redacting PII before it reaches the provider. Portkey's homepage makes the same PII claim about its own gateway: "Portkey automatically redacts sensitive data from your requests before they are sent to the LLM."

LLM gateway vs API gateway vs MCP gateway

All three sit in front of some kind of call and decide what happens to it, which is why the names get mixed up. They sit at different points in the stack.

The lines blur in practice. Kong's AI Gateway and LiteLLM both also route MCP traffic. TrueFoundry sells an LLM gateway and an MCP gateway in the same product, and Portkey's homepage advertises an "MCP Gateway" that centralizes authentication for MCP servers alongside its model routing. Check what a product's MCP feature covers before assuming it matches the name.

What is an MCP server? covers the agent-to-tool side of that table, and MCP permissions covers the per-tool access decision an MCP gateway makes.

Where it sits in an agent's stack

An LLM gateway sits between your app and the model. The agent harness is the loop around the model inside your app, deciding when to call the model again, when to call a tool, and when to stop. An MCP gateway sits between the agent and the tools the harness calls out to.

Each layer catches different problems. When a provider goes down, the LLM gateway fails over, and when a harness loops, its budget cap stops the spend. A tool call the agent shouldn't make never passes through the LLM gateway at all, so stopping it is the MCP gateway's job.

Next

What is an agent harness? covers the loop that decides when to call the model again versus call a tool. What is an AI agent? and What is an MCP server? cover the rest of the stack.

Questions