From the source
Perplexity's explainer defines an AI gateway as a proxy layer that routes application and agent calls to large language models while enforcing cost, security and logging controls in one place.
- The gateway accepts a single standard request, translates it for whichever provider is called, and lets teams add, swap or retire models by changing configuration rather than application code.
- Each request passes through authentication, policy checks such as token limits and PII redaction, routing by cost, latency or availability with failover, and logging of model, token counts, latency and outcome.
- Capabilities include semantic caching that returns saved answers for prompts close in meaning, guardrails that strip sensitive data and filter content, and audit logs that record every call.
- The trade-offs are stated plainly: a gateway adds latency, and it does not judge output quality, which stays the user's responsibility.
- Perplexity cites a June 2026 Broadcom VMware benchmark in which a tuned Envoy AI Gateway added about 2 ms per request at up to 224 concurrent users with routing and rate limiting but no content inspection.