cs.DCOct 26, 2026

Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy

Authors: Mithun Sasidharan, Srinivasan Manoharan, Rupesh Thakur, Abhinav TK, Ashwin Hegde, Anand Gautam

Organizations: PayPal Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large organizations consume large language models (LLMs) from several vendors at once, each with its own API dialect, credential model, streaming protocol and pricing. Integrated directly, this heterogeneity yields credential sprawl, format lock-in, unbounded spend and inconsistent governance. We present the design and production operation of the Cosmos LLM Gateway, the governed entry point for enterprise LLM traffic at PayPal. Built on Envoy, it splits work by latency profile: I/O-bound control decisions (authentication, credential brokering, policy and budget evaluation) run in an out-of-process external-processing service, while wire-format translation, token metering and prompt-cache shaping run in-proxy as a Rust WebAssembly filter, so that, by default, streamed response body chunks never wait on an out-of-process call. Models, providers and prices are largely validated data rather than code. Governance is declarative policy evaluated by Open Policy Agent and managed through a control plane with role-scoped authoring, approvals, pilots, shadow-mode rollout, usage notifications and versioned policy bundles. For use cases that require it, a data-loss-prevention layer can block, redact or pseudonymize detected sensitive content before it reaches a provider. Spend scopes are views over per-model usage counters, so a new budget applies to spend already recorded in its window without migration. As of September 2026, the gateway serves more than 12,000 developers and hundreds of business use cases, handling roughly 200 billion tokens per day across more than 100 models at 99.999% availability to business. In a benchmark of the core request path against a synthetic upstream, it adds a median of 4 ms and a p99 of 11 ms of overhead at more than 2,200 requests per second on two 4-vCPU instances. We report the architecture, policy model, evaluation and lessons learned.

Figures & tables

Figure 1: Architecture. One governed entry point fronts enterprise LLM traffic. On the unified API, the control service makes I/O-bound decisions out of process and fails closed, and the Wasm filter translates and meters in process. Real-time sessions use a connection-oriented path tuned for long-lived streams. The control plane publishes versioned policy bundles that the data plane enforces on every request. The shaded region is PayPal’s private network: self-hosted open-weight models run inside it, and external providers are reached with brokered credentials.
Settingcontrol ext_procmetrics ext_proc
Failure modefail closedfail open
Request headerssendsend
Request bodybufferedstreamed (mirrored)
Response headersskipsend
Response bodynonestreamed (mirrored)
Response trailerssendsend
Table 1: Default ext_proc processing modes. Response bodies never wait on an out-of-process call unless a use case’s data-loss-prevention policy requires response handling (section 8 ).
Figure 2: Lifecycle of a unified request. Routing decisions happen in the body phase because the model is in the body. Streamed response bytes pass through only the in-process filter on the critical path.
FailureBehavior
Control service unavailableRequest rejected (fail closed)
Access-policy evaluation errorDeny
Spend-policy evaluation errorDeny when enforcing; logged in shadow mode
Data-protection check cannot completeRequest not sent (fail closed)
Invalid registry / policy at bootProcess refuses to start
Policy bundle reload failsKeep last-known-good revision
Table 2: Failure semantics on the request path.
 Upstream format
Entry formatOAI ChatOAI Resp.Anth.@VertexGemini
Anthropic MessagesTTRT
OpenAI ChatT ∗TTT
OpenAI Responses–I––
Gemini generateContent–––I
Table 3: Conversion plan by entry and upstream format: OAI = OpenAI. I = identity, R = raw envelope rewrite, T = translation through the IR, – = not offered (by registry configuration, not code). Vertex denotes the Gemini Enterprise Agent Platform API. ∗ Same-format round trip that preserves Gemini tool-call thought signatures (section 7 ).
Figure 3: Data-loss-prevention flow. Detected sensitive content is blocked, redacted or replaced with format-preserving surrogates inside the enterprise boundary before the provider call; surrogates are mapped back in the response.
LayerSet byEffect
Catalog admissionplatform (registry)unknown model →\rightarrow 404
Model restrictionplatform adminleader allow/deny lists and exemptions →\rightarrow 404 (same as unknown)
BU allow/block listplatform admina model is open to, or closed to, named business units
BU model toggleBU adminswitch a model off for one’s own BU; cannot override a platform lock
Limited accessplatform adminbounded named roster with its own spend controls and expiry
Pilot cohortplatform admindate-bounded allow/restrict lists per cohort
Table 4: Access-control layers. Each layer narrows the access granted by the one above it, except authoritative pilot cohorts (section 10.2 ); spend caps are separate and can be raised within platform locks (section 9.2 ).
Figure 4: Policy lifecycle. Intent is authored and validated against the same rules the data plane enforces, approved when it originates from a business unit, persisted with an audit record, projected into a versioned bundle, and enforced on every request. New controls are rolled out in shadow mode before they block.
Figure 5: Model onboarding through the control plane: each step is a validated, audited data change; no code release is involved.
  Overhead (ms) 
InstancesRPSp50p95p99Failed
21,137410110
22,257410110
41,138410110
42,26249100
Table 5: Gateway overhead (time to the start of the upstream response, minus upstream service time) with a synthetic 110 ms upstream. RPS: achieved requests per second.
Figure 6: Overhead percentiles by instance count and load. Overhead was essentially unchanged when load or instance count doubled within the tested range.
MeasureValue
Developers using the gateway12,000+
Production use caseshundreds
Requests per daymillions
Tokens per day (all traffic)∼\sim 200 billion
Models / model families / providers100+ / multiple / multiple
Peak throughput from a single workload60,000 req/min
Table 6: Production operation (historical point-in-time figures as of September 2026, not service-level commitments). ATB is PayPal’s availability-to-business service-level indicator.
More in this topic

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 355 papers

Explore similar work

CardsList
Apr 22, 2026

Behavioral Consistency and Transparency Analysis on Large Language Model API Gateways

Third-party Large Language Model (LLM) API gateways are rapidly emerging as unified access points to models offered by multiple vendors. However, the internal routing, caching, and billing policies of these gateways are largely undisclosed, leaving users with limited visibility into whether requests are served by the advertised models, whether responses remain faithful to upstream APIs, or whether invoices accurately reflect public pricing policies. To address this gap, we introduce GateScope, a lightweight black-box measurement framework for evaluating behavioral consistency and operational transparency in commercial LLM gateways. GateScope is designed to detect key misbehaviors, including model downgrading or switching, silent truncation, billing inaccuracies, and instability in latency by auditing gateways along four critical dimensions: response content analysis, multi-turn conversation performance, billing accuracy, and latency characteristics. Our measurements across 10 real-world commercial LLM API gateways reveal frequent gaps between expected and actual behaviors, including silent model substitutions, degraded memory retention, deviations from announced pricing, and substantial variation in latency stability across platforms.

Guanjie Lin, Yinxin Wan, Shichao Pei +3

University of Massachusetts Boston · Boston, Massachusetts, USA · Arizona State University +2

LLM AuditingBlack-Box Model Auditing
AskPDF
Sep 1, 2026

From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix

Data-residency constraints force enterprises to self-host LLMs, but continuous adoption of newer models without decommissioning their predecessors expands the serving fleet, fragmenting a finite GPU pool. We consolidate traffic from over 200 internal applications onto a single model by closing quality gaps identified through production error analysis along three axes: instruction following, function-calling, and internal task distribution. Quality is tracked by offline benchmarks stratified to production traffic and scored by deterministic verifiers or calibrated LLM judges. Rather than optimising all objectives jointly, which introduces cross-domain reward interference, we train a separate GRPO expert per axis and merge them via two-stage SLERP. Each expert's reward exposes a distinct failure mode, namely semantic collapse, over-calling, and verbosity hacking, each requiring a domain-specific fix. In non-reasoning mode the recipe surpasses a ∼7×{\sim}7\times larger by total parameters baseline on the in-house Arena with 69.6 to 65.8, instruction following with 0.85 to 0.83, and function-calling with 0.79 to 0.77, while lifting general dialogue benchmarks. The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.

Olga Tsymboi, Dmitrii Stoianov, Ramil Latypov +11

T-Tech

LLM ServingInstruction FollowingModel MergingLLM Post-TrainingTool Use
AskPDF
Jun 3, 2026

Organizational Control Layer: Governance Infrastructure at the Execution Boundary of LLM Agent Systems

LLM-based agents are increasingly deployed in workflows where generated outputs may directly trigger state-changing actions. This creates an execution-boundary problem: proposed actions must be governed before they are executed. We study this problem through economically consequential multi-agent interactions and argue that deployment-grade agent systems should separate proposal generation from environment-facing execution. To operationalize this principle, we introduce the Organizational Control Layer (OCL), a model-agnostic governance infrastructure that intercepts generated actions before execution through policy enforcement and escalation, without modifying the underlying LLM generator. We evaluate OCL on adversarial buyer--seller negotiation environments adapted from AgenticPay. Across multiple frontier LLM backends, OCL reduces unsafe executions from 88% to near-zero while increasing valid success from 12% to 96%. Results further reveal a safety--utility tradeoff: strict governance improves compliance and reliability against policy and constraint violations, but can reduce flexibility in tightly constrained markets. These findings suggest that deployment-grade LLM agent systems require explicit governance at the boundary between language generation and executable actions. The source code is available at: https://github.com/SHITIANYU-hue/amai_ocl

Tianyu Shi, Yang Mo, Yiou Liu +6

McGill · Purdue · UNSW +3

Multi-Agent LLM SystemsAI Agent GovernanceLLM Agent Safety
AskPDF