Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation

Teng-Ruei Chen

cs.CROct 7, 2026

Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation

Authors: Teng-Ruei Chen

Organizations: Krixvon, Taipei 100, Taiwan

Abstract

LLM routers pick a cheap or expensive model per request by its content, and many gateways and some cloud platforms can log that choice with content logging off. We measure this privacy channel beyond token counts, accounting for noisy labels and repeated prompts. We run pre-registered studies on 1.7 million real requests (WildChat-1M, LMSYS-Chat-1M) with two cost/quality routers and a domain router, survey eleven systems' logging, and test post-processing defenses. At matched length, the shift's direction depends on category and router. For RouteLLM at the 50% operating point, harassment and self-harm requests reach the strong model 19 points less often than comparable ones on prompts unseen in exploration, medical requests (exploratory: LLM labels failed their gate) 31 points less often on distinct prompts (both post hoc), and sexual requests 10 points more often (secondary); the other router's four are negative. Twenty RouteLLM decisions separate frequent medical askers with AUC 0.71, exploratory and below the pre-registered primary endpoint's 0.75 (domain router: 0.92, an upper estimate). Per-category length-matched parity with accurate labels removes the gap on real traffic, costing at most 0.2 accuracy points on RouterBench (post hoc), where routers' gaps on sensitive subjects (13-42 points, pre-registered) exceed those of an oracle routing by realized accuracy gain (1-11, post hoc). Per-conversation stickiness, per-user budget bands, and pooled parity fail, the last as categories' shifts differ in size or sign. A post hoc exact per-user rate hides only even-prefix strong counts and forfeits most self-assessed routing value; it preserves odd-position decisions, from which a post hoc log attack reaches AUC 0.73 after 20 RouteLLM requests (exploratory).

Figures & tables

Fig. 1: Overview. Top: gateways can keep the router’s choice, the caller’s key, and token counts, often by default and with content logging off (Section III ). Bottom: what these choices reveal per request (Section V ; RouteLLM’s gaps: harassment and self-harm on never-seen and medical on distinct WildChat prompts are post hoc, medical labels exploratory, sexual secondary; notdiamond-0001’s four gaps are negative) and per user (Section VI ; 0.71 is the pre-registered primary endpoint, below 0.75; both AUCs are exploratory as the label gate failed), and what defenses cost (Section VII ; post hoc: parity’s RouterBench cost and the exact rate, under which original A2 reaches 0.53 and an odd-position log attack reaches 0.73 after 20 requests). Benchmark line: RouterBench; post hoc oracle at the same budget.

Work	Real router	Real traffic	Decision as channel	User-level accumulation	Log/billing observer	Defense evaluated
Cost/quality routers [ 1 , 2 , 4 ]	✓	partly a	–	–	–	–
Routing robustness and bias [ 7 , 8 , 27 ]	✓	partly b	partly c	–	–	–
Privacy-preserving routing [ 10 , 11 , 12 ]	✓	partly b	–	–	–	content only
AVEC [ 22 ]	–	–	✓	partly d	–	simulation
Multi-exit networks [ 15 ]	✓	–	✓	–	–	✓
Tor path selection [ 13 , 14 ]	✓	✓	✓	✓	–	✓

TABLE I: Related work by the properties this paper’s contribution turns on. Real router: an implemented router or selection mechanism, not a simulated one. Real traffic: evaluated on passively collected or released records of organic user activity, rather than researcher-issued or replayed requests. A check means that at least one cited work in the row has the property; partly means that it holds only in a weaker form (notes a–f).

Observer	Sees	Default
Org admin / platform team	per request: model, time, tokens, cost, key or member	OpenRouter, LiteLLM, Portkey: yes; Cloudflare, Helicone: no caller id; Kong, AWS: after setup
Security team	per request: caller identity, model, time	CloudTrail history: yes; SIEM: after export
Finance	aggregates by model and hour or day, per key, member, or principal on some platforms	partly
Vendor; observability subprocessor	per-request metadata, also in privacy modes	hosted vendor: yes; subprocessor: once configured
Network observer	no documented cleartext model id; serving model inferable from encrypted traffic [ 18 ]	not measured here

TABLE II: Observers of routing metadata in the eleven surveyed systems. Default: available without any non-default configuration.

Study	Frozen	Pre-registered endpoint	Outcome
E-3	2026-09-28	Medical $\|\Delta_{\mathrm{adj}}\|\geq 0.10$ (P1)	Met: $-0.251$
E-3	2026-09-28	User-level AUC $\geq$ 0.75 at $T{=}20$ (P2)	Not determinable (15 positive users; 30 required)
E-4	2026-09-29	Medical $\|\Delta_{\mathrm{adj}}\|\geq 0.10$ (P1)	Met: $-0.204$ (zero-shot labels; precision 0.55 in an AI assistant’s check)
E-4	2026-09-29	User-level AUC $\geq$ 0.75 at $T{=}20$ (P2)	Met by rule: 0.7515 [0.699, 0.799]; later shown to be the upper end of noise
E-4	2026-09-29	Same AUC, labels from later requests (co-primary)	Not met: 0.743 [0.673, 0.809]
E-5	2026-09-29	Medical $\|\Delta_{\mathrm{adj}}\|\geq 0.10$ (P1)	Met: $-0.331$

TABLE III: Pre-registration record: every primary and co-primary endpoint of E-3 to E-9, including failed ones, and the label gates. Frozen: date (UTC+8) of the protocol-freezing commit in our private repository, before data download or collection; the WildChat studies re-analyzed one release, each frozen after its predecessor’s results (E-5: before judging; E-8: before parity and band defenses).

Category	Data	Status	$n$	$P(\mathrm{strong})$	$\Delta_{\mathrm{adj}}$ [95% CI]	$\varepsilon$	$\Delta$ AUC	$\Delta_{\mathrm{adj}}$
				RouteLLM				notdiamond
Harassment	LMSYS held-out, all	confirmatory	7,140	0.187	$-0.316$ [ $-0.325$ , $-0.307$ ] †	0.99	$+0.014$	$-0.276$
Harassment	LMSYS held-out, new §	post hoc	1,786	0.351	$-0.193$ [ $-0.216$ , $-0.169$ ] †	0.44	$+0.010$	$-0.166$
Harassment	LMSYS exploration	exploratory	3,563	0.179	$-0.327$ [ $-0.338$ , $-0.316$ ]	1.04	$+0.013$ ¶	$-0.289$
Self-harm	LMSYS held-out, all	confirmatory	276	0.297	$-0.238$ [ $-0.289$ , $-0.190$ ] †	0.59	$+0.061$	$-0.097$
Self-harm	LMSYS held-out, new §	post hoc	164	0.342	$-0.186$ [ $-0.258$ , $-0.116$ ] †	0.43	$+0.050$	$-0.028$

TABLE IV: Per-request leakage at the 50% operating point (router sends 50% of non-sensitive requests to the strong model).

\Delta_{\mathrm{adj}}

: strong-routing rate of the group minus the length-reweighted rate of the comparison set, with 95% CI (request bootstrap on LMSYS, user-cluster bootstrap on WildChat).

\varepsilon

: per-request privacy loss at matched length ( 2 ).

\Delta

AUC: observer-AUC gain of the routing decision over a flexible 20-bin length model.

Fig. 2: The direction of per-request leakage depends on the category and the router, including on prompts never seen during exploration, and RouteLLM’s WildChat medical gap is negative at all 19 operating points. (a)

\Delta_{\mathrm{adj}}

(strong-routing rate of the category minus the length-reweighted rate of the comparison set) at the 50% operating point; blue circles: RouteLLM, green squares: notdiamond. Hollow markers are the pre-registered all-rows estimates, with intervals of the same type as in Table IV (which lists them for RouteLLM only). Harassment and self-harm: LMSYS held-out files; confirmatory co-primary endpoints for RouteLLM, secondary for notdiamond; 97.5% request-bootstrap CI for both routers. Sexual and violence: LMSYS held-out, secondary, 95% request-bootstrap CI. Medical: WildChat, 95% user-cluster bootstrap CI; pre-registered but exploratory, because the LLM-judge labels failed their pre-registered validity gate. Filled markers are post hoc. On LMSYS they use only prompts whose fingerprint never occurs in the exploration files, with a bootstrap over distinct prompts and the comparison set held fixed (97.5% for harassment and self-harm, 95% otherwise). On WildChat medical they use one row per distinct prompt, with a 95% bootstrap over distinct prompts; that interval ignores user clustering and holds the comparison set fixed, so it is narrower than the hollow one. Arrows run from all rows to the never-seen or distinct subset where the shift is visible; deduplication does not always shrink a gap (notdiamond, medical). Gray

n

: distinct prompts in the filled subset. Dashed lines:

\pm 0.10

, the magnitude threshold of the registered criteria. The orange bracket marks RouteLLM’s two co-primary gaps on never-seen prompts (

-0.193

and

-0.186

). On those prompts, notdiamond’s self-harm interval includes zero. (b) WildChat medical

\Delta_{\mathrm{adj}}

against the fraction

q

of comparison requests sent to the strong model (19 thresholds;

q=0.2

0.5

, and

0.8

were fixed in the protocol,

q=0.5

as the primary point; the other 16 are post hoc). The two hollow RouteLLM points (

q=0.05

0.10

) are where medical requests almost never reach the strong model (

P(\mathrm{strong})=0.001

and

0.007

). Only

q=0.5

carries an interval (95% user-cluster bootstrap); no CIs exist elsewhere on the curve. RouteLLM’s gap shrinks toward both ends, but its per-request privacy loss

\varepsilon

does not (

1.18

q=0.2

against

1.00

0.5

); notdiamond’s gap reaches

+0.01

q=0.05

. On LMSYS at the 20% and 80% operating points (no CIs), RouteLLM keeps its sign in all four categories, while notdiamond turns positive at 80% in all four (harassment

+0.116

, self-harm

+0.043

, sexual

+0.026

, violence

+0.057

Fig. 3: Both routers differentiate sensitive MMLU subjects far more than an oracle that routes by realized accuracy gain at the same budget (RouterBench, MMLU subjects). Filled dots: each router’s length-matched gap

\Delta_{\mathrm{adj}}

of ( 1 ) in strong-model share between the group and the non-sensitive prompts (those in none of the four groups), with 95% percentile-bootstrap CIs over prompts (2,000 resamples); these gaps are pre-registered (E-8b measurement M2), not a primary endpoint. Hollow squares: an oracle with the router’s budget that sends the prompts with the largest realized accuracy gain to the strong model, breaking ties at random. The oracle, and hence the excess, is post hoc and uses realized per-prompt correctness; it is one router that maximizes accuracy at that budget, not a deployable router. Its gaps come from a single random tie-break and have no CI. Numbers give the absolute excess

|\Delta_{\mathrm{adj}}(\text{router})-\Delta_{\mathrm{adj}}(\text{oracle})|

, 0.10–0.34; orange marks the largest (RouteLLM, psychology:

-0.422

vs.

-0.085

). The routers’ gaps are 13–42 points and the oracle’s 1–11 points; we compare them in points, not as ratios, because the oracle’s smallest gaps (

-0.011

and

-0.012

) are close to zero. Against Mixtral-8x7B, part of the oracle’s gap reflects accuracy: the strong model’s mean accuracy gain on these subjects is 0.11, 0.12, 0.05, and 0.08 lower than on non-sensitive prompts (exploratory; not plotted). Parentheses give the number of MMLU subjects per group; human sexuality (

n=131

) has the widest CIs. Groups are academic subjects, not users’ own questions. Sources: E-8b/results.json , E-8b/posthoc_e8b.json .

Fig. 4: User-level AUC rises modestly over the evaluated horizons and remains below the topic-only null; the pre-registered endpoint is missed. With a user’s first 20 requests, the bill-only attacker A1 ( 5 ) on RouteLLM reaches AUC 0.707 (red; 95% CI [0.671, 0.740]), below the pre-registered

T{=}20

threshold of 0.75 (black dashed line). Data: WildChat-1M users with at least 10 requests, on the pre-registered E-5 user split; positives have at least 20% medical requests (LLM-judge labels), negatives none. The numbers under the ticks count positive test users (negatives: 6,471, 6,471, 2,935, 857). The blue band and the red bar are 95% percentile-bootstrap CIs over test users (1,000 resamples, stratified by label). The gray dashed line and light gray band show the topic-only null, a model prediction rather than data: the mean and the 2.5–97.5th percentiles of 200 simulations in which each request’s strong probability depends only on its topic class. The arrow at

T{=}50

spans the gap between the two bands, consistent with between-user heterogeneity (Remark 3 ). Eligibility changes with

T

, so the curve does not establish an asymptotic ceiling. Heterogeneity is measured as the intra-class correlation

\rho

of each user’s routing decisions: 0.35 among 13,022 negative users and 0.21 among 818 positive users. The domain router (Vela; an upper estimate, as its labels helped select what the judges read) was evaluated only for

T\leq 20

. Length only is drawn for

T\leq 20

because at

T{=}50

, with only 63 positives, it falls to 0.47. A random-score control gives 0.49–0.51. The RouteLLM point at

T{=}20

is the pre-registered primary endpoint (P2). The other

T

values, notdiamond, the domain router, length only, the random control, the null and

\rho

are pre-registered secondary analyses. As the LLM-judge label-validity gate failed, every medical result here is exploratory. Table VII lists the plotted values.

Fig. 5: Per-request gaps are cheap to close when labels are accurate; user-level inference is not. Each marker is RouteLLM on WildChat (judge-labeled medical vs. other requests, 50% operating point) with no defense or one post-processing defense; per-category parity on its own is orange and all other defenses blue. Lines join no defense and random mixing (

\eta=0.25,0.5

; solid), the per-user budget bands (

\pm 4\to\pm 2\to\pm 1\to

exact rate; dashed), and parity (cheap classifier

\to

perfect labels; dash-dot). Marker shape gives what deployment needs, as in Table VIII ; gray dashed lines in (b) mark chance and the undefended A1. The

x

axis, skill kept, is the router’s own expected benefit relative to random routing at the same budget (0 = random, 1 = undefended). It is the router’s self-assessment, not measured quality: on RouterBench, undefended RouteLLM is 1.30 accuracy points worse than random routing at the same budget (Section VII-D ). (a) Per-category parity with perfect labels removes the per-request gap (

|\Delta_{\mathrm{adj}}|=0.001

) at a skill of 0.99, but it is not deployable; with the cheap classifier the gap stays at 0.151. Whiskers are 95% user-cluster bootstrap CIs (1,000 resamples) shown as absolute values; the perfect-label interval spans zero, so its whisker runs from 0 to 0.028. (b) User-level AUC after

T{=}20

requests for the bill-only attacker A1 (filled; whiskers are 95% percentile bootstrap CIs over test users, 1,000 resamples) and the original position-agnostic itemized-log attacker A2 (hollow; CI not drawn); a solid segment joins each defense’s A1 and A2. The defense-study split has 185 positive and 2,891 negative test users; there the undefended A1 is 0.718, against 0.707 on the E-5 split. Only the exact per-user rate, with or without parity, brings A1 to chance (0.500, degenerate CI), at a skill of 0.23, and only because

T

is even (at

T{=}5

: 0.63 and 0.59). Itemized logs still accumulate under it: A2 is 0.53–0.54 at

T{=}20

and rises to 0.60–0.62 at

T{=}50

(Table VIII ). A post hoc odd-position attacker, not plotted here, reaches 0.730 under the exact rate and 0.667 with cheap-classifier parity first at

T{=}20

(Table IX ). Status: the defense circled in red in (b), cheap-classifier parity plus a

\pm 2

band, is the pre-registered primary endpoint and failed all four of its criteria; the undefended baseline and the other nine defenses are pre-registered secondary analyses; the exact-rate variants ( ∗ ) and the skill-kept measure are post hoc. As for every medical result, the labels come from an LLM judge that failed its pre-registered precision gate, so all results shown are exploratory (Section X ). Data: E-8 results.json and posthoc_e8.json .

Policy	Acc. (%)	vs. rand. (pp)	$/1k	Max leak	Acc. (%)	vs. rand. (pp)	$/1k	Max leak
	RouteLLM (GPT-4 vs. Mixtral-8x7B)				notdiamond (GPT-4 vs. GPT-3.5)
Always weak	54.7	$+0.00$	0.13	0.000	61.9	$+0.00$	0.24	0.000
Always strong	78.1	$+0.00$	3.29	0.000	78.1	$+0.00$	3.29	0.000
Router (no defense)	64.0	$-1.30$	2.12	0.422	70.6	$+1.09$	1.40	0.338
Random mixing, $\eta=0.5$	65.0	$-0.77$	1.89	0.223	70.2	$+0.50$	1.57	0.171
Pooled parity (pre-registered)	63.8	$-1.52$	2.04	0.137	70.6	$+1.16$	1.36	0.166

TABLE V: Routing policies in real units on RouterBench (36,497 prompts with per-model correctness and dollar cost). Accuracy: mean score of the selected model. vs. rand.: accuracy minus random routing at the same budget. Max leak: largest

|\Delta_{\mathrm{adj}}|

over the four sensitive groups of Fig. 3 .

Group	$n$	$\Delta_{\mathrm{adj}}$ [95% CI]	Oracle	Gain gap	$\Delta_{\mathrm{adj}}$ [95% CI]	Oracle
		RouteLLM (GPT-4 vs. Mixtral-8x7B)			notdiamond (GPT-4 vs. GPT-3.5)
Medical (7 subjects)	1,417	$-0.241$ [ $-0.266$ , $-0.217$ ]	$-0.079$	$-0.11$	$-0.294$ [ $-0.316$ , $-0.272$ ]	$-0.011$
Psychology (2)	1,157	$-0.422$ [ $-0.443$ , $-0.401$ ]	$-0.085$	$-0.12$	$-0.128$ [ $-0.157$ , $-0.098$ ]	$-0.023$
Human sexuality	131	$-0.357$ [ $-0.419$ , $-0.289$ ]	$-0.043$	$-0.05$	$-0.338$ [ $-0.408$ , $-0.267$ ]	$-0.059$
Law (3)	1,763	$-0.407$ [ $-0.422$ , $-0.391$ ]	$-0.106$	$-0.08$	$-0.328$ [ $-0.347$ , $-0.309$ ]	$-0.012$

TABLE VI: Leakage on RouterBench’s MMLU subjects (comparison set: all other RouterBench prompts). Routers’

\Delta_{\mathrm{adj}}

: pre-registered (E-8b measurement M2). Oracle (post hoc, as is any comparison with it): a router with the same budget that sends the prompts with the largest realized accuracy gain to the strong model, with ties broken at random; it is one router that maximizes accuracy at that budget. Gain gap (exploratory): the strong model’s mean accuracy gain on the group minus its gain on the comparison set.

	Pre-registered E-5 split (Fig. 4 )						Defense-study split (Table VIII )
		Length	RouteLLM	notdiamond	Domain	Topic-only		Length	RouteLLM			notdiamond
$T$	Pos.	only	A1	A1	router	null	Pos.	only	A1	A1+L	A2	A1	A2
5	409	0.555	0.684	0.620	0.795	0.705	409	0.566	0.675	0.679	0.696	0.632	0.691
10	409	0.558	0.695	0.642	0.851	0.772	409	0.566	0.690	0.695	0.713	0.642	0.708
20	179	0.537	0.707	0.661	0.916	0.832	185	0.560	0.718	0.731	0.739	0.680	0.736
50	63	0.473	0.736	0.656	–	0.904	62	0.521	0.769	0.803	0.800	0.674	0.712

TABLE VII: User-level inference on WildChat (medical). A test user is positive if

\geq

20% of their requests are medical and negative if none are; the attacker sees their first

T

requests. A1: count of strong-routed requests (an aggregated bill); A1+L: plus mean length; A2: itemized log, per-request likelihood ratio of the routing bit given the length decile. Topic-only null: the AUC if users differed only in topic mix. Domain router: an upper estimate, as its labels helped select what the judges read. Pos.: positive test users.

Defense	Needs	(pp)	kept	$\Delta_{\mathrm{adj}}$	A1, $T{=}20$	A2, $T{=}20$	A2, $T{=}50$
		$\Delta$ Budget	Skill	Per-request	User-level AUC
No defense	–	$+0.0$	1.00	$-0.331$	0.718	0.739	0.800
Random mixing, $\eta=0.25$	–	$+0.4$	0.75	$-0.250$	0.713	0.729	0.781
Random mixing, $\eta=0.5$	–	$+0.8$	0.50	$-0.164$	0.688	0.711	0.757
Per-category parity, cheap classifier	classifier	$+0.0$	0.99	$-0.151$	0.648	0.664	0.719
Per-category parity, perfect labels	(not deployable)	$+0.0$	0.99	$-0.001$	0.597	0.610	0.642

TABLE VIII: Defenses as post-processing of RouteLLM on WildChat (medical). Skill kept (post hoc): the router’s own expected benefit relative to random routing at the same budget (0 = random, 1 = undefended); it is the router’s self-assessment, not measured quality (Section VII-D ). It replaces the pre-registered retained benefit, the defended router’s expected benefit as a fraction of the undefended one’s, on which the primary defense scores 0.33. User-level columns are AUCs of the attackers of Table VII .

RouteLLM: exact rate
$T$	$m$	Test users positive/negative	Original A2 [95% CI]	A2-odd [95% CI]	Paired difference [95% CI]
5	3	409/6502	$0.514\ [0.486,0.542]$	$0.684\ [0.661,0.707]$	$+0.170\ [0.139,0.202]$
10	5	409/6502	$0.502\ [0.471,0.532]$	$0.702\ [0.678,0.725]$	$+0.201\ [0.165,0.235]$
20	10	185/2891	$0.532\ [0.492,0.573]$	$0.730\ [0.700,0.763]$	$+0.199\ [0.148,0.252]$
50	25	62/831	$0.618\ [0.547,0.683]$	$0.793\ [0.743,0.836]$	$+0.175\ [0.097,0.259]$
RouteLLM: D2-cand + exact rate

TABLE IX: Odd-position attacks on exact-rate defenses on WildChat medical labels: all configured routers, defenses, and horizons. This analysis is post hoc and exploratory; the medical-label validity gate failed.

T

counts total original requests, and

m=\lceil T/2\rceil

counts the observed odd-position decisions. Original A2 fits and scores all positions; A2-odd fits only training users’ odd positions and scores those within the same test prefix. Intervals are unadjusted 95% percentile CIs from 1,000 label-stratified test-user bootstrap resamples (seed 20260928), with training fits fixed. The difference is A2-odd minus A2, using paired resamples.

Explore similar work

Apr 17, 2026cs.CR

Privacy-Preserving LLMs Routing

Large language model (LLM) routing has emerged as a critical strategy to balance model performance and cost-efficiency by dynamically selecting services from various model providers. However, LLM routing adds an intermediate layer between users and LLMs, creating new privacy risks to user data. These privacy risks have not been systematically studied. Although cryptographic techniques such as Secure Multi-Party Computation (MPC) enable privacy-preserving computation, their protocol design and implementation remain under-explored, and naïve implementations typically incur prohibitive computational overhead. To address this, we propose a privacy-preserving LLM routing framework (PPRoute). PPRoute includes multiple strategies to speed up encoder inference and nearest neighbor search under the MPC and maintain the quality of LLM routing. First, PPRoute uses MPC-friendly operations to boost the encoder inference. Second, PPRoute uses a multiple-step model training algorithm to maintain routing quality despite the constraints of the encrypted domain. Third, PPRoute proposes an unsorted Top-k algorithm with

O(1)

communication complexity for secure sorting in model search, significantly reducing communication latency. Across different datasets, PPRoute achieves the performance of plaintext counterparts, while achieving approximately a 20

\times

speedup over naïve MPC implementations.

Xidong Wu, Yukuan Zhang, Yuqiong Ji +3

Department of Electrical and Computer Engineering, University of Pittsburgh · Department of Computer Science, University of Central Florida · Department of Computer and Information Science, University of Pennsylvania +2

Large Language Model Routing Diffie-Hellman

Ask PDF

Jun 17, 2026cs.LG

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

We present RouteJudge, an online pairwise preference evaluation framework for LLM routing systems, with a public platform available at https://routejudge.cn. Different from model-level response evaluation, RouteJudge focuses on router-level decision quality. For each user query, multiple routing strategies independently recommend candidate models under the same model pool and budget constraints. The selected model responses are then presented to users through anonymous pairwise comparisons, and the resulting user preferences are attributed back to the routing strategies behind the compared responses. Each evaluation record stores the query, routing decisions, model responses, preference labels, cost, latency, and task metadata, enabling preference-aware, cost-aware, and task-conditioned analysis of LLM routers. To support the continuous expansion of routing methods in RouteJudge, we further release ORBIT (Optimal Routing and Budgeted Inference Toolbox), a modular and extensible toolbox that standardizes the end-to-end workflow of LLM routing. ORBIT provides unified interfaces for benchmark loading, query representation, router implementation, budget-aware evaluation, and method comparison, allowing researchers to develop and evaluate routing algorithms under consistent protocols. It also serves as the submission and integration layer for RouteJudge: researchers can implement routing methods within ORBIT, validate them on existing routing benchmarks, and submit compatible routers for online preference-based evaluation. The code of ORBIT is available at https://github.com/LAMDA-Model-Reuse/ORBIT.

Guannan Lai, Haoran Hu, Han-Jia Ye

School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI

Large Language Model Routing

Ask PDF

Aug 7, 2026cs.CL

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Tao Feng, Fangxu Yu, Haozhen Zhang +9

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2

Large Language Model Routing Inference Cost Sentence Encoders Infrastructure

Ask PDF