RouteAlign: Aligning Task Requirements with Model Profiles for LLM Service Selection

Jihao Zhao; Ding Chen; Yanfang Chen; Bo Tang; Feiyu Xiong; Junchi Yan; Zhiyu Li

cs.SENov 5, 2026

RouteAlign: Aligning Task Requirements with Model Profiles for LLM Service Selection

Authors: Jihao Zhao, Ding Chen, Yanfang Chen, Bo Tang, Feiyu Xiong, Junchi Yan, Zhiyu Li

Organizations: MemTensor (Shanghai) Technology Co., Ltd., Shanghai, China · Renmin University of China, Beijing, China · Shanghai Jiao Tong University, Shanghai, China

Abstract

LLM service selection requires performance evidence that reflects the demands of the current task. Aggregate scores obscure differences in capability and difficulty, while semantically related examples can involve different combinations of skills. We present RouteAlign, which uses task requirements to structure both the measurement of model performance and the selection of services. Task requirements describe capability demand and difficulty, enabling alignment with corresponding model performance measurements. Historical requirements determine how outcomes contribute to global profiles and local capability estimates; incoming requirements determine how those measurements inform the current decision. A shared scorer integrates the evidence, and a generation-cost preference controls selection among eligible services. Evaluation across diverse tasks and heterogeneous LLM services shows that RouteAlign extends the empirical quality-cost frontier toward higher task quality. Comparisons with open Jev-style decision backends show consistent quality gains, with lower generation cost than their strongest tested configuration. Ablations support the contribution of local performance evidence, while catalog-update experiments demonstrate effective reuse of the fitted scorer with refreshed model measurements.

Figures & tables

Figure 1: Conceptual comparison of query-model matching and requirement-profile alignment for LLM service selection. The task combines coding, reasoning, and data analysis with weights of 0.50, 0.33, and 0.17 at difficulty L3. The model profiles of services A, B, and C express performance in the same capability coordinates.

Figure 2: RouteAlign for LLM service selection. Offline preparation uses training records to construct model profiles, a historical evidence bank, and a shared scorer. The same profiler supplies requirements for outcome profiles and incoming tasks. Response assessments use separate annotations in the common schema. At invocation time, requirements align with global profiles and local evidence, metadata determines eligibility, and the cost preference shapes selection.

Selection policy	AIME	AH	GPQA	HLE	LCB	LMB	MMP	SQA	SWE	$\tau^{2}$	S-Avg.	D-Avg.	Cost ($)
	Per-dataset score (%)										Overall (%)
Random Router	61.11	68.14	73.33	10.34	62.78	64.86	84.44	34.13	23.33	40.48	47.38	52.29	51.11
KNNRouter	61.11	71.46	58.33	13.73	63.41	64.86	82.89	53.93	28.67	57.14	55.07	55.55	34.96
SVMRouter	77.78	76.99	55.00	9.72	58.36	72.97	83.78	54.01	16.67	41.67	53.80	54.69	6.98
MLPRouter	72.22	70.80	61.67	18.52	67.82	59.46	82.78	50.69	24.00	70.24	55.24	57.82	42.44
MM-Embed-S	66.67	69.69	81.67	15.74	79.18	70.27	83.33	53.00	28.00	53.57	56.78	60.11	56.28

Table 1: Task quality and generation cost across ten benchmarks. S-Avg. and D-Avg. denote Sample-Avg and Dataset-Avg. Bold marks the highest quality or lowest generation cost in each column.

	Quality (%)
Candidate or policy	S-Avg.	D-Avg.	Cost ($)
Intern-S1	37.74	43.03	13.85
DeepSeek V3-0324	40.41	42.60	4.78
DeepSeek V3.1-Terminus	44.18	51.72	9.43
DeepSeek R1-0528	47.85	55.57	60.75
Qwen3-235B	53.81	54.70	6.98

Table 2: Comparison with direct invocation of individual candidates. Scores (%) and generation costs (US dollars) use available records. GPT-5 covers all test requests; GPT-5-Chat excludes

\tau^{2}

-bench.

Figure 3: Task quality versus generation cost for the fixed-catalog configurations. Grey circles denote direct candidate invocation; colored markers denote selection policies. Dashed curves are schematic guides to the Pareto frontiers. Cost includes answer generation only; direct-candidate summaries use available records.

Decision backend	D-Avg.	S-Avg.	Cost ($)
Kev-0.8B	51.83	49.65	39.33
OpenJev-mjdileep-0.8B	47.44	45.85	52.57
Open-Jev-2B	48.93	48.86	39.52
Decider-2B	55.00	47.11	95.63
SemIf-4B	53.43	51.82	29.32
Kev-4B	61.43	55.43	102.97

Table 3: Open Jev-style backends with direct decision scoring (

\rho=1

), plus the strongest configuration across all three mixtures. Cost covers recorded candidate generation. Bold marks the highest aggregate quality.

Figure 4: Aggregate quality of nine open Jev-style backends under three scoring mixtures. Each row uses one backend for requirement prediction;

\rho

weights its decision score, with explicit requirement-profile compatibility weighted by

1-\rho

. Vertical lines mark full RouteAlign. All 27 configurations use the same held-out requests and catalog.

Removed evidence or preference	D-Avg.	$\Delta$ D	S-Avg.	$\Delta$ S	Cost ($)
Full RouteAlign	68.64	0.00	63.07	0.00	110.42
All local quality evidence	65.52	$-3.12$	61.41	$-1.66$	106.24
Scalar neighborhood estimate	68.03	$-0.61$	62.88	$-0.19$	110.41
Sample-category prior	68.05	$-0.59$	62.93	$-0.14$	111.85
Local capability conditioning	68.12	$-0.52$	63.08	$+0.01$	110.96
Relative local evidence	68.18	$-0.46$	63.22	$+0.15$	109.42

Table 4: Full ablation results for RouteAlign.

\Delta

denotes the change from full RouteAlign in percentage points. Bold identifies the reference configuration.

Selection policy	Total (s)	Per query (ms)
Random Router	0.0011	0.0003
KNNRouter	76.00	20.33
SVMRouter	83.86	22.43
MLPRouter	75.56	20.21
MM-Embed-S	4.23	1.13
MM-Embed-L	9.42	2.52

Table 5: Selection-stage runtime over 3,738 test requests. RouteAlign uses cached requirement profiles and embeddings. Candidate answer generation is excluded.

Stage	Parent	Event	Candidates
T0	n/a	Initial candidate pool	7
T1	T0	Add DeepSeek R1-0528	8
T2	T1	Add Qwen3-235B Thinking-2507	9
T3	T2	Add Claude Sonnet 4	10
T4	T3	Add Gemini 2.5 Pro	11
T5	T4	Add GPT-5	12

Table 6: Service catalog update protocol. Each configuration is defined by its parent and update event; candidates are indexed by their model configurations.

Selection policy	T0	T1	T2	T3	T4	T5	T6	T7	T8	Mean
	Main sequence							T6 branches
RouteLLM	54.70	54.70	54.70	54.70	54.70	66.42	66.42	54.70	66.42	58.61
RouterDC	55.79	55.56	58.69	58.69	58.69	65.24	65.24	58.69	65.24	60.20
AvengersPro	52.94	52.69	54.52	58.52	57.81	60.49	61.88	60.58	64.72	58.24
RouteAlign-Frozen	60.36	61.33	62.26	63.48	64.18	68.55	68.60	64.08	68.64	64.61

Table 7: Selection quality under controlled catalog changes. Entries are Dataset-Avg (%); Mean weights the nine configurations equally. T7 and T8 independently branch from T6. Bold marks the highest score in each column.

Capability coverage	Retained sources
C1: Task understanding	IFEval [ 49 ] , EmoryNLP, MELD
C2: Domain knowledge	MedQA, C-Eval
C3: Reasoning	MATH-500, MathBench, KOR-Bench, K&K ( kandk ), BBH [ 37 ]
C4: Software engineering	MBPP, HumanEval [ 6 ]
C5: Long-context synthesis	LongBench v2 [ 3 ]
C6: Structured information	FinQA [ 8 ]

Table 8: Sources retained for profiler SFT after excluding datasets in the routing test workload. The groups indicate sampling coverage; each task receives its own capability mixture and difficulty annotation.

Dataset	Train	Test
AIME	42	18
ArenaHard	525	226
GPQA	138	60
HLE	1511	648
LiveCodeBench	738	317
LiveMathBench	84	37

Table 9: Training and test set sizes.

Candidate checkpoint	Records	Datasets	Selections
intern-s1	3737	10	2
deepseek-v3-0324	3737	10	12
deepseek-v3.1-terminus	3737	10	37
deepseek-r1-0528	3737	10	181
qwen3-235b-a22b-2507	3737	10	807
qwen3-235b-a22b-thinking-2507	3737	10	89

Table 10: Candidate coverage and fixed-catalog selection counts. The selected candidates have observed outcomes for all 3,738 test requests.

Selection rule	D-Avg.	S-Avg.	Cost ($)
Full RouteAlign	68.64	63.07	110.42
Sample-category mean	66.35	61.49	126.50
Category and difficulty mean	66.42	61.45	110.68

Table 11: Comparison with selection by category means. Each reference policy selects the candidate with the highest empirical mean score in the predicted category or category-and-difficulty group, without the shared scorer. Quality is in percent.

Figure 5: Selection quality under controlled catalog updates. (a) Dataset-Avg along T0 through T6, with candidate counts in parentheses. (b) Changes in percentage points for the independent T7 removal and T8 repricing branches relative to T6. RouteAlign-Frozen retains the scorer fitted on the seven T0 candidates and refreshes candidate evidence after each update.

Project	Sizes	Decision mechanism	Checkpoint identifiers
Kev	0.8B, 4B, 9B	LoRA and pointer head	jaredpalmer/kev-0.8b , jaredpalmer/kev-4b , jaredpalmer/kev-9b
Open-Jev	2B, 9B	LoRA and scalar head	ZefanCai/Open-Jev-2B , ZefanCai/Open-Jev-9B
Decider	2B, 4B	Full fine-tuning and letter logits	Mapika/decider-2b , Mapika/decider-4b
SemIf	4B	Pretrained letter logits	Qwen/Qwen3.5-4B
OpenJev-mjdileep	0.8B	Pretrained affirmative-answer logits	Qwen/Qwen3.5-0.8B

Table 12: Open implementations used in the Jev-style comparison. Project names link to their repositories, and checkpoint identifiers specify the evaluated Qwen3.5-based models. SemIf and OpenJev-mjdileep use pretrained base checkpoints directly.

Backend	D	S	Cost	D	S	Cost	D	S	Cost	Time (ms)
	$\rho=0$			$\rho=0.5$			$\rho=1$
Kev-0.8B	48.08	47.14	36.58	51.39	49.02	38.11	51.83	49.65	39.33	1268.9
OpenJev-mjdileep-0.8B	48.76	45.77	48.61	49.86	45.80	51.56	47.44	45.85	52.57	969.7
Open-Jev-2B	47.70	46.54	50.21	51.78	49.83	49.90	48.93	48.86	39.52	1544.0
Decider-2B	56.42	46.63	123.39	56.11	47.27	108.88	55.00	47.11	95.63	593.4
SemIf-4B	49.35	51.47	30.99	54.29	52.62	33.55	53.43	51.82	29.32	1727.8

Table 13: All scoring mixtures for the open Jev-style backends. D and S denote Dataset-Avg and Sample-Avg (%); cost is in US dollars. The last column is mean requirement-prediction and scoring time on one A800, shared by the three offline replays.

Backend or policy	AIME	AH	GPQA	HLE	LCB	LMB	MMP	SQA	SWE	$\tau^{2}$
Kev-0.8B	61.11	69.03	73.33	8.80	59.94	54.05	82.33	43.76	20.67	45.24
OpenJev-mjdileep-0.8B	55.56	69.03	66.67	12.35	56.15	62.16	82.67	34.05	16.67	19.05
Open-Jev-2B	55.56	67.92	63.33	8.18	59.94	62.16	81.67	44.30	15.33	30.95
Decider-2B	88.89	70.35	70.00	15.43	70.98	62.16	83.22	29.43	22.67	36.90
SemIf-4B	66.67	65.04	78.33	9.26	58.36	59.46	83.33	50.46	8.67	54.76
Kev-4B	83.33	73.89	81.67	13.89	77.29	64.86	83.22	49.08	34.67	52.38

Table 14: Per-dataset quality (%) with direct decision scoring (

\rho=1

). Bold marks the highest score in each column among these configurations and RouteAlign.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 366 papers

COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use
Oct 20, 2026Amin Rezaei, Pouya ParsaLLM Routing LLM Agent Routing
University of Minnesota
Designing Service Systems from Textual Evidence
Mar 11, 2026Ruicheng Ao, Hongyu Chen, Siyang Gao +2Best-Arm Identification AI-Assisted Decision Making
Institute for Data, Systems, and Society, Massachusetts Institute of Technology. · School of Data Science, City University of Hong Kong. · Department of Decision Analytics and Operations, City University of Hong Kong.
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
Jun 17, 2026Guannan Lai, Haoran Hu, Han-Jia YeLLM Evaluation Pairwise Preference Evaluation
School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI

Oct 20, 2026J/K move · Enter open · S save

COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use

Amin Rezaei, Pouya Parsa

University of Minnesota

LLM Routing LLM Agent Routing Tool-Augmented Language Model Agents

Open paper AskPDF arXiv

LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change. We propose COMPASS, an agentic router in which an LLM follows a routing recipe, gathers performance and cost evidence through tools, and optionally inspects candidate answers before selecting a model. Our evaluations on LLMRouterBench and DeepSWE show that agentic routing can improve accuracy and lower model-call cost relative to specialized and trained routing baselines, without routing-specific training. On LLMRouterBench, COMPASS reaches 69.85% accuracy at $139.57, compared with RLCascadeRouter's 69.14% at $180.22. On DeepSWE under fixed five-fold cross-validation, it reaches 78.1% at $215, compared with 74.1% at $500 for the best single model. COMPASS also supports routing across benchmarks and adapts to an expanded model pool by reading new evaluation records, without changing the router's weights or prompt.

Abstract

LLM Routing

Explore similar work

COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use

Designing Service Systems from Textual Evidence

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing