Organizations: School of Artificial Intelligence, University of Science and Technology Beijing, Beijing, China · China Telecom Research Institute, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China
The rapid development of AI agent systems is leading to an emerging Internet of Agents, where specialized agents operate across local devices, edge nodes, private services, and cloud platforms. Although recent efforts have improved agent naming, discovery, and interaction, efficient request dispatch remains an open systems problem under latency, privacy, and cost constraints. In this paper, we present AgentGate, a lightweight structured routing engine for candidate-aware agent dispatch. Instead of treating routing as unrestricted text generation, AgentGate formulates it as a constrained decision problem and decomposes it into two stages: action decision and structural grounding. The first stage determines whether a query should trigger single-agent invocation, multi-agent planning, direct response, or safe escalation, while the second stage instantiates the selected action into executable outputs such as target agents, structured arguments, or multi-step plans. To adapt compact models to this setting, we further develop a routing-oriented fine-tuning scheme with candidate-aware supervision and hard negative examples. Experiments on a curated routing benchmark with several 3B--7B open-weight models show that compact models can provide competitive routing performance in constrained settings, and that model differences are mainly reflected in action prediction, candidate selection, and structured grounding quality. These results indicate that structured routing is a feasible design point for efficient and privacy-aware agent systems, especially when routing decisions must be made under resource-constrained deployment conditions.
Figures & tables
Fig. 1: Overview of the proposed agent network architecture and representative routing task examples handled by AgentGate.
Fig. 2: Detailed architecture of the AgentGate framework, illustrating the two-stage structured routing pipeline, heuristic fallback mechanisms, and confidence-aware backend selection.
Method
Action
Agent
Arg EM
Plan EM
Esc P
Esc R
Rule-based
0.7400
0.7650
0.7450
0.7800
0.0000
0.0000
Retrieve-rank
0.9250
0.9225
0.8600
0.8025
1.0000
0.8000
Tool-Calling (Qwen2.5-7B)
0.2400
0.4600
0.4600
0.8600
0.2400
1.0000
AgentGate (Qwen2.5-7B)
0.9425
0.8800
0.9325
0.8075
1.0000
1.0000
TABLE I: Comparison with alternative routing paradigms on the agentdns_runtime benchmark. Retrieve-rank is a competitive baseline, but AgentGate achieves better overall routing quality.
Fig. 3: Baseline comparison on the AgentDNS runtime benchmark. The left panel reports routing metrics, and the right panel reports escalation precision and recall.
Backbone
Action
Agent
Arg EM
Plan EM
Esc P
Esc R
Qwen2.5-3B
0.8975
0.7025
0.6925
0.8075
0.9231
0.4800
Qwen2.5-7B
0.9425
0.8800
0.9325
0.8075
1.0000
1.0000
Phi-3.5-mini
0.9625
0.9300
0.8600
0.8025
1.0000
0.8000
Mistral-7B
0.6500
0.2425
0.3500
0.8000
1.0000
0.8000
Llama2-7B
0.4500
0.5075
0.5675
0.8025
0.2475
1.0000
TABLE II: Backbone comparison under the AgentGate routing framework on the agentdns_runtime benchmark. Qwen2.5-7B gives the most balanced result set across routing and escalation metrics.
Backbone
Setting
Action
Agent
Arg EM
Qwen2.5-3B
Base
0.9531
0.9688
0.8750
Qwen2.5-3B
Fine-tuned
0.8906
0.9219
0.8906
Qwen2.5-7B
Base
0.9844
0.9844
0.9219
Qwen2.5-7B
Fine-tuned
0.9844
0.9375
0.8906
Mistral-7B
Base
0.9844
0.9844
0.8438
Mistral-7B
Fine-tuned
0.9844
0.9375
0.8906
TABLE III: Fine-tuned versus base models on two-stage routing probe. The main differences appear in argument exact match rather than in coarse routing accuracy.
Model
QPS
Route Lat. (ms)
Action Lat. (ms)
Selection Lat. (ms)
GPU Mem. (GiB)
Qwen2.5-3B
0.283
3531.5
709.2
3762.8
5.99
Qwen2.5-7B
0.317
3159.5
594.0
3420.4
14.56
Llama2-7B
0.120
8329.8
4745.5
4778.9
13.29
TABLE IV: System performance on a 20-sample runtime probe for the backbones with directly comparable two-stage measurements.
Fig. 4: Performance trade-offs across representative backbones.
Setting
Action
Agent
Arg EM
Plan EM
Esc P
Esc R
Qwen2.5-7B edge-only
1.0000
0.9400
0.9400
0.8600
1.0000
1.0000
Qwen2.5-7B hybrid
0.9800
0.9400
0.9000
0.8600
1.0000
0.9167
Qwen2.5-3B hybrid
0.9800
0.9400
0.9000
0.8600
1.0000
0.9167
Mistral-7B hybrid
0.9800
0.9400
0.9000
0.8600
1.0000
0.9167
Llama2-7B hybrid
0.9000
0.8600
0.8200
0.8600
0.7692
0.8333
TABLE V: Real edge–cloud routing results. The cloud backend uses MiniMax M2.5. Under forced fallback, most hybrid results approach the cloud-only line.
LLM agent routing selects models, tools, skills, or collaboration workflows for each task or step to balance capability, cost, latency, and reliability.
LLM agents achieve strong performance on complex reasoning tasks but incur high latency and compute cost. In practice, many queries fall within the capability boundary of cutting-edge LLMs and do not require full agent execution, making effective routing between LLMs and agents a key challenge. We study the problem of routing queries between lightweight LLM inference and full agent execution under realistic cold-start settings. To address this, we propose BoundaryRouter, a training-free routing framework that uses early behavioral experience and rubric-guided reasoning to decide whether to answer a query with direct LLM inference or escalate to an agent. BoundaryRouter builds a compact experience memory by executing both systems on a shared seed set and retrieves similar cases at inference time to guide routing decisions. To evaluate this method, we introduce RouteBench, a benchmark covering in-domain, paraphrased, and out-of-domain route settings. Experiments show that BoundaryRouter reduces inference time by 60.6% compared to the agent while improving performance by 28.6% over direct LLM inference, outperforming prompt-based and retrieval-only routing by an average of 37.9% and 8.2%, respectively.
Yimin Wang, Jiahao Qiu, Xuan Qi +6
University of Michigan · Shanghai Jiao Tong University · AI Lab, Princeton University +3
Agent performance depends on both the underlying model and the harness that manages its tool use and execution. Selecting a suitable pair requires accounting for their compatibility, yet training samples may cover only a subset of the growing combination space. We introduce HM-Router, a routing method that jointly selects a model and harness for each query. It learns separate model and harness representations shared across routes, with an interaction term inspired by canonical polyadic (CP) tensor decomposition to capture how their compatibility varies with the query. This sharing allows training samples from observed pairs to inform predictions for unobserved combinations. We curate a benchmark from 12 public agent benchmarks, covering 293 routes, 73 models, and 25 harnesses. HM-Router exceeds the strongest evaluated learned baseline by 7.3 percentage points in mean routing accuracy and leads at all seven evaluated cost budgets on the six-benchmark subset. When 90% of routes have their training outcomes withheld, allowing unobserved combinations improves normalized accuracy by 15.8 points over restricting the same router to observed routes. HM-Router has also demonstrated training sample efficiency for new routes and components and generalization to unseen benchmarks. Our code and data are open-sourced at https://github.com/hmarkc/HM-Router.
Hao Mark Chen, Royson Lee, Yasuyuki Okoshi +3
Imperial College London · Samsung · Institute of Science Tokyo
Serving web queries at scale with multi-agent systems requires jointly configuring large language models, collaboration modes, agent counts, interaction rounds, and skills. However, the resulting configuration space grows combinatorially, while sparse execution rewards provide limited fine-grained supervision for interdependent planning decisions. We propose EvoRouter, a self-evolving multi-agent routing framework driven by insight-guided on-policy distillation. A dedicated expert periodically synthesizes performance-cost feedback from configuration trajectories across tasks into reusable conditional routing insights about capability requirements, returns from resource scaling, and interactions among configuration choices. A privileged teacher combines these evolving insights with current execution feedback to provide dense token-level supervision for the router's own planning trajectories through on-policy distillation. Cost-aware reinforcement learning jointly optimizes task performance and computational cost. Together, these training signals internalize execution insights into the routing policy. The updated policy generates new trajectories that support subsequent insight extraction, forming an execution-driven self-improvement loop. We evaluate EvoRouter on 19 in-domain datasets spanning five domains and two out-of-domain benchmarks. EvoRouter achieves an average in-domain accuracy of 68.69%, outperforming the strongest routing baseline by 7.23 percentage points while reducing inference compute cost by 50.95%. Our code is open-sourced at https://github.com/CenYangtze/EvoRouter.
Yitao Xiao, Xu Zhang, Ye Cheng +6
Beijing University of Posts and Telecommunications, Beijing, China · Fudan University, Shanghai Innovation Institute, Shanghai, China · Nanyang Technological University, Singapore