Organizations: School of Artificial Intelligence, University of Science and Technology Beijing, Beijing, China · China Telecom Research Institute, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China
The rapid development of AI agent systems is leading to an emerging Internet of Agents, where specialized agents operate across local devices, edge nodes, private services, and cloud platforms. Although recent efforts have improved agent naming, discovery, and interaction, efficient request dispatch remains an open systems problem under latency, privacy, and cost constraints. In this paper, we present AgentGate, a lightweight structured routing engine for candidate-aware agent dispatch. Instead of treating routing as unrestricted text generation, AgentGate formulates it as a constrained decision problem and decomposes it into two stages: action decision and structural grounding. The first stage determines whether a query should trigger single-agent invocation, multi-agent planning, direct response, or safe escalation, while the second stage instantiates the selected action into executable outputs such as target agents, structured arguments, or multi-step plans. To adapt compact models to this setting, we further develop a routing-oriented fine-tuning scheme with candidate-aware supervision and hard negative examples. Experiments on a curated routing benchmark with several 3B--7B open-weight models show that compact models can provide competitive routing performance in constrained settings, and that model differences are mainly reflected in action prediction, candidate selection, and structured grounding quality. These results indicate that structured routing is a feasible design point for efficient and privacy-aware agent systems, especially when routing decisions must be made under resource-constrained deployment conditions.
Figures & tables
Fig. 1: Overview of the proposed agent network architecture and representative routing task examples handled by AgentGate.
Fig. 2: Detailed architecture of the AgentGate framework, illustrating the two-stage structured routing pipeline, heuristic fallback mechanisms, and confidence-aware backend selection.
Method
Action
Agent
Arg EM
Plan EM
Esc P
Esc R
Rule-based
0.7400
0.7650
0.7450
0.7800
0.0000
0.0000
Retrieve-rank
0.9250
0.9225
0.8600
0.8025
1.0000
0.8000
Tool-Calling (Qwen2.5-7B)
0.2400
0.4600
0.4600
0.8600
0.2400
1.0000
AgentGate (Qwen2.5-7B)
0.9425
0.8800
0.9325
0.8075
1.0000
1.0000
TABLE I: Comparison with alternative routing paradigms on the agentdns_runtime benchmark. Retrieve-rank is a competitive baseline, but AgentGate achieves better overall routing quality.
Fig. 3: Baseline comparison on the AgentDNS runtime benchmark. The left panel reports routing metrics, and the right panel reports escalation precision and recall.
Backbone
Action
Agent
Arg EM
Plan EM
Esc P
Esc R
Qwen2.5-3B
0.8975
0.7025
0.6925
0.8075
0.9231
0.4800
Qwen2.5-7B
0.9425
0.8800
0.9325
0.8075
1.0000
1.0000
Phi-3.5-mini
0.9625
0.9300
0.8600
0.8025
1.0000
0.8000
Mistral-7B
0.6500
0.2425
0.3500
0.8000
1.0000
0.8000
Llama2-7B
0.4500
0.5075
0.5675
0.8025
0.2475
1.0000
TABLE II: Backbone comparison under the AgentGate routing framework on the agentdns_runtime benchmark. Qwen2.5-7B gives the most balanced result set across routing and escalation metrics.
Backbone
Setting
Action
Agent
Arg EM
Qwen2.5-3B
Base
0.9531
0.9688
0.8750
Qwen2.5-3B
Fine-tuned
0.8906
0.9219
0.8906
Qwen2.5-7B
Base
0.9844
0.9844
0.9219
Qwen2.5-7B
Fine-tuned
0.9844
0.9375
0.8906
Mistral-7B
Base
0.9844
0.9844
0.8438
Mistral-7B
Fine-tuned
0.9844
0.9375
0.8906
TABLE III: Fine-tuned versus base models on two-stage routing probe. The main differences appear in argument exact match rather than in coarse routing accuracy.
Model
QPS
Route Lat. (ms)
Action Lat. (ms)
Selection Lat. (ms)
GPU Mem. (GiB)
Qwen2.5-3B
0.283
3531.5
709.2
3762.8
5.99
Qwen2.5-7B
0.317
3159.5
594.0
3420.4
14.56
Llama2-7B
0.120
8329.8
4745.5
4778.9
13.29
TABLE IV: System performance on a 20-sample runtime probe for the backbones with directly comparable two-stage measurements.
Fig. 4: Performance trade-offs across representative backbones.
Setting
Action
Agent
Arg EM
Plan EM
Esc P
Esc R
Qwen2.5-7B edge-only
1.0000
0.9400
0.9400
0.8600
1.0000
1.0000
Qwen2.5-7B hybrid
0.9800
0.9400
0.9000
0.8600
1.0000
0.9167
Qwen2.5-3B hybrid
0.9800
0.9400
0.9000
0.8600
1.0000
0.9167
Mistral-7B hybrid
0.9800
0.9400
0.9000
0.8600
1.0000
0.9167
Llama2-7B hybrid
0.9000
0.8600
0.8200
0.8600
0.7692
0.8333
TABLE V: Real edge–cloud routing results. The cloud backend uses MiniMax M2.5. Under forced fallback, most hybrid results approach the cloud-only line.
LLM agent routing selects models, tools, skills, or collaboration workflows for each task or step to balance capability, cost, latency, and reliability.
Beijing University of Posts and Telecommunications, Beijing, China · Fudan University, Shanghai Innovation Institute, Shanghai, China · Nanyang Technological University, Singapore