When an LLM-based agent improves on a task, is the gain from the model itself or from the reasoning paradigm wrapped around it? We study this question by comparing six inference-time paradigms, namely Direct, CoT, ReAct, Plan-Execute, Reflection, and ReCode, across four frontier LLMs and ten benchmarks, yielding roughly 18,000 runs. We find that reasoning structure helps dramatically on some tasks but hurts on others: ReAct improves over Direct by 44pp on GAIA, while CoT degrades performance by 15pp on HumanEval. No single paradigm dominates, and oracle per-task selection beats the best fixed paradigm by 17.1pp on average. Motivated by this complementarity, we propose a select-then-solve approach: before answering each task, a lightweight embedding-based router selects the most suitable paradigm. Across four models, the router improves average accuracy from 47.6% to 53.1%, outperforming the best fixed paradigm at 50.3% by 2.8pp and recovering up to 37% of the oracle gap. In contrast, zero-shot self-routing only works for GPT-5 at 67.1% and fails for weaker models, all trailing the learned router. Our results argue that reasoning paradigm selection should be a per-task decision made by a learned router, not a fixed architectural choice.
Figures & tables
Figure 1: Direct prompting (gray), best single paradigm per dataset (colored), and oracle per-task selection (blue) for GPT-5. The best paradigm differs across tasks, and the oracle substantially exceeds any fixed choice, motivating our select-then-solve approach.
Paradigm
Key Idea
Control
Tools
LM Calls
Budget
Direct
Free-form answer
None
No
1
—
CoT
Step-by-step reasoning
Instructed
No
1
—
ReAct
Thought–action loop
Orchestrated
Yes
1–15
Tmax=15
Plan-Execute
Plan, then act
Orchestrated
Yes
2–16
no replanning
Reflection
Critique and revise
Orchestrated
Yes
3–45
Mmax=3
ReCode
Code as reasoning
Substrate
Yes
2–10
hierarchical
Table 1: Reasoning paradigms studied in the main paper, organized by the degree of reasoning control and tool access.
Model
Paradigm
HEval
M500
AIME
HPQA
NQ
MMLU
HLE
GAIA
SEAL
τ -b
Avg
GPT-5
Direct
85.0
86.0
90.0
62.0
37.0
88.0
20.0
28.0
16.0
5.9
51.8
CoT
70.0
84.0
86.7
58.0
26.0
89.0
16.0
40.0
34.0
9.8
51.3
ReAct
86.0
80.0
86.7
70.0
27.0
89.0
26.0
72.0
28.0
5.9
57.1
Plan-Exec
35.0
75.0
90.0
69.0
27.0
83.0
22.0
68.0
48.0
7.8
52.5
Reflect.
85.0
69.0
75.0
67.0
29.0
85.0
22.0
58.0
46.0
5.9
54.2
ReCode
95.0
86.0
88.3
61.0
35.0
84.0
20.0
38.0
10.0
7.8
52.5
Table 2: Complete success rates (%) across all models, paradigms, and datasets. Best and second best paradigm per (model, dataset) are highlighted. Avg is the unweighted mean across all ten datasets.
Model
Direct
Best-Single
Oracle
Oracle Gap
GPT-5
60.3%
61.8% (ReAct)
74.1%
+12.4pp
Gemini-Flash
48.0%
56.1% (CoT)
75.2%
+19.1pp
Qwen3-Max
41.3%
46.5% (ReAct)
72.5%
+21.6pp
Qwen3-30B
21.9%
31.7% (ReAct)
55.7%
+24.1pp
Average
42.9%
49.0%
69.4%
+17.1pp
Table 3: Oracle analysis: per-task best paradigm selection achieves substantially higher accuracy than any single paradigm. Oracle Gap = Oracle − Best-single.
Figure 2: The select-then-solve pipeline. A Paradigm Selector encodes, classifies, and routes each task to one of six paradigms. Only the selected paradigm is executed.
Figure 3: Router comparison across four models. The embedding router (green) consistently outperforms Direct and Best-single baselines. Self-routing (red) shows mixed results. The oracle (blue) shows substantial remaining headroom.
Model
Direct
CoT
ReAct
Plan-Exec
ReCode
Reflect.
Router
GPT-5
65.9
9.6
14.4
7.9
1.3
0.9
Gemini
73.4
13.5
6.5
0.4
6.1
0.0
Qwen3-Max
58.9
19.7
2.6
7.0
10.9
0.9
Qwen3-30B
47.2
26.2
15.7
3.1
6.6
1.3
Self
GPT-5
65.1
18.8
15.3
0.4
0.4
0.0
Gemini
72.1
10.9
16.6
0.0
0.4
0.0
Table 4: Paradigm distribution (%) predicted by the learned router vs. zero-shot self-routing on the test set. Self-routing shows degenerate biases; the learned router produces diverse, model-adapted distributions.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Domain
Eval Instances
Metric
Notes
HumanEval
Code
100
Execution
function completion
MATH500
Math
100
Numeric
multi-step derivation
AIME
Math
60
Numeric
olympiad-style problems
HotpotQA
QA
100
Text match
multi-hop QA
NQ
QA
100
Text match
factoid QA
MMLU
Knowledge
100
Letter match
broad subject coverage
Appendix
Table 5: Evaluation sets used in the paper. “Eval instances” refers to the number of tasks actually run per model-paradigm pair.
Paradigm
Avg LM Calls
Avg Tool Calls
Tools?
Multi-turn?
Key Mechanism
Direct
1.0
0
No
No
Single-shot generation
CoT
1.0
0
No
No
Step-by-step verbal reasoning
ReAct
1–15
0–14
Yes
Yes
Thought-action-observation loop
Plan-Exec
2–16
0–15
Yes
Yes
Plan first, then execute with tools
Reflection
3–45
0–42
Yes
Yes
Solve, critique, revise (up to 3 rounds)
ReCode
2–10
0–8
Yes
Yes
Recursive code decomposition
Appendix
Table 6: Detailed comparison of the six paradigms. LM calls and tool calls are per-task averages on GPT-5.
Method
GPT-5
Gemini
Qwen3-Max
Qwen3-30B
Avg
Base
Direct
60.3
55.5
49.8
24.9
47.6
Best-single
62.4
55.5
50.7
32.8
50.3
HC
Handcrafted (LR)
59.4
58.5
51.5
34.9
51.1
Handcrafted (MLP)
62.4
55.5
53.3
37.1
52.1
Emb
Embedding (LR)
64.2
61.9
53.3
32.3
52.9
Combined (LR)
64.2
61.0
54.6
32.8
53.1
Appendix
Table 7: Downstream task accuracy (%) on the held-out test set. Each router selects a paradigm per task (including Direct); accuracy is the fraction of tasks solved correctly under the selected paradigm. Best router per model is bolded .
Figure 4: Cost-effectiveness scatter plot: success rate vs. average tokens per task. Points closer to the top-left corner represent better cost-efficiency.
Figure 5: Router comparison across four models. The embedding router (green) consistently outperforms Direct and Best-single baselines. Self-routing (red) fails to improve over Direct on most models. The oracle (blue) shows substantial remaining headroom.
Figure 6: Oracle gap recovery across models. Each group shows the progression from Direct (gray) through handcrafted router (orange) and embedding router (green) to the Oracle upper bound (blue). The embedding router consistently narrows the gap, with the largest recovery on Gemini.
Figure 7: Paradigm distribution comparison: learned router (left) vs. zero-shot self-routing (right). The learned router produces diverse, model-adapted distributions, while self-routing shows degenerate biases (GPT-5 selects Direct 100%; Qwen3-30B over-selects ReAct at 48%).
Figure 8: Success rate heatmap for GPT-5 across paradigms (rows) and datasets (columns). Clear task-paradigm interactions are visible: no single row dominates all columns.
Figure 9: Jaccard similarity between paradigm success sets (aggregated across all models). Lower values indicate greater complementarity; paradigms solve different subsets of tasks.
LLM agents achieve strong performance on complex reasoning tasks but incur high latency and compute cost. In practice, many queries fall within the capability boundary of cutting-edge LLMs and do not require full agent execution, making effective routing between LLMs and agents a key challenge. We study the problem of routing queries between lightweight LLM inference and full agent execution under realistic cold-start settings. To address this, we propose BoundaryRouter, a training-free routing framework that uses early behavioral experience and rubric-guided reasoning to decide whether to answer a query with direct LLM inference or escalate to an agent. BoundaryRouter builds a compact experience memory by executing both systems on a shared seed set and retrieves similar cases at inference time to guide routing decisions. To evaluate this method, we introduce RouteBench, a benchmark covering in-domain, paraphrased, and out-of-domain route settings. Experiments show that BoundaryRouter reduces inference time by 60.6% compared to the agent while improving performance by 28.6% over direct LLM inference, outperforming prompt-based and retrieval-only routing by an average of 37.9% and 8.2%, respectively.
Yimin Wang, Jiahao Qiu, Xuan Qi +6
University of Michigan · Shanghai Jiao Tong University · AI Lab, Princeton University +3
LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change. We propose COMPASS, an agentic router in which an LLM follows a routing recipe, gathers performance and cost evidence through tools, and optionally inspects candidate answers before selecting a model. Our evaluations on LLMRouterBench and DeepSWE show that agentic routing can improve accuracy and lower model-call cost relative to specialized and trained routing baselines, without routing-specific training. On LLMRouterBench, COMPASS reaches 69.85% accuracy at $139.57, compared with RLCascadeRouter's 69.14% at $180.22. On DeepSWE under fixed five-fold cross-validation, it reaches 78.1% at $215, compared with 74.1% at $500 for the best single model. COMPASS also supports routing across benchmarks and adapts to an expanded model pool by reading new evaluation records, without changing the router's weights or prompt.
Large language models often apply complex reasoning indiscriminately, leading to overthinking on simple tasks and underthinking on harder ones. We present Route-and-Generate (Rouge), a hierarchical training framework that enables LLMs to adaptively decide when and how much reasoning is required for each input. Our approach decouples reasoning effort allocation from answer generation by introducing a routing policy that allocates a fixed sampling budget across multiple reasoning formats, while the base model is optimized with GRPO using per-sample rewards. To promote efficient reasoning, we employ a rank-based reward that favors shorter correct solutions, and update the routing policy via a closed-form KL-regularized policy improvement step distilled into a neural router. This design encourages concise answers on easy problems while reserving longer reasoning for harder ones, achieving improved accuracy-efficiency trade-offs across multiple reasoning benchmarks.
Tien-Phat Nguyen, Truong Nguyen, Thanh-Thien Le +4
Hanoi University of Science and Technology, Vietnam · Monash University, Australia · New York University, USA +1