A model is only half of an AI agent system. The other half is the inference strategy: the reasoning procedure that turns model calls into a response. Today that second half is written by hand, and the experience of building one strategy never carries to the next. We introduce Strategist, a meta-agent that evolves inference strategies across tasks. Each strategy is written as a blueprint,executable code that other strategies can reuse, and together the blueprints form a library. Given a task and a small development set, Strategist retrieves related blueprints, writes candidate strategies, tests them on real samples, and revises them from the results. Every candidate then enters the library, so later tasks build on earlier discoveries. On 30 benchmarks, Strategist beats the strongest baseline on 24 and matches its average accuracy (74.7% vs.74.0%) at 39% lower inference cost. Its strategies also transfer to unseen tasks and other models without further search. The part of an AI system that is not improved on its own now does.
Figures & tables
Figure 2: Strategy design is per task; Strategist makes it cumulative. By hand and in prior automated search, nothing carries between tasks. Strategist keeps every strategy in a library that the next task starts from.
Figure 3: The Strategy Blueprint is a freeform graph that grows through the library. (a) Two foundations ( chat and agent ) sit at the top. Strategies compose from them and from each other, forming a directed graph whose levels emerge naturally from composition depth. (b) Four protocols govern the graph: every node shares one abstract interface, each node can only call nodes created before it, no cycles are allowed, and no duplicates should exist.
Figure 4: Strategy blueprint captures reasoning structure. This simplified blueprint example defines the steps, control flow, and the model tier for each step. Prompts are not specified; filling in produces a runnable strategy.
Figure 5: The Strategist four-stage iterative pipeline. Each iteration: (1) Retrieve relevant blueprints and diverse inspiration from the Library; (2) Propose K candidates as a mix of new , composed , and reuse blueprints paired with task-specific prompts; (3) Evaluate each candidate, seeing full traces on the training set and aggregated scores on validation; (4) Accumulate the underlying blueprints into the library with measurement cards and experience notes.
Math
AQuA-RAT ( Ling et al., 2017 )
Gaokao-Math ( Zhong et al., 2024 )
GSM8K ( Cobbe et al., 2021 )
AIME 2024 (MAA, 2024 )
MATH500 ( Hendrycks et al., 2021b )
ASDiv ( Miao et al., 2020 )
SVAMP ( Patel et al., 2021 )
MinervaM ( Lewkowycz et al., 2022 )
Commonsense
StrategyQA ( Geva et al., 2021 )
CommonsenseQA ( Talmor et al., 2019 )
PIQA ( Bisk et al., 2020 )
HellaSwag ( Zellers et al., 2019 )
OpenBookQA ( Mihaylov et al., 2018 )
SIQA ( Sap et al., 2019 )
COPA ( Roemmele et al., 2011 )
WinoGrande ( Sakaguchi et al., 2020 )
SciQ ( Welbl et al., 2017 )
BoolQ ( Clark et al., 2019 )
Table 1: 40 benchmarks across seven task categories. We randomly select 30 for the main iterative search and hold out the remaining 10 for one-shot retrieval. Each benchmark is split into training, validation, hidden, and test sets.
Figure 6: Ours among all 108 baseline cells (accuracy vs weighted cost per sample, test set). Each small marker is one baseline cell (a baseline strategy at one model tier and mode); the star is Strategist ’s strategy. BL, the comparison point, is the cell chosen on the training split (Figure 7 ). The highest marker in a panel is the best of 108 cells chosen by looking at the test results , which no method can do in advance; it is shown for context, not as the comparison point. Appendix Figure 10 shows the same view on the training split, where the bias runs the other way. Against BL, Ours is more than 2% cheaper on 26 of 30 benchmarks and within 2 accuracy points or better on 28.
Figure 7: Per-benchmark head-to-head: Strategist vs the baseline (BL) on the 30 main benchmarks. For each benchmark, test accuracy (%, left axis) and weighted cost per sample (log scale, right axis) of Ours and of BL. BL is the baseline cell (one of the 18×6 baseline–configuration pairs) that is most accurate on the benchmark’s training split, scored on the test set; Strategist ’s strategy is also chosen without seeing the test set, so this is the like-for-like comparison behind the win counts. Numbers above each pair are relative changes (green: improvement; grey: regression under 2%; orange: larger regression).
Table 8
Figure 8: An example reasoning strategy discovered by Strategist for the Amazon-Rec task. The strategy decomposes the task into a profile, signal extraction, conditional judge, merge, and audit stage. The structure and branch conditions were generated by the strategist, not by us.
Figure 9: Accuracy across data splits over the 10-iteration search loop. Each point is the mean accuracy of an iteration’s five candidates on a split. Bold curves are the mean across the 30 main benchmarks; faint background curves are individual benchmarks. The training–hidden gap stays between 1.2 and 2.5 points, not widening over iterations.
Strategist variants
Other frameworks
Benchmark
full
no library
zero-shot
AFlow
ADAS
EvoFlow
StrategyLLM
AgentSquare
BBH
85.0
31.0
27.5
38.0
34.5
34.5
35.5
27.5
Gaokao-Math
94.5
65.5
81.5
84.0
82.0
86.5
89.5
82.5
LSAT-LR
82.0
81.0
71.5
81.0
74.5
75.5
77.0
73.5
LSAT-RC
81.5
75.0
38.0
74.5
75.5
75.0
74.5
74.0
SAT-Math
98.2
99.0
failed
99.0
96.9
96.9
97.9
97.9
Table 5: Ablations and cross-framework comparison on 10 benchmarks. Test accuracy (%); best per row bolded. External frameworks share the same meta-LLM and executor; “failed” is runtime crash.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Tier
Model
Parameters
Cost weight
fast
Qwen3-0.6B
0.60 B
1.00×
balanced
Qwen3-1.7B
1.70 B
2.83×
complex
Qwen3-4B
4.00 B
6.67×
Appendix
Table 6: Tier cost weights used throughout the paper. Per-token weights are proportional to parameter counts, matching the leading 2N FLOPs/token term for dense transformers ( Kaplan et al., 2020 ) .
Tier/mode
GPT-5 backend
Gemini-2.5 backend
fast/nothink
gpt-5-nano (effort=minimal)
gemini-2.5-flash-lite
fast/thinking
gpt-5-nano (effort=medium)
gemini-2.5-flash-lite (thinking)
balanced/nothink
gpt-5-mini (effort=minimal)
gemini-2.5-flash
balanced/thinking
gpt-5-mini (effort=medium)
gemini-2.5-flash (thinking)
complex/nothink
gpt-5 (effort=minimal)
gemini-2.5-flash
complex/thinking
gpt-5 (effort=medium)
gemini-2.5-flash (thinking)
Appendix
Table 7: Tier-mode mapping for cross-model transfer. GPT-5 uses three model sizes (nano/mini/full) crossed with two reasoning-effort levels (minimal/medium). Gemini-2.5 uses flash-lite for the fast tier and flash for the balanced and complex tiers; the API sets its own thinking budget, so the nothink and thinking modes issue the same call. Each strategy executes its own routing; only the tier-to-model mapping changes.
Field
Sample value
search_queries
“cite-then-extract on long context multi-hop”
“self-consistency vote at fast tier with 3 samples”
“multi-agent debate but only on questionable cases”
“decomposition into sub-questions then aggregate”
experience (workflow, obs, rec)
self_consistency : “card shows +12pp on math/fast/nothink; blind matched at 11pp” / “use as cheap lift on short-math fast-tier; skip on commonsense.”
multi_agent_debate : “2-agent fails by mutual-error agreement on 30% of NQ-Open misses” / “add a third tie-breaking agent or vote.”
Appendix
Table 8: Sampled search_queries and experience notes. The strategist writes both at every iteration; queries are the input to library retrieval, and experience notes are the writeback that next-iteration strategists (or strategists on a different task) read in the card.
Iter
Kind
Best candidate on hidden
Hidden Acc
Hidden Cost ( k )
1
reference
baseline_cot (BL)
85.0
21.2
2
reuse
difficulty_routed_certificate_mcq
89.0
18.5
3
reuse
difficulty_routed_certificate_mcq
91.0
21.7
4
reuse
difficulty_routed_certificate_mcq
92.0
20.9
5
composed
intended_operation_commitment_cot_math
90.0
16.2
6
new
rotated_label_invariance_math
88.0
18.6
Appendix
Table 9: Iteration-by-iteration progression on AQuA-RAT. For each iteration, the candidate with the highest hidden-set accuracy and its weighted cost per sample; iteration 1 is the reference BL (CoT on complex/thinking ). The final strategy (bold, from iteration 5) beats BL on both validation and hidden ( +5 points at 0.77× cost on hidden); the iteration-4 candidate, the most accurate on hidden, is not picked because on validation it matches BL’s accuracy at 14% higher cost.
Benchmark
Ours Acc
BL Acc
Δ Acc
Ours Cost ( k )
BL Cost ( k )
Δ Cost
Result
SQuAD v2
79.1
73.2
↑ 8.0%
5.6
25.0
↓ 77.4%
Win
TruthfulQA
69.0
64.5
↑ 7.0%
5.8
7.5
↓ 21.9%
Win
2WikiMQA
76.5
72.5
↑ 5.5%
33.8
98.6
↓ 65.7%
Win
Amazon-Rec
28.0
24.0
↑ 16.7%
45.6
70.3
↓ 35.1%
Win
SAT-Math
98.2
94.5
↑ 3.8%
4.9
12.1
↓ 59.4%
Win
LSAT-AR
81.0
77.6
↑ 4.4%
54.9
82.0
↓ 33.0%
Win
Appendix
Table 10: Per-benchmark main results across 30 benchmarks. BL is the baseline cell (of the 18×6 matrix) with the highest accuracy on the benchmark’s training split, scored on the test set. Δ values are relative changes; colors follow the 2-point accuracy and 2% cost tie bands. Result: Win = better on accuracy or cost and worse on neither; Mixed = better on one and worse on the other; Lose = worse on at least one and better on neither. Cost is weighted tokens per sample (parameter-count weights), in thousands.
Figure 10: The same view on the training split. As Figure 6 , but every baseline cell and Strategist ’s final strategy are scored on the benchmark’s training split (the rows the strategist sees) instead of the test set. This comparison is biased toward BL: BL is, by definition, the most accurate of the 108 cells on exactly these rows, so it sits at the top of each panel, while Strategist ’s strategy was chosen on the separate validation and hidden splits. Only the test set (Figure 7 , Figure 6 ) scores both on questions neither was chosen on, which is why the paper reports it.
Benchmark
Ours Acc
Ours Cost ( k )
BL Acc
BL Cost ( k )
Δ Acc
Δ Cost
ReCLor
83.5
47.4
85.0
72.0
↓ 1.8%
↓ 34.2%
SAT-EN
86.5
16.7
86.5
37.6
0.0%
↓ 55.6%
TriviaQA
50.5
25.8
46.0
64.6
↑ 9.8%
↓ 60.0%
MATH500
94.5
47.2
92.5
61.0
↑ 2.2%
↓ 22.6%
BoolQ
85.5
32.7
73.5
1.0
↑ 16.3%
↑ 3103%
SIQA
77.5
6.7
73.5
1.5
↑ 5.4%
↑ 337%
Appendix
Table 11: Full per-benchmark one-shot library retrieval results. Accuracy is on the test set; cost is weighted tokens per sample (parameter-count weights). Δ Acc and Δ Cost are relative changes vs the baseline (formula and color/arrow conventions match Table 10 ).
GPT-5
Gemini-2.5
Benchmark
BL Acc
Ours Acc
Δ Acc
BL $
Ours $
Δ Cost
BL Acc
Ours Acc
Δ Acc
BL $
Ours $
Δ Cost
AQuA-RAT
31.0
89.5
↑ 189%
12.39
1.67
↓ 86.5%
71.0
90.5
↑ 27.5%
2.22
1.07
↓ 51.6%
2WikiMQA
29.5
78.5
↑ 166%
7.45
3.67
↓ 50.8%
21.5
77.5
↑ 260%
2.02
0.57
↓ 72.0%
HotpotQA
39.0
76.5
↑ 96.2%
7.71
7.13
↓ 7.5%
33.0
76.5
↑ 132%
2.52
1.20
↓ 52.2%
MBPP+
56.5
82.5
↑ 46.0%
2.16
1.89
↓ 12.3%
69.5
80.5
↑ 15.8%
1.79
1.37
↓ 23.7%
SQuAD2
57.5
74.0
↑ 28.7%
0.29
0.11
↓ 61.5%
74.0
70.5
↓ 4.7%
0.37
0.16
↓ 57.9%
Appendix
Table 12: Full per-benchmark cross-model transfer results. For each benchmark, the Strategist -discovered strategy and the strongest leaderboard baseline workflow are both re-executed on the new backend; Δ Acc and Δ Cost compare them on the same backend. 200 deterministic samples per cell. BL andOurs are total USD across all 200 samples, computed at the official list prices published on the OpenAI (GPT-5) and Google (Gemini-2.5) websites; the same per-token rates are applied to baseline and Strategist strategies.
An LLM agent is defined by two things: the weights inside its model and the harness of components assembled around it. Harnesses are still handcrafted, and HarnessX, which evolves them automatically, starts each benchmark from a handcrafted harness, reports gains on the tasks it evolved on, and budgets 100 to 175 million meta-agent tokens per benchmark. We propose LiteEvo, a lightweight harness-evolution algorithm whose tool-free meta-agents mine agent trajectories for reusable components, curate them into a versioned library, and compose each round's harness from it, starting every benchmark from the same neutral harness and never naming the benchmark. Evolving on the graded tasks of five agentic benchmarks with a frozen Qwen3.5-9B, LiteEvo lifts pass@2 by 10.5 to 67.7pp and reaches comparable or higher pass@2 than a reproduction of HarnessX (71.0 against 67.3 on average) at 13.0 lower mean API cost. Harnesses evolved on train tasks keep their gains on unseen test tasks of four benchmarks, and LiteEvo also lifts Claude Code with Sonnet 4.6 by 1.2 to 71.4pp.
Large language model (LLM) agents can now post-train an LLM end-to-end, raising the prospect of recursive self-improvement (RSI). Yet this progress is measured by aggregate benchmark scores, which cannot tell whether an agent executes a fixed plan well or strategically revises the plan when it fails. We separate these two capabilities: execution-level capability, iterating within an established training strategy, and strategy-level capability, revising that strategy as experimental evidence accumulates. Analyzing 1,338 post-training trajectories of frontier agents, we find that agents reliably execute post-training but lock into a default strategy, which follows the agent rather than the task, and only 2.1% of transitions between adjacent training runs ever change strategy. We then test whether the agent lacks experience, reasoning, or the decision to switch. (1) Experience improves execution but not the strategy. (2) Additional reasoning compute yields front-loaded gains on easier tasks but refines, rather than revises, the committed strategy. (3) Human review before training changes which strategy the agent locks into, not whether it locks in, whereas a single mid-run instruction outperforms the agent's own continuation by up to 17.44 points under the same budget. In conclusion, what the agent lacks is the decision to reopen a committed strategy and try another one. Realizing RSI therefore calls for interaction protocols and training signals that make strategy revision an explicit, rewarded decision.
Joy Jia Yin Lim, Xin Huang, Hao Peng +5
Tsinghua University · University of Electronic Science and Technology of China · Renmin University of China
Advances in inference methods have enabled language models to improve their predictions without additional training. These methods often prioritize raw performance over cost-effective compute usage. However, computational efficiency is key for real-world applications with resource constraints. We provide a systematic analysis of the inference scaling strategies self-consistency, self-refinement, multi-agent debate, and mixture-of-agents, to study their computational performance tradeoffs. We evaluate methods on two reasoning benchmarks (MMLU-Pro, BBH) and include extensive parameter configurations (e.g., scaling the number of parallel predictions, agents, and debate rounds) across different model sizes. Across 34 configurations and over 100 evaluations, we compute the Pareto-optimal front to select methods that achieve the best accuracy with the lowest computational budget. Notably, inference scaling improves accuracy by up to +7.1% points over chain-of-thought at the highest evaluated budgets (20x the CoT compute budget) on MMLU-Pro. With an equal computing budget, debate and mixture-of-agents outperform self-consistency by 1.3% and 2.7% points, respectively. While self-consistency saturates earlier, multi-agent gains persist, particularly on more complicated tasks. We identify a simple multi-agent design guideline: mixture-of-agents is most efficient when the number of parallel generations exceeds the number of sequential aggregations.
Florian Valentin Wunderlich, Lars Benedikt Kaesberg, Jan Philip Wahle +2