Reinforcement Learning (RL) has been promising in single-turn LLM fine-tuning. However, long-horizon agentic reasoning introduces increasingly branching interactions and sparse rewards, exposing several limitations of RL: its heavyweight backpropagation-based training stack makes it impractical to fine-tune larger LLMs, and longer-horizon trajectories make credit assignment in RL substantially harder. This paper argues that evolution strategies (ES) can be a better choice for fine-tuning long-horizon LLM agents. Compared with agentic RL, ES offers three key advantages: 1) Model Scalability: ES enables full-parameter optimization with only minimal, inference-level GPU memory, making it possible to fine-tune large LLMs. 2) Flexibility: its lightweight, black-box feedback interface makes ES fine-tuning easy to compose with prompt-space evolution (e.g., skill optimization & test-time compute); and 3) Long-Horizon Scalability: ES performs trajectory-level parameter attribution without decomposing rewards across horizons, yielding better scalability than Agentic RL as the horizon length grows. Based on this insight, we propose Agentic ESOpt, a full-parameter agentic fine-tuning framework tailored to flexible parameter--context co-evolution. At each step, Agentic ESOpt samples perturbations around the current LLM parameters, evaluates the resulting agents with rewards, and applies an online reward-weighted update. To improve the exploration--adaptation trade-off, Agentic ESOpt further introduces a cosine decay schedule of the perturbation scale σ. On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B improves the No Skill baseline by 6.69%. In test-time automatic heuristic design, Agentic ESOpt performs online prompt--parameter co-evolution, improving its matched baseline in 28 of 36 settings.
Figures & tables
Figure 1: (a) Long-horizon agentic reasoning introduces new challenges for Agentic RL (b), including high GPU memory requirements in training and difficult credit assignment across horizons. Agentic ESOpt (c) addresses these issues through inference-level GPU memory, flexible black-box feedback, and better long-horizon scalability.
Figure 2: Detailed workflow of Agentic ESOpt. Starting from the current LLM, Agentic ESOpt samples parameter perturbations, evaluates the perturbed agents in the environment, normalizes their scalar rewards, and applies a reward-weighted ES update. Compared with Agentic RL, Agentic ESOpt provides model scalability , optimization flexibility , and long-horizon scalability . Its lightweight black-box interface also allows easy composition with prompt-space optimization methods such as Trace2Skill (LLM-based) and EoH (heuristic-based), enabling on-the-fly parameter adaptation within existing test-time compute procedures.
Figure 3: Agentic Sudoku performance grouped by minimum successful horizon H∗ . (a) reports final success rate averaged over 3 runs with standard-deviation error bars for PPO, the stronger GRPO-B configuration, and Agentic ESOpt. Red/green annotations below the Agentic ESOpt values report its difference from the stronger Agentic RL result. (b) and (c) show evaluation curves for the methods. Both Vanilla ES and Agentic ESOpt use G=32 .
Method
GPU Mem Req.
H∗=5
H∗=10
H∗=15
Qwen3.5-27B
51.75GB
86.46±3.90
50.00±2.55
28.13±2.55
Qwen3.5-4B
8.41GB
63.54±7.80
31.25±4.42
10.42±1.47
+ Agentic PPO ∗
89.40GB
90.63±0.00
56.25±0.00
0.00±0.00
+ Agentic GRPO †
58.88GB
80.21±1.47
44.79±2.95
30.21±2.95
+ Agentic GRPO ‡
58.88GB
85.42±1.47
67.71±1.47
40.63±2.55
+ Agentic ESOpt (G=32)
8.41GB
89.58±2.95
62.50±2.55
53.13±2.55
Table 1: Agentic Sudoku final evaluation success rate ( × 100) and GPU memory requirement, grouped by minimum successful horizon H∗ . Values are reported as mean ± standard deviation.
H∗=5
H∗=10
H∗=15
Method
FLOPs
Time
FLOPs
Time
FLOPs
Time
Qwen3.5-4B + Agentic GRPO
3.2 EFLOPs
5.4 h
7.6 EFLOPs
13.1 h
10.9 EFLOPs
19.0 h
Qwen3.5-4B + Agentic ESOpt ( G =32)
3.1 EFLOPs
3.1 h
6.3 EFLOPs
5.8 h
9.4 EFLOPs
9.4 h
Table 2: Training compute and wall-clock time on agentic Sudoku. Both Agentic GRPO and Agentic ESOpt are executed on the same four NVIDIA H100 GPUs and fully utilize the available hardware.
Math reasoning
DocVQA
DAPO
AIME 2026
ANLS
Accuracy
Model
Method
Mean@4
Pass@4
Mean@4
Pass@4
Mean@4
Max@4
Mean@4
Pass@4
Qwen3.5-27B
No Skill
65.8
87.0
76.7
93.3
0.5036
0.7843
51.8
69.0
Qwen3.5-4B
No Skill
63.0
86.0
55.8
86.7
0.3875
0.5981
40.3
53.0
Qwen3.5-4B
Agentic GRPO + No Skill
68.8
83.0
58.3
76.7
0.4627
0.5398
48.0
56.0
Qwen3.5-4B
Agentic ESOpt + No Skill
76.8
86.0
70.8
96.7
0.5043
0.6507
52.5
61.0
Table 3: Math-reasoning and DocVQA results under No Skill and Trace2Skill contexts. Qwen3.5-27B No Skill is an evaluation-only baseline; all optimized rows use Qwen3.5-4B. DAPO, AIME 2026, and DocVQA accuracy are reported in %. Mean@4 averages four sampled Pass@1 accuracies; Pass@4 or Max@4 represent the best of four samples.
Model
Method
Reddit (21)
GitLab (32)
CMS (35)
Map (28)
OSS (46)
Dataset Avg.
Strong frozen baseline (evaluation only)
GPT-5.4
No Skill
47.62
46.88
46.67
19.05
21.01
34.14 ± 0.76
GPT-5.4-mini
No Skill
39.68
29.17
30.48
13.10
13.77
23.23 ± 1.14
GPT-5.4-nano
No Skill
39.68
27.08
19.05
11.90
8.70
18.79 ± 0.99
Qwen3.5-27B
No Skill
50.79
35.42
41.90
8.33
21.01
29.47 ± 1.14
Qwen3.5-27B
Agentic ESOpt + No Skill
49.21
43.75
49.52
14.29
30.43
36.16 ± 0.70
Table 4: WebArena-Lite success rates (%). We report the average over 3 runs and the standard deviations. Improved Agentic ESOpt cells relative to their paired baseline are shaded green; bold denotes the best result in each column. The five displayed site categories contain 162 tasks; the remaining Wikipedia tasks are included in Dataset Avg.
Method
TSP N=20
TSP N=50
KP N=50,W=12.5
KP N=100,W=25
ASP N=15,W=10
ASP N=21,W=15
Optimal
3.8199
5.6750
20.0370
40.2710
3,003
43,596
Greedy Construct
4.4797
6.9590
19.9850
40.2250
1,530
15,050
Total Evaluations: T=1000
Sample
4.3286
6.7110
19.9896
40.2297
2,753
30,336.67
Agentic ESOpt + Sample
4.2336
6.5488
19.9899
40.2314
2,729
31,082.67
Δ vs Sample
↓ 22.96%
↓ 18.56%
↑ 0.79%
↑ 4.42%
↓ 8.76%
↑ 5.96%
Table 5: Design constructive heuristics at total evaluation T∈{1000,2000} . TSP is minimized; KP and ASP are maximized. The Optimal row shows optimum objective values, not an experimental baseline. Sample and EoH are each paired with their Agentic ESOpt counterpart; the Δ rows report the gap-ratio gain defined in the text. Baselines and Agentic ESOpt use LLaMA-3.1-8B-Instruct for all runs.
Figure 9
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Update-magnitude threshold
Fraction of parameters
∣Δθ∣≤1.0×10−3
83.68%
∣Δθ∣≤1.5×10−3(=σ)
96.26%
∣Δθ∣≤2.0×10−3
99.42%
Appendix
Table 6: Distribution of parameter-update magnitudes after Agentic ESOpt for WebArena on Qwen3.5-27B with perturbation scale σt=1.5×10−3 .
Figure 6: Two-dimensional Gaussian-smoothed reward landscapes around the Qwen3.5-4B checkpoint for Sudoku tasks with H∗∈{5,10,15} . Each panel marks the unperturbed checkpoint and the best and worst raw perturbations in the displayed slice. The heatmap value is the smoothed average terminal reward in the local neighborhood. As H∗ increases, the absolute reward level and contrast decrease, reflecting increasingly sparse trajectory feedback, while local parameter-space variation remains observable.
Figure 7: Turn-level diagnostics for long-horizon Sudoku. Left: under the schematic relation SH∗=pH∗ , the same per-turn error rate produces a larger trajectory-level penalty as the minimum successful horizon grows. For example, p=0.95 gives SH∗=0.774 , 0.599 , and 0.463 for H∗=5 , 10 , and 15 , respectively. Right: realized-horizon diagnostics on the H∗=15 setting. Agentic GRPO training uses temperature 1 , top- p=1 , and top- k=−1 , while its evaluation uses temperature 0.7 , top- p=0.8 , and top- k=20 ; Agentic ESOpt uses the latter decoder for both training and evaluation. Agentic GRPO reaches the 45-turn interaction budget by step 60, whereas Agentic ESOpt remains near the 15-turn minimum and ends at 15.41 turns.
Profile
Mask count
Sigma schedule
Maximum turns
Vanilla ES
5 / 10
1.0×10−3 constant
15 / 30
Vanilla ES
15
5.0×10−4 constant
45
Agentic ESOpt
5 / 10
1.0×10−3→2.5×10−4 cosine
15 / 30
Agentic ESOpt
15
7.0×10−4→5.0×10−4 cosine
45
Appendix
Table 7: Sudoku ES configurations. All profiles use G=32 , 32 puzzles per generation, 100 generations, and α=5×10−4 .
Parameter
Value
Backbone
Qwen3.5-4B
ES generations
25
Population size G
16
Cases per direction
16
Update scale α
5×10−4
σ schedule
10−3→5×10−4 cosine
Appendix
Table 8: Agentic ESOpt training configuration for Math reasoning.
Parameter
Value
Backbone
Qwen3.5-4B
Training records
400
Prompt batch size
20
Rollouts per prompt
8
Training epochs
1
Training rounds
20
Appendix
Table 9: Multi-turn Agentic GRPO training configuration for Math reasoning.
Parameter
Value
Backbone
Qwen3.5-4B
ES generations
40
Population size G
16
Cases per direction
16
Update scale α
5×10−4
σ schedule
10−3→5×10−4 cosine
Appendix
Table 10: Agentic ESOpt training configuration for DocVQA.
Parameter
Value
Backbone
Qwen3.5-4B
Training records
50
Prompt batch size
4
Rollouts per prompt
8
Training epochs
15
Training rounds
180
Appendix
Table 11: Multi-turn Agentic GRPO training configuration for DocVQA.
Figure 8: Repeated-sampling profiles for DAPO, AIME 2026, DocVQA ANLS, and DocVQA accuracy. The k=1 markers use the reported Mean@4 values as estimates of single-sample performance, while stars show the originally reported Pass@4 or Max@4 values. Curves report the newly estimated best-of- k results for k∈{2,8,16,32} ; the newly computed k=4 curve points are omitted to keep the original four-sample measurements visually distinct.
Figure 9: Average realized trajectory length during DocVQA training. Agentic GRPO and Agentic ESOpt use the same rollout decoder; their training schedules are aligned to a common update axis and shown through step 40.
Condition
Parameter stage
Skill stage
Data per stage
Additional ES
No Skill baseline
none
none
–
none
Agentic ESOpt + No Skill
70 ES generations
none
G=8 directions × 8 tasks/generation
–
Trace2Skill baseline
none
70 skill iterations
8 tasks × 8 rollouts/iteration
none
Agentic ESOpt + Trace2Skill
same 70-generation ES run
one post-hoc distillation
all completed ES trajectories
none
Appendix
Table 12: WebArena-Lite adaptation stages on Qwen3.5-27B. Both Agentic ESOpt conditions share the same No Skill parameter updates; the combined condition adds one post-hoc skill-distillation stage.
Figure 10: Full WebArena-Lite evaluation success rate during the No Skill Qwen3.5-27B Agentic ESOpt run. Evaluation is performed every 10 ES updates; the curve starts from the 29.50% base checkpoint and reaches 35.76% after 70 updates.
Outer scaffold
T
Outer generations
Candidates/generation
ES update batches
Directions/batch
σ0→σT
EoH ( k=1 )
1,000
25
10(2+2k)=40
50
10
10−3→0
EoH ( k=3 )
2,000
25
10(2+2k)=80
50
30
10−3→0
Independent Sample
1,000
50
20
50
20
10−3→0
Independent Sample
2,000
100
20
100
20
10−3→0
Appendix
Table 13: AHD proposal budgets and Agentic ESOpt update units. Matched baselines use the same proposal counts. In EoH, m1 and m2 each form one ES update batch per outer generation.
Method
TSP N=50
TSP N=100
CVRP N=50,C=50
CVRP N=100,C=50
BPP N=500,C=150
BPP N=1,000,C=150
ACO
5.992
8.948
11.355
18.778
208.828
417.938
Total Evaluations: T=1000
EoH
6.030
8.859
9.328
15.666
203.162
405.172
Agentic ESOpt + EoH
5.882
8.391
9.265
16.073
203.500
405.787
Δ vs EoH
↓ 2.53%
↓ 5.57%
↓ 0.68%
↑ 2.53%
↑ 0.17%
↑ 0.15%
Total Evaluations: T=2000
Appendix
Table 14: AHD ACO-style results at total evaluation budgets T∈{1000,2000} . Lower is better for all columns. The Δ rows report the ratio EoH/(AgenticESOpt+EoH)−1 ; positive ratios denote improvement. Improvements are shaded green with green deltas; regressions remain unshaded with neutral-gray deltas.
Table 24
Task
EoH
Agentic ESOpt + EoH
p -value
TSP ( N=50 , ↓ )
6.5517±0.0729
6.5007±0.0868
0.0258
KP ( N=100,W=25 , ↑ )
40.1562±0.0024
40.1578±0.0017
0.0100
Appendix
Table 17: Repeated-run analysis for constructive AHD. Each method is evaluated over 20 independent runs. Lower is better for TSP and higher is better for the native KP objective.
Task
Sample
Agentic ESOpt + Sample
Design constructive heuristics for TSP
54.8
60.6
Design constructive heuristics for KP
41.2
48.6
Design constructive heuristics for ASP
51.7
56.7
Appendix
Table 18: End-to-end runtime for constructive Sample at T=1000 . All values are in minutes.
Setting
Backbone
Update batches
Directions/update
Reward load/direction
σ0→σT
α
Sudoku, H∗=5/10
Qwen3.5-4B
100
32
32 puzzles
10−3→2.5×10−4
5×10−4
Sudoku, H∗=15
Qwen3.5-4B
100
32
32 puzzles
7×10−4→5×10−4
5×10−4
Math
Qwen3.5-4B
25
16
16 problems
10−3→5×10−4
5×10−4
DocVQA
Qwen3.5-4B
40
16
16 examples
10−3→5×10−4
5×10−4
WebArena-Lite
Qwen3.5-27B
70
8
8 web tasks
1.5×10−3→1.5×10−3 constant
2.5×10−4
AHD + EoH, T=1,000
Llama-3.1-8B
50
10
one heuristic objective
10−3→0
5×10−4
Appendix
Table 19: Cross-setting Agentic ESOpt configurations. “Update batches” counts parameter updates, and “reward load/direction” specifies how each perturbation is scored. All settings use full-parameter, one-sided Gaussian perturbations and population z-score normalization.
Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.
Xin Qiu, Yulu Gan, Conor F. Hayes +6
Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1
Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive. Evolution strategies (ES) enable memory-efficient full-parameter post-training without backpropagation and can eventually match the performance of gradient-based reinforcement learning (RL). However, resource-constrained settings typically offer only a few GPUs, so the high GPU-hour requirements of ES translate into prohibitively long training times. To address this, we introduce Cooperative Parameter-subspace Evolution Strategy (CoPES), a cooperative coevolutionary method that decomposes the full parameter space into lower-dimensional subspaces and searches over them cooperatively to improve optimization efficiency. We post-train a Qwen3.5-4B tool-using agent for the math task and evaluate it on five benchmarks of varying difficulty. Under the GPU-hour budget of full-parameter GRPO's best validation checkpoint, CoPES recovers 92% of GRPO's validation-accuracy gain, versus 67% for standard ES, while its theoretical GPU memory requirement is less than one-eighth that of full-parameter GRPO. It consistently outperforms standard ES and LoRA-based GRPO on all evaluated pass@k metrics across the five benchmarks. Additional experiments further show the advantage of CoPES on the question-answering task. These results demonstrate an improved trade-off between memory requirements and training time for agentic LLM post-training under resource constraints. The code is open-sourced in https://github.com/MetaronWang/CoPES
Zhiyuan Wang, Shengcai Liu, Jiahao Wu +5
Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology · Hong Kong Polytechnic University · Hong Kong University of Science and Technology +2
Synchronous policy updates can limit the efficiency of post-training for long-horizon LLM agents. When trajectories span multiple interaction turns, completion times can vary substantially, and synchronous update barriers leave faster workers idling, wasting compute resources. Asynchronous reinforcement learning addresses this inefficiency but introduces policy lag and off-policy optimization. Evolution strategies (ES) are a backpropagation-free alternative to gradient-based training, yet require many perturbed-policy evaluations and have remained largely synchronous in current LLM post-training. We introduce bounded-staleness asynchronous ES and investigate its training behavior under stale fitness measurements. On WebShop, we systematically vary the lag and fraction of stale fitness measurements and additionally introduce two stress tests to probe extreme staleness cases. Natural asynchrony closely tracks synchronous ES, and degradation in evaluation accuracy remains small even under extreme staleness, including when all fitness measurements are stale. These findings suggest that ES can tolerate substantial policy staleness without explicit off-policy correction, supporting its use in asynchronous post-training for long-horizon LLM agents.