Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements

Zhi Zheng; Rongsheng Chen; Yunpeng Ba; Zhenkun Wang; Yee Whye Teh; Wee Sun Lee

cs.LGAug 18, 2026

Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements

Authors: Zhi Zheng, Rongsheng Chen, Yunpeng Ba, Zhenkun Wang, Yee Whye Teh, Wee Sun Lee

Organizations: National University of Singapore · Southern University of Science and Technology · Oxford

Abstract

Reinforcement Learning (RL) has been promising in single-turn LLM fine-tuning. However, long-horizon agentic reasoning introduces increasingly branching interactions and sparse rewards, exposing several limitations of RL: its heavyweight backpropagation-based training stack makes it impractical to fine-tune larger LLMs, and longer-horizon trajectories make credit assignment in RL substantially harder. This paper argues that evolution strategies (ES) can be a better choice for fine-tuning long-horizon LLM agents. Compared with agentic RL, ES offers three key advantages: 1) Model Scalability: ES enables full-parameter optimization with only minimal, inference-level GPU memory, making it possible to fine-tune large LLMs. 2) Flexibility: its lightweight, black-box feedback interface makes ES fine-tuning easy to compose with prompt-space evolution (e.g., skill optimization & test-time compute); and 3) Long-Horizon Scalability: ES performs trajectory-level parameter attribution without decomposing rewards across horizons, yielding better scalability than Agentic RL as the horizon length grows. Based on this insight, we propose Agentic ESOpt, a full-parameter agentic fine-tuning framework tailored to flexible parameter--context co-evolution. At each step, Agentic ESOpt samples perturbations around the current LLM parameters, evaluates the resulting agents with rewards, and applies an online reward-weighted update. To improve the exploration--adaptation trade-off, Agentic ESOpt further introduces a cosine decay schedule of the perturbation scale $σ$ . On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B improves the No Skill baseline by 6.69%. In test-time automatic heuristic design, Agentic ESOpt performs online prompt--parameter co-evolution, improving its matched baseline in 28 of 36 settings.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Long-horizon agentic reasoning introduces new challenges for Agentic RL (b), including high GPU memory requirements in training and difficult credit assignment across horizons. Agentic ESOpt (c) addresses these issues through inference-level GPU memory, flexible black-box feedback, and better long-horizon scalability.

Figure 2: Detailed workflow of Agentic ESOpt. Starting from the current LLM, Agentic ESOpt samples parameter perturbations, evaluates the perturbed agents in the environment, normalizes their scalar rewards, and applies a reward-weighted ES update. Compared with Agentic RL, Agentic ESOpt provides model scalability , optimization flexibility , and long-horizon scalability . Its lightweight black-box interface also allows easy composition with prompt-space optimization methods such as Trace2Skill (LLM-based) and EoH (heuristic-based), enabling on-the-fly parameter adaptation within existing test-time compute procedures.

Panel of Figure 3 — Figure 3: Agentic Sudoku performance grouped by minimum successful horizon $H^{*}$ . (a) reports final success rate averaged over 3 runs with standard-deviation error bars for PPO, the stronger GRPO-B configuration, and Agentic ESOpt. Red/green annotations below the Agentic ESOpt values report its difference from the stronger Agentic RL result. (b) and (c) show evaluation curves for the methods. Both Vanilla ES and Agentic ESOpt use $G=32$ .

Method	GPU Mem Req.	$H^{*}=5$	$H^{*}=10$	$H^{*}=15$
Qwen3.5-27B	51.75GB	$86.46\pm 3.90$	$50.00\pm 2.55$	$28.13\pm 2.55$
Qwen3.5-4B	8.41GB	$63.54\pm 7.80$	$31.25\pm 4.42$	$10.42\pm 1.47$
+ Agentic PPO $*$	89.40GB	$\mathbf{90.63\pm 0.00}$	$56.25\pm 0.00$	$0.00\pm 0.00$
+ Agentic GRPO $\dagger$	58.88GB	$80.21\pm 1.47$	$44.79\pm 2.95$	$30.21\pm 2.95$
+ Agentic GRPO $\ddagger$	58.88GB	$85.42\pm 1.47$	$\mathbf{67.71\pm 1.47}$	$40.63\pm 2.55$
+ Agentic ESOpt (G=32)	8.41GB	$89.58\pm 2.95$	$62.50\pm 2.55$	$\mathbf{53.13\pm 2.55}$

Table 1: Agentic Sudoku final evaluation success rate (

\times

100) and GPU memory requirement, grouped by minimum successful horizon

H^{*}

. Values are reported as mean

\pm

standard deviation.

	$H^{*}=5$		$H^{*}=10$		$H^{*}=15$
Method	FLOPs	Time	FLOPs	Time	FLOPs	Time
Qwen3.5-4B + Agentic GRPO	3.2 EFLOPs	5.4 h	7.6 EFLOPs	13.1 h	10.9 EFLOPs	19.0 h
Qwen3.5-4B + Agentic ESOpt ( $G$ =32)	3.1 EFLOPs	3.1 h	6.3 EFLOPs	5.8 h	9.4 EFLOPs	9.4 h

Table 2: Training compute and wall-clock time on agentic Sudoku. Both Agentic GRPO and Agentic ESOpt are executed on the same four NVIDIA H100 GPUs and fully utilize the available hardware.

		Math reasoning				DocVQA
		DAPO		AIME 2026		ANLS		Accuracy
Model	Method	Mean@4	Pass@4	Mean@4	Pass@4	Mean@4	Max@4	Mean@4	Pass@4
Qwen3.5-27B	No Skill	65.8	87.0	76.7	93.3	0.5036	0.7843	51.8	69.0
Qwen3.5-4B	No Skill	63.0	86.0	55.8	86.7	0.3875	0.5981	40.3	53.0
Qwen3.5-4B	Agentic GRPO + No Skill	68.8	83.0	58.3	76.7	0.4627	0.5398	48.0	56.0
Qwen3.5-4B	Agentic ESOpt + No Skill	76.8	86.0	70.8	96.7	0.5043	0.6507	52.5	61.0

Table 3: Math-reasoning and DocVQA results under No Skill and Trace2Skill contexts. Qwen3.5-27B No Skill is an evaluation-only baseline; all optimized rows use Qwen3.5-4B. DAPO, AIME 2026, and DocVQA accuracy are reported in %. Mean@4 averages four sampled Pass@1 accuracies; Pass@4 or Max@4 represent the best of four samples.

Model	Method	Reddit (21)	GitLab (32)	CMS (35)	Map (28)	OSS (46)	Dataset Avg.
Strong frozen baseline (evaluation only)
GPT-5.4	No Skill	47.62	46.88	46.67	19.05	21.01	34.14 $\pm$ 0.76
GPT-5.4-mini	No Skill	39.68	29.17	30.48	13.10	13.77	23.23 $\pm$ 1.14
GPT-5.4-nano	No Skill	39.68	27.08	19.05	11.90	8.70	18.79 $\pm$ 0.99
Qwen3.5-27B	No Skill	50.79	35.42	41.90	8.33	21.01	29.47 $\pm$ 1.14
Qwen3.5-27B	Agentic ESOpt + No Skill	49.21	43.75	49.52	14.29	30.43	36.16 $\pm$ 0.70

Table 4: WebArena-Lite success rates (%). We report the average over 3 runs and the standard deviations. Improved Agentic ESOpt cells relative to their paired baseline are shaded green; bold denotes the best result in each column. The five displayed site categories contain 162 tasks; the remaining Wikipedia tasks are included in Dataset Avg.

Method	TSP $N=20$	TSP $N=50$	KP $N=50,W=12.5$	KP $N=100,W=25$	ASP $N=15,W=10$	ASP $N=21,W=15$
Optimal	3.8199	5.6750	20.0370	40.2710	3,003	43,596
Greedy Construct	4.4797	6.9590	19.9850	40.2250	1,530	15,050
Total Evaluations: $T=1000$
Sample	4.3286	6.7110	19.9896	40.2297	2,753	30,336.67
Agentic ESOpt + Sample	4.2336	6.5488	19.9899	40.2314	2,729	31,082.67
$\Delta$ vs Sample	$\downarrow$ 22.96%	$\downarrow$ 18.56%	$\uparrow$ 0.79%	$\uparrow$ 4.42%	$\downarrow$ 8.76%	$\uparrow$ 5.96%

Table 5: Design constructive heuristics at total evaluation

T\in\{1000,2000\}

. TSP is minimized; KP and ASP are maximized. The Optimal row shows optimum objective values, not an experimental baseline. Sample and EoH are each paired with their Agentic ESOpt counterpart; the

\Delta

rows report the gap-ratio gain defined in the text. Baselines and Agentic ESOpt use LLaMA-3.1-8B-Instruct for all runs.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Update-magnitude threshold	Fraction of parameters
$\|\Delta\theta\|\leq 1.0\times 10^{-3}$	$83.68\%$
$\|\Delta\theta\|\leq 1.5\times 10^{-3}\;(=\sigma)$	$96.26\%$
$\|\Delta\theta\|\leq 2.0\times 10^{-3}$	$99.42\%$

Appendix

Table 6: Distribution of parameter-update magnitudes after Agentic ESOpt for WebArena on Qwen3.5-27B with perturbation scale

\sigma_{t}=1.5\times 10^{-3}

Panel of Figure 6 — Figure 6: Two-dimensional Gaussian-smoothed reward landscapes around the Qwen3.5-4B checkpoint for Sudoku tasks with $H^{*}\in\{5,10,15\}$ . Each panel marks the unperturbed checkpoint and the best and worst raw perturbations in the displayed slice. The heatmap value is the smoothed average terminal reward in the local neighborhood. As $H^{*}$ increases, the absolute reward level and contrast decrease, reflecting increasingly sparse trajectory feedback, while local parameter-space variation remains observable.

Panel of Figure 7 — Figure 7: Turn-level diagnostics for long-horizon Sudoku. Left: under the schematic relation $S_{H^{*}}=p^{H^{*}}$ , the same per-turn error rate produces a larger trajectory-level penalty as the minimum successful horizon grows. For example, $p=0.95$ gives $S_{H^{*}}=0.774$ , $0.599$ , and $0.463$ for $H^{*}=5$ , $10$ , and $15$ , respectively. Right: realized-horizon diagnostics on the $H^{*}=15$ setting. Agentic GRPO training uses temperature $1$ , top- $p=1$ , and top- $k=-1$ , while its evaluation uses temperature $0.7$ , top- $p=0.8$ , and top- $k=20$ ; Agentic ESOpt uses the latter decoder for both training and evaluation. Agentic GRPO reaches the 45-turn interaction budget by step 60, whereas Agentic ESOpt remains near the 15-turn minimum and ends at 15.41 turns.

Profile	Mask count	Sigma schedule	Maximum turns
Vanilla ES	5 / 10	$1.0\times 10^{-3}$ constant	15 / 30
Vanilla ES	15	$5.0\times 10^{-4}$ constant	45
Agentic ESOpt	5 / 10	$1.0\times 10^{-3}\!\rightarrow\!2.5\times 10^{-4}$ cosine	15 / 30
Agentic ESOpt	15	$7.0\times 10^{-4}\!\rightarrow\!5.0\times 10^{-4}$ cosine	45

Appendix

Table 7: Sudoku ES configurations. All profiles use

G=32

, 32 puzzles per generation, 100 generations, and

\alpha=5\times 10^{-4}

Parameter	Value
Backbone	Qwen3.5-4B
ES generations	25
Population size $G$	16
Cases per direction	16
Update scale $\alpha$	$5\times 10^{-4}$
$\sigma$ schedule	$10^{-3}\!\rightarrow\!5\times 10^{-4}$ cosine

Appendix

Table 8: Agentic ESOpt training configuration for Math reasoning.

Parameter	Value
Backbone	Qwen3.5-4B
Training records	400
Prompt batch size	20
Rollouts per prompt	8
Training epochs	1
Training rounds	20

Appendix

Table 9: Multi-turn Agentic GRPO training configuration for Math reasoning.

Parameter	Value
Backbone	Qwen3.5-4B
ES generations	40
Population size $G$	16
Cases per direction	16
Update scale $\alpha$	$5\times 10^{-4}$
$\sigma$ schedule	$10^{-3}\!\rightarrow\!5\times 10^{-4}$ cosine

Appendix

Table 10: Agentic ESOpt training configuration for DocVQA.

Parameter	Value
Backbone	Qwen3.5-4B
Training records	50
Prompt batch size	4
Rollouts per prompt	8
Training epochs	15
Training rounds	180

Appendix

Table 11: Multi-turn Agentic GRPO training configuration for DocVQA.

Figure 8: Repeated-sampling profiles for DAPO, AIME 2026, DocVQA ANLS, and DocVQA accuracy. The

k=1

markers use the reported Mean@4 values as estimates of single-sample performance, while stars show the originally reported Pass@4 or Max@4 values. Curves report the newly estimated best-of-

k

results for

k\in\{2,8,16,32\}

; the newly computed

k=4

curve points are omitted to keep the original four-sample measurements visually distinct.

Figure 9: Average realized trajectory length during DocVQA training. Agentic GRPO and Agentic ESOpt use the same rollout decoder; their training schedules are aligned to a common update axis and shown through step 40.

Condition	Parameter stage	Skill stage	Data per stage	Additional ES
No Skill baseline	none	none	–	none
Agentic ESOpt + No Skill	70 ES generations	none	$G=8$ directions $\times$ 8 tasks/generation	–
Trace2Skill baseline	none	70 skill iterations	8 tasks $\times$ 8 rollouts/iteration	none
Agentic ESOpt + Trace2Skill	same 70-generation ES run	one post-hoc distillation	all completed ES trajectories	none

Appendix

Table 12: WebArena-Lite adaptation stages on Qwen3.5-27B. Both Agentic ESOpt conditions share the same No Skill parameter updates; the combined condition adds one post-hoc skill-distillation stage.

Figure 10: Full WebArena-Lite evaluation success rate during the No Skill Qwen3.5-27B Agentic ESOpt run. Evaluation is performed every 10 ES updates; the curve starts from the 29.50% base checkpoint and reaches 35.76% after 70 updates.

Outer scaffold	$T$	Outer generations	Candidates/generation	ES update batches	Directions/batch	$\sigma_{0}\!\rightarrow\!\sigma_{T}$
EoH ( $k=1$ )	1,000	25	$10(2+2k)=40$	50	10	$10^{-3}\!\rightarrow 0$
EoH ( $k=3$ )	2,000	25	$10(2+2k)=80$	50	30	$10^{-3}\!\rightarrow 0$
Independent Sample	1,000	50	20	50	20	$10^{-3}\!\rightarrow 0$
Independent Sample	2,000	100	20	100	20	$10^{-3}\!\rightarrow 0$

Appendix

Table 13: AHD proposal budgets and Agentic ESOpt update units. Matched baselines use the same proposal counts. In EoH, m1 and m2 each form one ES update batch per outer generation.

Method	TSP $N=50$	TSP $N=100$	CVRP $N=50,C=50$	CVRP $N=100,C=50$	BPP $N=500,C=150$	BPP $N=1{,}000,C=150$
ACO	5.992	8.948	11.355	18.778	208.828	417.938
Total Evaluations: $T=1000$
EoH	6.030	8.859	9.328	15.666	203.162	405.172
Agentic ESOpt + EoH	5.882	8.391	9.265	16.073	203.500	405.787
$\Delta$ vs EoH	$\downarrow$ 2.53%	$\downarrow$ 5.57%	$\downarrow$ 0.68%	$\uparrow$ 2.53%	$\uparrow$ 0.17%	$\uparrow$ 0.15%
Total Evaluations: $T=2000$

Appendix

Table 14: AHD ACO-style results at total evaluation budgets

T\in\{1000,2000\}

. Lower is better for all columns. The

\Delta

rows report the ratio

\mathrm{EoH}/(\mathrm{Agentic\ ESOpt+EoH})-1

; positive ratios denote improvement. Improvements are shaded green with green deltas; regressions remain unshaded with neutral-gray deltas.

Table 24

Task	EoH	Agentic ESOpt + EoH	$p$ -value
TSP ( $N=50$ , $\downarrow$ )	$6.5517\pm 0.0729$	$6.5007\pm 0.0868$	$0.0258$
KP ( $N=100,W=25$ , $\uparrow$ )	$40.1562\pm 0.0024$	$40.1578\pm 0.0017$	$0.0100$

Appendix

Table 17: Repeated-run analysis for constructive AHD. Each method is evaluated over 20 independent runs. Lower is better for TSP and higher is better for the native KP objective.

Task	Sample	Agentic ESOpt + Sample
Design constructive heuristics for TSP	54.8	60.6
Design constructive heuristics for KP	41.2	48.6
Design constructive heuristics for ASP	51.7	56.7

Appendix

Table 18: End-to-end runtime for constructive Sample at

T=1000

. All values are in minutes.

Setting	Backbone	Update batches	Directions/update	Reward load/direction	$\sigma_{0}\!\rightarrow\!\sigma_{T}$	$\alpha$
Sudoku, $H^{*}=5/10$	Qwen3.5-4B	100	32	32 puzzles	$10^{-3}\!\rightarrow 2.5{\times}10^{-4}$	$5{\times}10^{-4}$
Sudoku, $H^{*}=15$	Qwen3.5-4B	100	32	32 puzzles	$7{\times}10^{-4}\!\rightarrow 5{\times}10^{-4}$	$5{\times}10^{-4}$
Math	Qwen3.5-4B	25	16	16 problems	$10^{-3}\!\rightarrow 5{\times}10^{-4}$	$5{\times}10^{-4}$
DocVQA	Qwen3.5-4B	40	16	16 examples	$10^{-3}\!\rightarrow 5{\times}10^{-4}$	$5{\times}10^{-4}$
WebArena-Lite	Qwen3.5-27B	70	8	8 web tasks	$1.5{\times}10^{-3}\!\rightarrow 1.5{\times}10^{-3}$ constant	$2.5{\times}10^{-4}$
AHD + EoH, $T=1{,}000$	Llama-3.1-8B	50	10	one heuristic objective	$10^{-3}\!\rightarrow 0$	$5{\times}10^{-4}$

Appendix

Table 19: Cross-setting Agentic ESOpt configurations. “Update batches” counts parameter updates, and “reward load/direction” specifies how each perturbation is scored. All settings use full-parameter, one-sided Gaussian perturbations and population z-score normalization.

Resources	Type	License	URL
Qwen3.5-4B	Base LLM	Apache-2.0 License	https://huggingface.co/Qwen/Qwen3.5-4B
Qwen3.5-9B	Base LLM	Apache-2.0 License	https://huggingface.co/Qwen/Qwen3.5-9B
Qwen3.5-27B	Base LLM	Apache-2.0 License	https://huggingface.co/Qwen/Qwen3.5-27B
LLaMA-3.1-8B-Instruct	Base LLM	Llama 3.1 Community License	https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
verl	RL framework	Apache-2.0 License	https://github.com/volcengine/verl
EoH	AHD framework	MIT License	https://github.com/FeiLiu36/EoH

Appendix

Table 20: A summary of licenses and external resources used in this work.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 586 papers

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
Sep 29, 2025Xin Qiu, Yulu Gan, Conor F. Hayes +6Fine-Tuning Evolutionary Optimization
Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1
Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training
Aug 3, 2026Zhiyuan Wang, Shengcai Liu, Jiahao Wu +5Evolutionary Optimization Evolution Strategies
Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology · Hong Kong Polytechnic University · Hong Kong University of Science and Technology +2
Asynchrony Is Nearly Free for Evolution Strategies on Long-Horizon LLM Agentic Tasks
Oct 3, 2026William Hoy, Jingxuan Fan, Nurcin Celik +1Evolution Strategies RL Post-Training
University of Miami · Harvard University

Sep 29, 2025J/K move · Enter open · S save

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang+5

Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA+1

Fine-Tuning Evolutionary Optimization LLM Fine-Tuning Evolution Strategies RL Fine-Tuning

Open paper AskPDF arXiv

Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.

Abstract

LLM Agent Training

Explore similar work

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training

Asynchrony Is Nearly Free for Evolution Strategies on Long-Horizon LLM Agentic Tasks