Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

Wanyi Chen; Xiao Yang; Xu Yang; Tianming Sha; Qizheng Li; Zhuo Wang; Bowen Xian; Fang Kong

cs.AIApr 12, 2026

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

Authors: Wanyi Chen, Xiao Yang, Xu Yang, Tianming Sha, Qizheng Li, Zhuo Wang, Bowen Xian, Fang Kong,

Organizations: Soochow University · Microsoft Research Asia · Stony Brook University · Peking University ·

Abstract

We introduce Agent2 RL-Bench, a compact diagnostic benchmark for evaluating agentic RL post-training, which tests whether LLM agents can autonomously design, implement, debug, and execute post-training pipelines that improve foundation models. RL post-training increasingly drives model alignment and specialization, yet existing benchmarks are largely static, rewarding supervised fine-tuning or script generation without assessing an agent's ability to close an interactive RL loop. Agent2 RL-Bench provides a unified agent-facing interface: each run starts from an isolated workspace containing a base model, task data, instructions, and a grading API, and agents must iterate within a fixed budget by training models and submitting artifacts for evaluation. The benchmark spans six tasks across three levels, from static rule-based training to judge-based optimization and closed-loop online RL with trajectory collection. Two diagnostic skills, namely runtime recording and post-hoc summarization, enable structured analysis of agent behavior, facilitating smooth and effective iteration of the benchmark's evaluation framework. Across five agent systems and six driver LLMs, agents show intelligent behavior but clear limitations: one RL-oriented run improves ALFWorld from 4.85 to 93.28 via SFT warm-up and GRPO with online rollouts, yet DeepSearchQA remains difficult, most successful routes rely on supervised pipelines, and interactive outcomes show large single-run differences across agent stacks. Overall, Agent2 RL-Bench shows that current agents can sometimes engineer online RL, but stable agent-driven RL post-training remains rare under fixed budgets. It also demonstrates that our benchmark provides a strong and effective evaluation framework for future research in this direction. Code is available at https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/rl/autorl_bench/README.md

Figures & tables

Figure 1: Overview of Agent 2 RL-Bench. Top-left: the three-level training-structure ladder, from static rule-based verification (L1), to static judge-based reward (L2), to interactive rollout with trajectory-level feedback (L3). Top-right: the L3 online feedback loop, where agents collect trajectories, receive rewards, and update models through environment interaction. Bottom: the shared run-level workflow: a benchmark instance (task instructions, training data, base model $M_{0}$ , 12h budget) is placed in an isolated workspace; the agent writes code, trains models, submits artifacts, and receives scalar best-so-far feedback from the grading server.

Level	Type	Benchmark	Reward	Horizon
L1	Static rule-based	GSM8K	Rule-based	Short
L1	Static rule-based	HumanEval	Rule-based	Short
L2	Static judge-based	AlpacaEval 2.0	LLM judge	Short
L3	Interactive rollout	ALFWorld	Sparse binary	Long
L3	Interactive rollout	WebShop	Dense continuous	Long
L3	Interactive rollout	DeepSearchQA	Judge-based	Medium

Table 1: Benchmark overview organized by training structure.

Panel A: Qwen2.5-7B-Instruct (scaffold comparison; Claude Code mode-swept)
		L1: Static Rule		L2: Judge	L3: Interactive Rollout
System	Driver	GSM8K	HumanEval	AlpacaEval	ALFWorld	WebShop	DeepSearchQA
Baseline	N/A	77.70	75.61	18.90	4.85	69.00	11.75
OpenHands	GPT-5.4	81.24 (+3.54)	78.05 (+2.44)	18.77 ( $-$ 0.13)	9.41 (+4.56)	40.00 ( $-$ 29.00)	7.80 ( $-$ 3.95)
OpenCode	GPT-5.4	82.27 (+4.57)	79.27 (+3.66)	20.33 (+1.43)	19.33 (+14.48)	54.50 ( $-$ 14.50)	21.00 (+9.25)
Claude Code	Opus 4.6	85.06 (+7.36)	81.71 (+6.10)	30.76 (+11.86)	95.52 (+90.67)	84.00 (+15.00)	15.00 (+3.25)

Table 2: Agent 2 RL-Bench leaderboard. All runs use a 12h budget. Colored parenthetical values are absolute score changes from the baseline in the same panel. Panel A (7B-Instruct): scaffold comparison. OpenHands/OpenCode: single run (GPT-5.4 driver). Claude Code: best observed per-task operating-mode result (Opus 4.6 driver; full mode ablation in Table 3 ). Panel B (8B-Base): controlled single-run study on Qwen3-8B-Base across three CLI scaffolds and six driver LLMs.

Figure 2: Interactive-task scaling across agent stacks. Rows plot best-so-far score vs. elapsed hours, context tokens, and submissions; static-task scaling is in Appendix Figure 6 .

Figure 3: Method-switch trajectories for three representative 7B-Instruct runs. Blue/gray show best-so-far/per-submission scores; green/red mark successful/failed route changes, showing discrete route pivots and engineering interventions rather than smooth hill-climbing.

7B-Instruct
Mode	GSM8K	HumanEval	AlpacaEval	ALFWorld	WebShop	DeepSearchQA
Baseline	77.70	75.61	18.90	4.85	69.00	11.75
SFT-only	84.00 (+6.30)	81.71 (+6.10)	21.32 (+2.42)	67.91 (+63.06)	76.00 (+7.00)	14.00 (+2.25)
RL-oriented	81.88 (+4.18)	81.71 (+6.10)	20.96 (+2.06)	93.28 (+88.43)	79.00 (+10.00)	15.00 (+3.25)
Free	82.26 (+4.56)	79.27 (+3.66)	30.76 (+11.86)	95.52 (+90.67)	84.00 (+15.00)	13.00 (+1.25)
Multi	85.06 (+7.36)	75.61 (0.00)	19.54 (+0.64)	13.43 (+8.58)	79.00 (+10.00)	11.50 ( $-$ 0.25)

Table 3: Claude Code mode ablation. Parenthetical values are absolute score changes from the corresponding base-model baseline. Top: Qwen2.5-7B-Instruct (12h, Opus 4.6). Bottom: Qwen3-8B-Base (12h, Opus 4.6, Free vs. Multi only). Bold marks the best score within each base-model block.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Coverage of the six benchmark tasks across four manually defined structural dimensions: capability breadth, interaction intensity, reward complexity, and decision horizon. Scores are ordinal (1 to 5) and summarize benchmark-level metadata rather than empirical results.

Benchmark	Capability	Interaction	Reward	Horizon
GSM8K	2	1	1	1
HumanEval	2	1	1	1
AlpacaEval	2	1	4	1
ALFWorld	4	5	2	5
WebShop	4	5	5	5
DeepSearchQA	5	4	4	3

Appendix

Table 4: Ordinal benchmark-design scores used in Figure 4 .

Task	Structure	Evaluator Type	Reward Source	Eval Size	Horizon	Primary Difficulty
GSM8K	Static rule-based	OpenCompass	Exact-match answer check	standard test split	Short	reasoning-to-training transfer
HumanEval	Static rule-based	OpenCompass	unit-test execution	82 held-out / 82 train-visible	Short	code-generation post-training
AlpacaEval 2.0	Static judge-based	LLM judge	pairwise preference judging	reference split	Short	non-programmable reward
ALFWorld	Interactive rollout	environment evaluator	sparse success reward	134 episodes	Long	rollout and trajectory collection
WebShop	Interactive rollout	environment evaluator	task-specific dense reward	100 instructions	Long	reward shaping and stability
DeepSearchQA	Interactive tool-use	custom ReAct evaluator	search-and-judge loop	200 samples	Medium	tool use, search, and judged QA

Appendix

Table 5: Task protocol summary following the current implementation.

Metric Type	Definition
Benchmark-native outputs	baseline score, per-submission score, best score, improvement over baseline, submission count, runtime metadata
Process metrics	valid-submission rate, time to first improvement, time to best, score trajectory, time-used ratio, last-submit gap
Offline diagnostics	route labels, coding-success indicators, case-study inspection, trace-based interpretation

Appendix

Table 6: Reported metric types in Agent 2 RL-Bench.

Dimension	Configuration
7B Main Scaffold Comparison
Agents	OpenHands, OpenCode, Claude Code
Driver models	GPT-5.4 (OH/OC), Claude Opus 4.6 (CC)
Base model	Qwen2.5-7B-Instruct
Time budget	12h single run per system/task/mode where applicable
CC modes	SFT-only, RL-oriented, Free, Multi

Appendix

Table 7: Experimental setup used for the main experiments.

Route	Operational definition
GRPO	train.py contains explicit GRPO-style training components
PPO	train.py contains explicit PPO-style training components
SFT-only	train.py contains supervised fine-tuning components but no explicit RL trainer
SFT+RL	train.py contains both supervised and RL-style training components
Copy	the run copies or reuses the base model without substantive training logic
Placeholder / Missing	the training script is absent or effectively a placeholder rather than executable training code

Appendix

Table 8: Route labels used in the technical-route analysis.

Rule category	Implemented behavior
Test-set isolation	Held-out evaluation data are not mounted into the agent workspace. Agents only receive training-visible data and task descriptions.
Submission interface	Scores are produced only through the grading server. Agents cannot directly overwrite leaderboard files or final result tables.
Baseline reuse	Trivial submission of the untouched base model is treated as non-improving behavior and maps back to baseline performance.
Artifact logging	Each run stores score trajectories, run metadata, and workspace artifacts for later inspection.
Evaluator serialization	Heavyweight evaluation is serialized within a run to reduce contention and make score traces easier to audit.
Path-based caching	Identical submissions may be cached by resolved path and file-modification state to avoid repeated redundant evaluation, without changing the best-score semantics.

Appendix

Table 9: Implemented audit and anti-cheating rules.

Failure type	Typical task	Observable symptom	Main audit artifact
Rollout construction failure	ALFWorld, WebShop	The run produces training code, but the environment loop is incomplete, brittle, or never yields a usable trajectory set.	train.py , task-specific eval.py , missing or degenerate submissions
Reward/data design failure	WebShop, DeepSearchQA	The run implements non-trivial collection and training logic, yet the resulting data or reward signal is too sparse, brittle, or misaligned to improve performance.	Generated training script, score trajectory, failure notes in summary.md
Training collapse / instability	WebShop, ALFWorld	Early submissions exist, but later iterations regress or fail during model loading, runtime startup, or evaluation.	scores.json , run logs, final workspace artifacts
Early coding failure	All tasks, especially weaker-model settings	The run terminates before producing substantive training code or a valid submission.	Missing or placeholder train.py , zero-submission runs
Static shortcut without transfer	GSM8K, HumanEval versus L3 tasks	The scaffold can assemble a workable static SFT/RL route, but the same competence does not carry over once interactive rollout becomes mandatory.	Cross-task comparison of route labels, submissions, and improvements

Appendix

Table 10: Failure taxonomy used for offline diagnosis of interactive-task runs.

Task	Mode	Subs	Code	Models	Zero%	Best@Sub	Primary Strategies
GSM8K	SFT	92	14	146	20%	#90	SFT, model soup, seed sweep
HumanEval	SFT	195	28	336	23%	#26	SFT, model averaging, task arithmetic
AlpacaEval	Free	23	11	47	0%	#23	SFT $\to$ DPO, synthetic preference gen.
ALFWorld	Free	4	4	5	0%	#4	Demonstration SFT (env. interaction)
WebShop	Free	16	4	20	19%	#13	Trajectory collection $\to$ SFT
DeepSearchQA	Free	145	47	285	0%	#39	DPO, GRPO, KTO, SLERP, RFT

Appendix

Table 11: Exploratory Claude Code workspace statistics per task. Submissions = total valid grading requests; Code files = distinct .py files written under code/ ; Model versions = directories under output/ ; Zero rate = fraction of submissions scoring 0 because training or evaluation did not complete successfully; Best@Sub = submission number at which the best score was achieved.

Parameter	Under-fit	Sweet spot	Over-fit
Learning rate	5e-5 $\to$ 10.5	1e-4 $\to$ 11 to 13	2e-4 $\to$ 4.0
LoRA rank	16 $\to$ 10.5	32 $\to$ 11 to 13	64 $\to$ 4.0 to 7.5
Epochs	2 $\to$ 10.5	3 to 4 $\to$ 11 to 13	5+ $\to$ 6.5
Training loss	$>$ 1.5	1.1 to 1.2	$<$ 0.5

Appendix

Table 12: DeepSearchQA SFT sensitivity analysis from early local sweeps (Qwen3-8B-Base, 12h). The optimization landscape has a narrow sweet spot; deviations in any direction degrade performance sharply.

Figure 5: Mode $\times$ Task improvement heatmap for Claude Code. Stars mark the best mode per task; triangles mark the worst. Rank reversals between modes confirm that no single training paradigm uniformly dominates.

Mode	GSM8K	HE	AE	ALF	WS	DSQA
SFT-only	84.00	81.71	21.32	67.91	76.00	14.00
RL-oriented	81.88	81.71	20.96	93.28	79.00	15.00
Free	82.26	79.27	30.76	95.52	84.00	13.00
Multi	85.06	75.61	19.54	13.43	79.00	11.50

Appendix

Table 13: Claude Code mode ablation (Qwen2.5-7B-Instruct, 12h). Bold = best, underline = worst per task.

Task	RL Methods Tried	Stable?	Best Pipeline
GSM8K	GRPO ( $\times$ 15 variants)	Partial	SFT + model soup
HumanEval	GRPO, DPO, RFT	No	SFT + model averaging
AlpacaEval	GRPO, DPO ( $\times$ 31)	Yes (DPO)	SFT $\to$ DPO
ALFWorld	GRPO + online rollout	Yes	SFT $\to$ GRPO (RL-oriented)
WebShop	DPO, ORPO, KTO	Partial	Traj. collection $\to$ SFT
DeepSearchQA	DPO, GRPO, KTO, ORPO	Partial	SFT $\to$ DPO pipeline

Appendix

Table 14: RL method audit across Claude Code workspaces. “Stable” means that training completed reliably and produced non-degenerate results.

Mode	GSM8K	HumanEval	AlpacaEval	ALFWorld	WebShop	DSQA
Free	74.00	84.15	26.91	97.76	78.00	23.00
Multi	80.97	60.98	10.33	85.82	13.00	7.00
Baseline	83.02	62.20	14.68	8.96	0.00	12.25

Appendix

Table 15: Multi-task degradation on Qwen3-8B-Base (12h). Free mode scores are from single-task runs. The pattern mirrors Table 3 : Multi wins on GSM8K but severely degrades interactive tasks.

		L1: Static Rule		L2: Judge	L3: Interactive Rollout
Scaffold	Driver	GSM8K ( $\Delta$ )	HumanEval ( $\Delta$ )	AlpacaEval ( $\Delta$ )	ALFWorld ( $\Delta$ )	WebShop ( $\Delta$ )	DSQA ( $\Delta$ )
N/A	Baseline	83.02	62.20	14.68	8.96	0.00	12.25
Claude Code	Opus 4.6	88.78 ( +5.76 )	84.15 ( +21.95 )	26.91 ( +12.23 )	97.76 ( +88.80 )	78.00 ( +78.00 )	23.00 ( +10.75 )
Claude Code	Sonnet 4.5	88.70 ( +5.68 )	65.85 ( +3.65 )	19.80 ( +5.12 )	15.67 ( +6.71 )	68.00 ( +68.00 )	18.50 ( +6.25 )
Codex CLI	GPT-5.4	84.00 ( +0.98 )	72.25 ( +10.05 )	22.82 ( +8.14 )	90.93 ( +81.97 )	57.00 ( +57.00 )	21.00 ( +8.75 )
Codex CLI	GPT-5.2	89.46 ( +6.44 )	69.51 ( +7.31 )	18.67 ( +3.99 )	87.31 ( +78.35 )	75.00 ( +75.00 )	12.00 ( $-$ 0.25 )

Appendix

Table 16: Complete Qwen3-8B-Base controlled study (12h, single run).

\Delta

computed against canonical baseline [83.02 / 62.20 / 14.68 / 8.96 / 0.00 / 12.25].

Dimension	Claude Code	Codex
Driver LLM	Claude Opus 4.6	GPT-5.4
Submissions	16	56
Iteration style	lower-frequency, more selective	higher-frequency, broader search
Best $\Delta$	+10.75	+8.75

Appendix

Table 17: Agent behavior comparison on DeepSearchQA (Qwen3-8B-Base, 12h).

Figure 6: Scaling analysis for the three static tasks across all agent stacks. Top: Time scaling. Middle: Token scaling. Bottom: Submission scaling. Compared with the interactive-task scaling in Figure 2 , static tasks typically plateau earlier and show weaker separation between strong agent stacks.

Figure 7: Improvement over canonical Qwen3-8B-Base baseline ( $\Delta$ in percentage points) for six agent stacks across all tasks (12h). Interactive tasks (ALFWorld, WebShop) show larger observed differences across driver and scaffold choices than static tasks.

Method	Attempted cells	Adopted in best	Adoption rate
SFT (plain)	19	18	95%
Model merging	11	4	36%
RFT	5	3	60%
GRPO	8	2	25%
DPO	4	1	25%
KTO / ORPO	3	0	0%

Appendix

Table 18: Method attempt vs. adoption frequency across reconstructable Claude Code 7B cells. "Attempted" counts cells in which at least one submission is tagged with the method; "Adopted in best" counts cells whose highest-scoring submission used the method.

Sub #	Script	Score	Method
baseline	N/A	62.20%	Qwen3-8B-Base (unaligned)
#3	sft_train.py	2.44%	SFT v1
#5	sft_train_v3.py	43.90%	SFT v3 (prompt alignment)
#7	grpo_train.py	79.27%	GRPO on SFT checkpoint (LoRA)
#11	rft_train.py	84.15%	RFT (rejection sampling fine-tuning)

Appendix

Table 19: Submission progression for the HumanEval trace walkthrough (Claude Code Free, Qwen3-8B-Base). Each row corresponds to a qualitatively distinct strategy shift.

Sub #	Script	Score	Method
baseline	N/A	8.96%	Qwen3-8B-Base (unaligned)
#1	train_sft.py	14.93%	SFT v1: ReAct examples
#2	train_sft_v2.py	16.42%	Hyperparameter tuning
#4	train_sft_v4.py	19.40%	Action format alignment ( put $\to$ move ) + synthetic data
#5	train_sft_v5.py	45.52%	Model rollout trajectories (+26.1pp)
#7	train_sft_v7.py	95.52%	Prompt format alignment (+50.0pp)

Appendix

Table 20: Submission progression for the ALFWorld trace walkthrough (Claude Code Free, Qwen3-8B-Base, 12h). Shaded rows mark the two largest score jumps.

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 215 papers

PostTrainBench: Can LLM Agents Automate LLM Post-Training?
Mar 9, 2026Ben Rank, Hardik Bhatnagar, Ameya Prabhu +4LLM Post-Training Autonomous Research Agents
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +2
Agentick: A Unified Benchmark for General Sequential Decision-Making Agents
May 7, 2026Roger Creus Castanyer, Pablo Samuel Castro, Glen BersethAI Agent Evaluation AI Agent Benchmarks
Mila Quebec AI Institute · Université de Montréal · Mila Quebec AI Institute Université de Montréal +1
Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach
Oct 14, 2026Jiarui Qin, Junjie Huang, Yunjia Xi +5LLM Agent Evaluation Agent Evaluation
Tencent Youtu Lab · Shanghai Jiao Tong University

Mar 9, 2026J/K move · Enter open · S save

PostTrainBench: Can LLM Agents Automate LLM Post-Training?

Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg+3

ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center+2

LLM Post-Training Autonomous Research Agents AI Agent Benchmarks LLM Agent Evaluation

Open paper AskPDF arXiv

AI agents have become surprisingly proficient at software engineering over the past year, largely due to improvements in reasoning capabilities. This raises a deeper question: can these systems extend their capabilities to automate AI research itself? In this paper, we explore post-training, the critical phase that turns base LLMs into useful assistants. We introduce PostTrainBench to benchmark how well LLM agents can perform post-training autonomously under bounded compute constraints (10 hours on one H100 GPU). We ask frontier agents (e.g., Claude Code with Opus 4.6) to optimize the performance of a base LLM on a particular benchmark (e.g., Qwen3-4B on AIME). Importantly, we do not provide any predefined strategies to the agents and instead give them full autonomy to find necessary information on the web, run experiments, and curate data. We find that frontier agents make substantial progress but generally lag behind instruction-tuned LLMs from leading providers: 23.2% for the best agent vs. 51.1% for official instruction-tuned models. However, agents can exceed instruction-tuned models in targeted scenarios: GPT-5.1 Codex Max achieves 89% on BFCL with Gemma-3-4B vs. 67% for the official model. We also observe several failure modes worth flagging. Agents sometimes engage in reward hacking: training on the test set, downloading existing instruction-tuned checkpoints instead of training their own, and using API keys they find to generate synthetic data without authorization. These behaviors are concerning and highlight the importance of careful sandboxing as these systems become more capable. Overall, we hope PostTrainBench will be useful for tracking progress in AI R&D automation and for studying the risks that come with it. Website and code are available at https://posttrainbench.com/.

Abstract

Agentic RL

Explore similar work

PostTrainBench: Can LLM Agents Automate LLM Post-Training?

Agentick: A Unified Benchmark for General Sequential Decision-Making Agents

Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach