We introduce Agent2 RL-Bench, a compact diagnostic benchmark for evaluating agentic RL post-training, which tests whether LLM agents can autonomously design, implement, debug, and execute post-training pipelines that improve foundation models. RL post-training increasingly drives model alignment and specialization, yet existing benchmarks are largely static, rewarding supervised fine-tuning or script generation without assessing an agent's ability to close an interactive RL loop. Agent2 RL-Bench provides a unified agent-facing interface: each run starts from an isolated workspace containing a base model, task data, instructions, and a grading API, and agents must iterate within a fixed budget by training models and submitting artifacts for evaluation. The benchmark spans six tasks across three levels, from static rule-based training to judge-based optimization and closed-loop online RL with trajectory collection. Two diagnostic skills, namely runtime recording and post-hoc summarization, enable structured analysis of agent behavior, facilitating smooth and effective iteration of the benchmark's evaluation framework. Across five agent systems and six driver LLMs, agents show intelligent behavior but clear limitations: one RL-oriented run improves ALFWorld from 4.85 to 93.28 via SFT warm-up and GRPO with online rollouts, yet DeepSearchQA remains difficult, most successful routes rely on supervised pipelines, and interactive outcomes show large single-run differences across agent stacks. Overall, Agent2 RL-Bench shows that current agents can sometimes engineer online RL, but stable agent-driven RL post-training remains rare under fixed budgets. It also demonstrates that our benchmark provides a strong and effective evaluation framework for future research in this direction. Code is available at https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/rl/autorl_bench/README.md
Figures & tables
Figure 1: Overview of Agent 2 RL-Bench. Top-left: the three-level training-structure ladder, from static rule-based verification (L1), to static judge-based reward (L2), to interactive rollout with trajectory-level feedback (L3). Top-right: the L3 online feedback loop, where agents collect trajectories, receive rewards, and update models through environment interaction. Bottom: the shared run-level workflow: a benchmark instance (task instructions, training data, base model M0 , 12h budget) is placed in an isolated workspace; the agent writes code, trains models, submits artifacts, and receives scalar best-so-far feedback from the grading server.
Level
Type
Benchmark
Reward
Horizon
L1
Static rule-based
GSM8K
Rule-based
Short
L1
Static rule-based
HumanEval
Rule-based
Short
L2
Static judge-based
AlpacaEval 2.0
LLM judge
Short
L3
Interactive rollout
ALFWorld
Sparse binary
Long
L3
Interactive rollout
WebShop
Dense continuous
Long
L3
Interactive rollout
DeepSearchQA
Judge-based
Medium
Table 1: Benchmark overview organized by training structure.
L1: Static Rule
L2: Judge
L3: Interactive Rollout
System
Driver
GSM8K
HumanEval
AlpacaEval
ALFWorld
WebShop
DeepSearchQA
Panel A: Qwen2.5-7B-Instruct (scaffold comparison; Claude Code mode-swept)
Baseline
N/A
77.70
75.61
18.90
4.85
69.00
11.75
OpenHands
GPT-5.4
81.24 (+3.54)
78.05 (+2.44)
18.77 ( − 0.13)
9.41 (+4.56)
40.00 ( − 29.00)
7.80 ( − 3.95)
OpenCode
GPT-5.4
82.27 (+4.57)
79.27 (+3.66)
20.33 (+1.43)
19.33 (+14.48)
54.50 ( − 14.50)
21.00 (+9.25)
Claude Code
Opus 4.6
85.06 (+7.36)
81.71 (+6.10)
30.76 (+11.86)
95.52 (+90.67)
84.00 (+15.00)
15.00 (+3.25)
Table 2: Agent 2 RL-Bench leaderboard. All runs use a 12h budget. Colored parenthetical values are absolute score changes from the baseline in the same panel. Panel A (7B-Instruct): scaffold comparison. OpenHands/OpenCode: single run (GPT-5.4 driver). Claude Code: best observed per-task operating-mode result (Opus 4.6 driver; full mode ablation in Table 3 ). Panel B (8B-Base): controlled single-run study on Qwen3-8B-Base across three CLI scaffolds and six driver LLMs.
Figure 2: Interactive-task scaling across agent stacks. Rows plot best-so-far score vs. elapsed hours, context tokens, and submissions; static-task scaling is in Appendix Figure 6 .
Figure 3: Method-switch trajectories for three representative 7B-Instruct runs. Blue/gray show best-so-far/per-submission scores; green/red mark successful/failed route changes, showing discrete route pivots and engineering interventions rather than smooth hill-climbing.
Mode
GSM8K
HumanEval
AlpacaEval
ALFWorld
WebShop
DeepSearchQA
7B-Instruct
Baseline
77.70
75.61
18.90
4.85
69.00
11.75
SFT-only
84.00 (+6.30)
81.71 (+6.10)
21.32 (+2.42)
67.91 (+63.06)
76.00 (+7.00)
14.00 (+2.25)
RL-oriented
81.88 (+4.18)
81.71 (+6.10)
20.96 (+2.06)
93.28 (+88.43)
79.00 (+10.00)
15.00 (+3.25)
Free
82.26 (+4.56)
79.27 (+3.66)
30.76 (+11.86)
95.52 (+90.67)
84.00 (+15.00)
13.00 (+1.25)
Multi
85.06 (+7.36)
75.61 (0.00)
19.54 (+0.64)
13.43 (+8.58)
79.00 (+10.00)
11.50 ( − 0.25)
Table 3: Claude Code mode ablation. Parenthetical values are absolute score changes from the corresponding base-model baseline. Top: Qwen2.5-7B-Instruct (12h, Opus 4.6). Bottom: Qwen3-8B-Base (12h, Opus 4.6, Free vs. Multi only). Bold marks the best score within each base-model block.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Coverage of the six benchmark tasks across four manually defined structural dimensions: capability breadth, interaction intensity, reward complexity, and decision horizon. Scores are ordinal (1 to 5) and summarize benchmark-level metadata rather than empirical results.
Benchmark
Capability
Interaction
Reward
Horizon
GSM8K
2
1
1
1
HumanEval
2
1
1
1
AlpacaEval
2
1
4
1
ALFWorld
4
5
2
5
WebShop
4
5
5
5
DeepSearchQA
5
4
4
3
Appendix
Table 4: Ordinal benchmark-design scores used in Figure 4 .
Task
Structure
Evaluator Type
Reward Source
Eval Size
Horizon
Primary Difficulty
GSM8K
Static rule-based
OpenCompass
Exact-match answer check
standard test split
Short
reasoning-to-training transfer
HumanEval
Static rule-based
OpenCompass
unit-test execution
82 held-out / 82 train-visible
Short
code-generation post-training
AlpacaEval 2.0
Static judge-based
LLM judge
pairwise preference judging
reference split
Short
non-programmable reward
ALFWorld
Interactive rollout
environment evaluator
sparse success reward
134 episodes
Long
rollout and trajectory collection
WebShop
Interactive rollout
environment evaluator
task-specific dense reward
100 instructions
Long
reward shaping and stability
DeepSearchQA
Interactive tool-use
custom ReAct evaluator
search-and-judge loop
200 samples
Medium
tool use, search, and judged QA
Appendix
Table 5: Task protocol summary following the current implementation.
Metric Type
Definition
Benchmark-native outputs
baseline score, per-submission score, best score, improvement over baseline, submission count, runtime metadata
Process metrics
valid-submission rate, time to first improvement, time to best, score trajectory, time-used ratio, last-submit gap
Table 6: Reported metric types in Agent 2 RL-Bench.
Dimension
Configuration
7B Main Scaffold Comparison
Agents
OpenHands, OpenCode, Claude Code
Driver models
GPT-5.4 (OH/OC), Claude Opus 4.6 (CC)
Base model
Qwen2.5-7B-Instruct
Time budget
12h single run per system/task/mode where applicable
CC modes
SFT-only, RL-oriented, Free, Multi
Appendix
Table 7: Experimental setup used for the main experiments.
Route
Operational definition
GRPO
train.py contains explicit GRPO-style training components
PPO
train.py contains explicit PPO-style training components
SFT-only
train.py contains supervised fine-tuning components but no explicit RL trainer
SFT+RL
train.py contains both supervised and RL-style training components
Copy
the run copies or reuses the base model without substantive training logic
Placeholder / Missing
the training script is absent or effectively a placeholder rather than executable training code
Appendix
Table 8: Route labels used in the technical-route analysis.
Rule category
Implemented behavior
Test-set isolation
Held-out evaluation data are not mounted into the agent workspace. Agents only receive training-visible data and task descriptions.
Submission interface
Scores are produced only through the grading server. Agents cannot directly overwrite leaderboard files or final result tables.
Baseline reuse
Trivial submission of the untouched base model is treated as non-improving behavior and maps back to baseline performance.
Artifact logging
Each run stores score trajectories, run metadata, and workspace artifacts for later inspection.
Evaluator serialization
Heavyweight evaluation is serialized within a run to reduce contention and make score traces easier to audit.
Path-based caching
Identical submissions may be cached by resolved path and file-modification state to avoid repeated redundant evaluation, without changing the best-score semantics.
Appendix
Table 9: Implemented audit and anti-cheating rules.
Failure type
Typical task
Observable symptom
Main audit artifact
Rollout construction failure
ALFWorld, WebShop
The run produces training code, but the environment loop is incomplete, brittle, or never yields a usable trajectory set.
train.py , task-specific eval.py , missing or degenerate submissions
Reward/data design failure
WebShop, DeepSearchQA
The run implements non-trivial collection and training logic, yet the resulting data or reward signal is too sparse, brittle, or misaligned to improve performance.
Generated training script, score trajectory, failure notes in summary.md
Training collapse / instability
WebShop, ALFWorld
Early submissions exist, but later iterations regress or fail during model loading, runtime startup, or evaluation.
scores.json , run logs, final workspace artifacts
Early coding failure
All tasks, especially weaker-model settings
The run terminates before producing substantive training code or a valid submission.
Missing or placeholder train.py , zero-submission runs
Static shortcut without transfer
GSM8K, HumanEval versus L3 tasks
The scaffold can assemble a workable static SFT/RL route, but the same competence does not carry over once interactive rollout becomes mandatory.
Cross-task comparison of route labels, submissions, and improvements
Appendix
Table 10: Failure taxonomy used for offline diagnosis of interactive-task runs.
Task
Mode
Subs
Code
Models
Zero%
Best@Sub
Primary Strategies
GSM8K
SFT
92
14
146
20%
#90
SFT, model soup, seed sweep
HumanEval
SFT
195
28
336
23%
#26
SFT, model averaging, task arithmetic
AlpacaEval
Free
23
11
47
0%
#23
SFT → DPO, synthetic preference gen.
ALFWorld
Free
4
4
5
0%
#4
Demonstration SFT (env. interaction)
WebShop
Free
16
4
20
19%
#13
Trajectory collection → SFT
DeepSearchQA
Free
145
47
285
0%
#39
DPO, GRPO, KTO, SLERP, RFT
Appendix
Table 11: Exploratory Claude Code workspace statistics per task. Submissions = total valid grading requests; Code files = distinct .py files written under code/ ; Model versions = directories under output/ ; Zero rate = fraction of submissions scoring 0 because training or evaluation did not complete successfully; Best@Sub = submission number at which the best score was achieved.
Parameter
Under-fit
Sweet spot
Over-fit
Learning rate
5e-5 → 10.5
1e-4 → 11 to 13
2e-4 → 4.0
LoRA rank
16 → 10.5
32 → 11 to 13
64 → 4.0 to 7.5
Epochs
2 → 10.5
3 to 4 → 11 to 13
5+ → 6.5
Training loss
> 1.5
1.1 to 1.2
< 0.5
Appendix
Table 12: DeepSearchQA SFT sensitivity analysis from early local sweeps (Qwen3-8B-Base, 12h). The optimization landscape has a narrow sweet spot; deviations in any direction degrade performance sharply.
Figure 5: Mode × Task improvement heatmap for Claude Code. Stars mark the best mode per task; triangles mark the worst. Rank reversals between modes confirm that no single training paradigm uniformly dominates.
Mode
GSM8K
HE
AE
ALF
WS
DSQA
SFT-only
84.00
81.71
21.32
67.91
76.00
14.00
RL-oriented
81.88
81.71
20.96
93.28
79.00
15.00
Free
82.26
79.27
30.76
95.52
84.00
13.00
Multi
85.06
75.61
19.54
13.43
79.00
11.50
Appendix
Table 13: Claude Code mode ablation (Qwen2.5-7B-Instruct, 12h). Bold = best, underline = worst per task.
Task
RL Methods Tried
Stable?
Best Pipeline
GSM8K
GRPO ( × 15 variants)
Partial
SFT + model soup
HumanEval
GRPO, DPO, RFT
No
SFT + model averaging
AlpacaEval
GRPO, DPO ( × 31)
Yes (DPO)
SFT → DPO
ALFWorld
GRPO + online rollout
Yes
SFT → GRPO (RL-oriented)
WebShop
DPO, ORPO, KTO
Partial
Traj. collection → SFT
DeepSearchQA
DPO, GRPO, KTO, ORPO
Partial
SFT → DPO pipeline
Appendix
Table 14: RL method audit across Claude Code workspaces. “Stable” means that training completed reliably and produced non-degenerate results.
Mode
GSM8K
HumanEval
AlpacaEval
ALFWorld
WebShop
DSQA
Free
74.00
84.15
26.91
97.76
78.00
23.00
Multi
80.97
60.98
10.33
85.82
13.00
7.00
Baseline
83.02
62.20
14.68
8.96
0.00
12.25
Appendix
Table 15: Multi-task degradation on Qwen3-8B-Base (12h). Free mode scores are from single-task runs. The pattern mirrors Table 3 : Multi wins on GSM8K but severely degrades interactive tasks.
L1: Static Rule
L2: Judge
L3: Interactive Rollout
Scaffold
Driver
GSM8K ( Δ )
HumanEval ( Δ )
AlpacaEval ( Δ )
ALFWorld ( Δ )
WebShop ( Δ )
DSQA ( Δ )
N/A
Baseline
83.02
62.20
14.68
8.96
0.00
12.25
Claude Code
Opus 4.6
88.78 ( +5.76 )
84.15 ( +21.95 )
26.91 ( +12.23 )
97.76 ( +88.80 )
78.00 ( +78.00 )
23.00 ( +10.75 )
Claude Code
Sonnet 4.5
88.70 ( +5.68 )
65.85 ( +3.65 )
19.80 ( +5.12 )
15.67 ( +6.71 )
68.00 ( +68.00 )
18.50 ( +6.25 )
Codex CLI
GPT-5.4
84.00 ( +0.98 )
72.25 ( +10.05 )
22.82 ( +8.14 )
90.93 ( +81.97 )
57.00 ( +57.00 )
21.00 ( +8.75 )
Codex CLI
GPT-5.2
89.46 ( +6.44 )
69.51 ( +7.31 )
18.67 ( +3.99 )
87.31 ( +78.35 )
75.00 ( +75.00 )
12.00 ( − 0.25 )
Appendix
Table 16: Complete Qwen3-8B-Base controlled study (12h, single run). Δ computed against canonical baseline [83.02 / 62.20 / 14.68 / 8.96 / 0.00 / 12.25].
Dimension
Claude Code
Codex
Driver LLM
Claude Opus 4.6
GPT-5.4
Submissions
16
56
Iteration style
lower-frequency, more selective
higher-frequency, broader search
Best Δ
+10.75
+8.75
Appendix
Table 17: Agent behavior comparison on DeepSearchQA (Qwen3-8B-Base, 12h).
Figure 6: Scaling analysis for the three static tasks across all agent stacks. Top: Time scaling. Middle: Token scaling. Bottom: Submission scaling. Compared with the interactive-task scaling in Figure 2 , static tasks typically plateau earlier and show weaker separation between strong agent stacks.
Figure 7: Improvement over canonical Qwen3-8B-Base baseline ( Δ in percentage points) for six agent stacks across all tasks (12h). Interactive tasks (ALFWorld, WebShop) show larger observed differences across driver and scaffold choices than static tasks.
Method
Attempted cells
Adopted in best
Adoption rate
SFT (plain)
19
18
95%
Model merging
11
4
36%
RFT
5
3
60%
GRPO
8
2
25%
DPO
4
1
25%
KTO / ORPO
3
0
0%
Appendix
Table 18: Method attempt vs. adoption frequency across reconstructable Claude Code 7B cells. "Attempted" counts cells in which at least one submission is tagged with the method; "Adopted in best" counts cells whose highest-scoring submission used the method.
Sub #
Script
Score
Method
baseline
N/A
62.20%
Qwen3-8B-Base (unaligned)
#3
sft_train.py
2.44%
SFT v1
#5
sft_train_v3.py
43.90%
SFT v3 (prompt alignment)
#7
grpo_train.py
79.27%
GRPO on SFT checkpoint (LoRA)
#11
rft_train.py
84.15%
RFT (rejection sampling fine-tuning)
Appendix
Table 19: Submission progression for the HumanEval trace walkthrough (Claude Code Free, Qwen3-8B-Base). Each row corresponds to a qualitatively distinct strategy shift.
Sub #
Script
Score
Method
baseline
N/A
8.96%
Qwen3-8B-Base (unaligned)
#1
train_sft.py
14.93%
SFT v1: ReAct examples
#2
train_sft_v2.py
16.42%
Hyperparameter tuning
#4
train_sft_v4.py
19.40%
Action format alignment ( put → move ) + synthetic data
#5
train_sft_v5.py
45.52%
Model rollout trajectories (+26.1pp)
#7
train_sft_v7.py
95.52%
Prompt format alignment (+50.0pp)
Appendix
Table 20: Submission progression for the ALFWorld trace walkthrough (Claude Code Free, Qwen3-8B-Base, 12h). Shaded rows mark the two largest score jumps.
AI agents have become surprisingly proficient at software engineering over the past year, largely due to improvements in reasoning capabilities. This raises a deeper question: can these systems extend their capabilities to automate AI research itself? In this paper, we explore post-training, the critical phase that turns base LLMs into useful assistants. We introduce PostTrainBench to benchmark how well LLM agents can perform post-training autonomously under bounded compute constraints (10 hours on one H100 GPU). We ask frontier agents (e.g., Claude Code with Opus 4.6) to optimize the performance of a base LLM on a particular benchmark (e.g., Qwen3-4B on AIME). Importantly, we do not provide any predefined strategies to the agents and instead give them full autonomy to find necessary information on the web, run experiments, and curate data. We find that frontier agents make substantial progress but generally lag behind instruction-tuned LLMs from leading providers: 23.2% for the best agent vs. 51.1% for official instruction-tuned models. However, agents can exceed instruction-tuned models in targeted scenarios: GPT-5.1 Codex Max achieves 89% on BFCL with Gemma-3-4B vs. 67% for the official model. We also observe several failure modes worth flagging. Agents sometimes engage in reward hacking: training on the test set, downloading existing instruction-tuned checkpoints instead of training their own, and using API keys they find to generate synthetic data without authorization. These behaviors are concerning and highlight the importance of careful sandboxing as these systems become more capable. Overall, we hope PostTrainBench will be useful for tracking progress in AI R&D automation and for studying the risks that come with it. Website and code are available at https://posttrainbench.com/.
Ben Rank, Hardik Bhatnagar, Ameya Prabhu +4
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +2
AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across these approaches. We present Agentick, a benchmark for sequential decision-making agents designed to evaluate RL, LLM, VLM, hybrid, and human agents on common ground and to power research on the fundamental challenges of sequential decision-making. Agentick provides 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities, all exposed through a single Gymnasium-compatible interface. The benchmark ships with a Coding API, oracle reference policies for all tasks, pre-built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation spanning 27 configurations and over 90,000 episodes reveals that no single approach dominates: GPT-5 mini leads overall at 0.309 oracle-normalized score while PPO dominates planning and multi-agent tasks; the reasoning harness multiplies LLM performance by 3-10x; and ASCII observations consistently outperform natural language. These findings highlight the substantial room for improvement that remains across all agent paradigms. Agentick's capability-decomposed, multi-modal design provides the empirical infrastructure needed to drive progress toward general autonomous agents, both as an evaluation framework and as a training ground for RL post-training of foundation models in truly sequential environments.
Roger Creus Castanyer, Pablo Samuel Castro, Glen Berseth
Mila Quebec AI Institute · Université de Montréal · Mila Quebec AI Institute Université de Montréal +1
Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior.