Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

Jiarui Qin; Junjie Huang; Yunjia Xi; Di Yin; Yong Yu; Weinan Zhang; Xing Sun; Weiwen Liu

cs.AIOct 14, 2026

Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

Authors: Jiarui Qin, Junjie Huang, Yunjia Xi, Di Yin, Yong Yu, Weinan Zhang, Xing Sun, Weiwen Liu

Organizations: Tencent Youtu Lab · Shanghai Jiao Tong University

Abstract

Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior.

Figures & tables

Figure 1: ASAP enables end-to-end evaluation of pretrained base models. (A) A short prefix from a reference trajectory. (B) The frozen base model continues the trajectory through its native completion interface. (C) Minimal recovery handles premature empty submissions and repeated actions.

Model	Pass@1	Pass@8	#Inter	Pass@1	Pass@8	#Inter
	SWE-bench Verified		SWE-agent	SWE-bench Verified		OpenHands
Qwen3-30B-A3B-Base	7.38	24.20	0	4.00	14.40	0
Qwen3.5-9B-Base	21.85	46.60	16	32.95	55.40	51
Qwen3.5-35B-A3B-Base	34.23	56.20	67	39.23	61.80	69
Gemma4-26B-A4B-Base	7.12	23.00	0	4.67	18.20	0
GLM-4.5-Base	15.97	41.40	4	19.73	44.80	11

Table 1: Main results (%) across agentic benchmarks. We report Pass@1, Pass@8, and the number of instances resolved in all eight runs (#Inter). Full Pass@k results are provided in Appendix A.3 . Bold : best result within each benchmark–scaffold setting.

Training phase	General	LR decay	Long-context training
	Stage1	Stage2	Stage3	Stage4	Stage5
Context length	8K	8K	32K	128K	256K
Trajectory data	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$

Table 2: Training stages of the internal 30B-A3B MoE base model

M

. Trajectory-like data is first introduced at stage3.

Figure 2: Pretraining progress of internal model

M

. (a) ASAP Pass@k with SWE-agent on SWE-bench Verified across nine checkpoints. (b) ASAP scores sensitivity within stage1 to stage4, comparing with the conventional pretraining evaluations.

Figure 3: Behavioral change across different pretraining stages. Stage2 brings parsability and valid tool calls close to saturation, whereas the main transition in autonomous editing, active testing, applicable patches, and task resolution occurs at stage3.

	Qwen3.5-35B-A3B			MiMo-V2.5
Metric	Base	Post	$\Delta$	Base	Post	$\Delta$
Pass@1 / success rate	34.23	49.40	+15.17	37.70	74.00	+36.30
Valid source edit	69.99	79.41	+9.42	62.12	96.34	+34.22
Active testing	85.90	96.80	+10.90	76.80	99.77	+22.97
Valid re-edit after a failed test	31.80	63.78	+31.98	28.28	57.27	+28.99
Explicit submission	20.57	52.17	+31.60	26.81	89.02	+62.21

Table 3: Behavioral differences between base and post-trained models under their respective evaluation protocols.

\Delta

is in percentage points.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Temperature	0.7
top_p	0.95
Completions calls per step	1
Generated span per step	Thinking + action
max_tokens (per step)	2,048
max_steps	30

Appendix

Table 4: Key ASAP configuration parameters.

Model	Pass@1	Pass@2	Pass@4	Pass@8	#Inter
SWE-bench Verified Scaffold: SWE-agent
Qwen3-30B-A3B-Base	7.38	12.32	18.37	24.20	0
Qwen3.5-9B-Base	21.85	30.61	38.81	46.60	16
Qwen3.5-35B-A3B-Base	34.23	42.82	50.15	56.20	67
Gemma4-26B-A4B-Base	7.12	11.84	17.60	23.00	0
GLM-4.5-Base	15.97	24.46	33.47	41.40	4

Appendix

Table 5: Main results (%) across agentic benchmarks. #Inter is the number of instances resolved in all 8 runs. For SWE-bench Verified, we evaluate each model with both SWE-agent and OpenHands; Terminal-Bench 2.1 uses the Terminus-2 scaffold; BrowseComp-Plus uses its native scaffold. Bold : best in panel.

Model	Format	Pass@1	Pass@3
Qwen3.5-35B-A3B	JSON	4.33	13.00
	Markdown	21.67	35.00
	XML	27.00	38.00
	Markdown+XML	33.00	50.00
MiMo-V2.5-Base	JSON	14.67	30.00
	Markdown	30.67	47.00

Appendix

Table 6: Input-format ablation on a randomly sampled 100-instance subset of SWE-bench Verified using the SWE-agent scaffold. We sample three trajectories per instance and report Pass@1 and Pass@3 using the unbiased Pass@

k

estimator.

Model	Prefix length $K$	Pass@1	Pass@3
Qwen3.5-35B-A3B	0	24.00	41.00
	1	27.67	43.00
	3 (selected)	33.00	50.00
MiMo-V2.5-Base	0	28.00	46.00
	1	32.00	53.00
	3 (selected)	42.00	59.00

Appendix

Table 7: Prefix-length ablation on 100 randomly sampled SWE-bench Verified instances. Results (%) are computed from three independent runs. Increasing the prefix length consistently improves performance while preserving the relative model ordering.

Model	Metric	Prefix A	Prefix B	$\Delta$
Qwen3.5-35B-A3B	Pass@1	33.00	34.00	$+1.00$
Qwen3.5-35B-A3B	Pass@3	50.00	48.00	$-2.00$
MiMo-V2.5-Base	Pass@1	42.00	44.33	$+2.33$
MiMo-V2.5-Base	Pass@3	59.00	56.00	$-3.00$

Appendix

Table 8: Robustness to independently constructed reference prefixes on the same 100-instance subset of SWE-bench Verified. Prefix A denotes our default prefix strategy, while Prefix B is independently generated by GLM-5.3-Flash.

\Delta

denotes Prefix B minus Prefix A.

	Base / ASAP		70K SFT
Model	Pass@1	Pass@8	Pass@1	Pass@3
Qwen3-30B-A3B	7.38	24.20	19.60	31.80
Qwen3.5-9B	21.85	46.60	24.60	37.60
Qwen3.5-35B-A3B	34.23	56.20	37.47	53.20
Gemma4-26B-A4B	7.12	23.00	18.70	31.60

Appendix

Table 9: Direct base-model evaluation with ASAP and end-to-end SWE-agent evaluation after the same 70K-example lightweight SFT. Base results use eight runs and SFT results use three runs. All values are percentages. Bold : best in column.

Model	Trajectories	Network	Network Rate	Gold	Gold Rate	Docs
Qwen3-30B-A3B-Base	4,000	86	2.15%	9	0.23%	23
Gemma4-26B-A4B-Base	4,000	25	0.63%	5	0.13%	8
Mimo-V2.5-Base	4,000	9	0.23%	1	0.03%	0
Qwen3.5-35B-A3B-Base	4,000	9	0.23%	1	0.03%	2
Qwen3.5-9B-Base	4,000	8	0.20%	1	0.03%	2
GLM-4.5-Base	4,000	7	0.18%	0	0.00%	6

Appendix

Table 10: Trajectory-level audit of attempted external network access. “Gold” denotes accesses that appear to seek a reference solution or patch, while “Docs” denotes accesses to external documentation.

Metric	Qwen3.5-35B-A3B-Base	MiMo-V2.5-Base
Pass@1	99/300 (33.0%)	126/300 (42.0%)
Pass@3	50/100 (50.0%)	59/100 (59.0%)
Empty patch	73/300 (24.3%)	101/300 (33.7%)
Non-empty unsuccessful/error	128/300 (42.7%)	73/300 (24.3%)
Non-empty success rate	43.6%	63.3%
Repetition-based termination	9/300 (3.0%)	24/300 (8.0%)

Appendix

Table 11: Performance and failure patterns on SWEBench Verified using ASAP . “Non-empty success rate” denotes the fraction of submitted non-empty patches that pass the official grader.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Apr 12, 2026Wanyi Chen, Xiao Yang, Xu Yang +7Agentic RL AI Agent Benchmarks
Soochow University · Microsoft Research Asia · Stony Brook University +2
Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models
Oct 7, 2026Tan Yu, Alexander Bukharin, Khushi Bhardwaj +19AI Coding Agents AI Agent Evaluation
NVIDIA · University of Minnesota – Twin Cities · University of California, Berkeley
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
May 21, 2026Asaf Yehudai, Lilach Eden, Michal Shmueli-ScheuerLLM Agent Evaluation AI Agent Benchmarks
IIBM Research

Apr 12, 2026J/K move · Enter open · S save

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

Wanyi Chen, Xiao Yang, Xu Yang, Tianming Sha+6

Soochow University · Microsoft Research Asia · Stony Brook University+2

Agentic RL AI Agent Benchmarks LLM Post-Training LLM Agent Evaluation

Open paper AskPDF arXiv

We introduce Agent2 RL-Bench, a compact diagnostic benchmark for evaluating agentic RL post-training, which tests whether LLM agents can autonomously design, implement, debug, and execute post-training pipelines that improve foundation models. RL post-training increasingly drives model alignment and specialization, yet existing benchmarks are largely static, rewarding supervised fine-tuning or script generation without assessing an agent's ability to close an interactive RL loop. Agent2 RL-Bench provides a unified agent-facing interface: each run starts from an isolated workspace containing a base model, task data, instructions, and a grading API, and agents must iterate within a fixed budget by training models and submitting artifacts for evaluation. The benchmark spans six tasks across three levels, from static rule-based training to judge-based optimization and closed-loop online RL with trajectory collection. Two diagnostic skills, namely runtime recording and post-hoc summarization, enable structured analysis of agent behavior, facilitating smooth and effective iteration of the benchmark's evaluation framework. Across five agent systems and six driver LLMs, agents show intelligent behavior but clear limitations: one RL-oriented run improves ALFWorld from 4.85 to 93.28 via SFT warm-up and GRPO with online rollouts, yet DeepSearchQA remains difficult, most successful routes rely on supervised pipelines, and interactive outcomes show large single-run differences across agent stacks. Overall, Agent2 RL-Bench shows that current agents can sometimes engineer online RL, but stable agent-driven RL post-training remains rare under fixed budgets. It also demonstrates that our benchmark provides a strong and effective evaluation framework for future research in this direction. Code is available at https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/rl/autorl_bench/README.md

Abstract

LLM Agent Evaluation

Explore similar work

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models

Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents