Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agents

Jihan Yao; Sihan Zeng; Shangbin Feng; Zhiyuan Fan; Banghua Zhu; Yulia Tsvetkov

cs.CLSep 25, 2026

Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agents

Authors: Jihan Yao, Sihan Zeng, Shangbin Feng, Zhiyuan Fan, Banghua Zhu, Yulia Tsvetkov

Organizations: University of Washington · Independent Researcher · HKUST

Abstract

Long-horizon coding agents receive verifiable rewards only after completing expensive sequences of tool calls. This increases inference cost, amplifies early wrong hypotheses, and can lead to sparse terminal reward and unstable training. We introduce Contextual Early Reward (CER), which predicts terminal reward through behavioral evidence in a trajectory prefix. CER synthesizes adaptive rubrics specific to the current task and stage through experiences summarized from related historical tasks. In test-time scaling on SWE-bench Verified, CER improves RM@8 over the strongest baseline by 4.2 percentage points (pp) on Nemotron 3 Ultra and 2.0 pp on Qwen 3.6 27B; on Nemotron, it takes only 15.3% tokens to match the best baseline performance. In RL training experiments, CER exceeds full-rollout TMax by 1.9 pp while using 52.7% fewer online policy-and-judge tokens. Together, CER provides an interpretable, efficient, and dense evaluation method for long-horizon coding agents.

Figures & tables

Figure 1: CER overview. At cutoff

t

, CER scores a group of unfinished rollout prefixes with contextual rubrics to predict their final rewards. For test-time scaling, rubrics are generated from retrieved cross-task experience and comparable prefix samples, and only top-ranked prefixes are expanded. For reinforcement learning, online rollouts stop at

t

, and teacher-summarized rubrics score each prefix to provide group-relative advantages for the policy update. With access to verified rollouts, teacher rubrics provide finer-grained and more accurate signals.

	Nemotron 3 Ultra				Qwen 3.6 27B
Method	RM@4	RM@8	Top-tie	Tok./Ins.	RM@4	RM@8	Top-tie	Tok./Ins.
	% $\uparrow$	% $\uparrow$	% $\downarrow$	M $\downarrow$	% $\uparrow$	% $\uparrow$	% $\downarrow$	M $\downarrow$
SWE-RM	62.2	61.6	17.8	25.67	66.2	67.6	18.0	5.27
OpenHands Critic@20	57.6	57.8	3.2	3.91	64.4	64.2	1.6	1.57
OpenHands Critic@40	60.4	58.4	2.8	5.70	65.0	64.0	0.6	2.99
OpenHands Critic@60	61.8	60.4	2.4	7.93	65.4	63.4	1.2	4.07

Table 1: Test-time scaling results on SWE-bench Verified. Top-tie is the fraction of tasks having trajectories receive the equal highest score. Token usage includes online rollout and judging costs. The best results are in bold , and the second-best results are underlined , excluding random selection and best-of-N. With only 15.3% and 74.2% of the required tokens on Nemotron 3 Ultra and Qwen 3.6 27B, CER matches the best execution-free baseline performance. With no more computation, CER improves upon the best baseline by 4.2 and 2.0 pp on the two models, respectively.

Method	Fold 0	Fold 1	Fold 2	Mean
Qwen 3.5 9B	$44.6\,\pm\,1.2$	$43.1\,\pm\,1.7$	$45.6\,\pm\,2.7$	$44.5\,\pm\,0.8$
TMax	$49.7\,\pm\,1.0$	$49.1\,\pm\,2.7$	$50.3\,\pm\,2.0$	$49.7\,\pm\,1.1$
TMax-40	$49.6\,\pm\,1.6$	$49.1\,\pm\,2.3$	$50.7\,\pm\,1.9$	$49.8\,\pm\,0.8$
CER	$\mathbf{52.7\,\pm\,2.1}$	$\mathbf{49.8\,\pm\,2.1}$	$\mathbf{52.4\,\pm\,1.0}$	$\mathbf{51.6\,\pm\,0.9}$

Table 2: Mean test resolved rate (%)

\pm

95% CI on the test set, computed from the variance of eight evaluation replicates (one rollout per test instance each); CIs reflect evaluation sampling only, not training variance. CER exceeds full-rollout TMax by 1.9 pp.

Figure 2: Mean resolved rate versus cumulative accounted training tokens across three folds on SWE-bench Verified. Light curves show individual folds, bold curves show their arithmetic means, and numbers indicate the actual trained groups. CER shows a clear trend toward more stable training, lower token usage, and better final performance.

CER	@20	@40	@60	@80
Nemotron	63%	68%	68%	69%
$-$ w/ Qwen exp.	64%	69%	70%	70%
Qwen	73%	75%	76%	75%
$-$ w/ Nemotron exp.	67%	68%	69%	70%

Table 3: Cross-model experience transfer on the same 100-instance subset. Entries are resolved RM@8 (%) at each budget

k

. Experience from the stronger model improves the weaker model’s TTS performance, whereas the reverse transfer is detrimental.

Figure 3: Token entropy across the first 100 turns of TMax training rollouts. Light lines show fold means and the dark line shows the pooled mean; shading denotes the 10th–90th percentile across trajectory-level turn means within each turn. The dashed line marks CER’s 40-turn cutoff.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Rubric–verifier pairwise agreement throughout the CER runs used in Figure 2 . Light lines show individual folds and the dark line is the pair-count-weighted mean. Only pairs with different terminal rewards are included; rubric-score ties count as incorrect.

Fold	Train (250)	Validation (50)	Test (200)
0	django/django (103), matplotlib/matplotlib (27), psf/requests (4), pydata/xarray (16), sphinx-doc/sphinx (36), sympy/sympy (64)	django/django (13), matplotlib/matplotlib (7), pallets/flask (1), psf/requests (4), pydata/xarray (6), sphinx-doc/sphinx (8), sympy/sympy (11)	astropy/astropy (22), django/django (115), mwaskom/seaborn (2), pylint-dev/pylint (10), pytest-dev/pytest (19), scikit-learn/scikit-learn (32)
1	django/django (102), pydata/xarray (16), pylint-dev/pylint (6), scikit-learn/scikit-learn (25), sphinx-doc/sphinx (36), sympy/sympy (65)	django/django (13), mwaskom/seaborn (2), pydata/xarray (6), pylint-dev/pylint (4), scikit-learn/scikit-learn (7), sphinx-doc/sphinx (8), sympy/sympy (10)	astropy/astropy (22), django/django (116), matplotlib/matplotlib (34), pallets/flask (1), psf/requests (8), pytest-dev/pytest (19)
2	astropy/astropy (14), django/django (207), matplotlib/matplotlib (24), pylint-dev/pylint (5)	astropy/astropy (8), django/django (24), matplotlib/matplotlib (10), mwaskom/seaborn (2), pallets/flask (1), pylint-dev/pylint (5)	psf/requests (8), pydata/xarray (22), pytest-dev/pytest (19), scikit-learn/scikit-learn (32), sphinx-doc/sphinx (44), sympy/sympy (75)

Appendix

Table 4: Repository and instance distribution of the three best-effort repository-separated RL folds. Fold 2 is fully repository-disjoint; Django is the only repository shared between test and model-development data in Folds 0 and 1.

Component	Hyperparameter	Value
Rollout	Samples per task group	8
	Samples per global batch	128
	Sampling temperature / top- $p$	1.0 / 0.95
	Context / Completion maximum length	65,536 / 20,480 tokens
	Rollout step limit	250 (TMax and TMax-40); 40 (CER)
	Dynamic sampling	true

Appendix

Table 5: Full RL training and evaluation hyperparameters, shared across all three methods unless explicitly distinguished.

	Step Budget	20	40	60	80
	Subset Size	453/386	313/98	188/17	110/4
Nemotron 3 Ultra	SWE-RM	59.4%	52.4%	44.7%	40.9%
	Adaptive Rubrics	61.1%	54.0%	48.4%	43.6%
	CER	62.9%	56.2%	50.5%	45.5%
Qwen 3.6 27B	SWE-RM	63.0%	44.9%	-	-
	Adaptive Rubrics	64.8%	46.9%	-	-

Appendix

Table 6: RM@8 on fully unsubmitted subsets only. At each cutoff, we retain only tasks for which every candidate trajectory remains unsubmitted. All compared methods are evaluated on the same resulting task subset. We compare with the strongest two terminal reward baselines in the main TTS experiments, SWE-RM and Adaptive Rubrics. The two subset sizes are of model Nemotron 3 Ultra and Qwen 3.6 27B respectively. We report results only for subsets containing more than 50 tasks for statistical stability.

Model	@20	@40	@60	@80
Nemotron 3 Ultra	55.0%	55.7%	55.9%	58.8%
Qwen 3.6 27B	50.7%	51.4%	55.4%	60.4%

Appendix

Table 7: The percent of retrieved experience summarized from the same-repository tasks in TTS experiments. CER leverages slightly more in-domain experience than out-of-domain experience.

Method	@20	@40	@60	@80	Avg.
Nemotron 3 Ultra
CER	63	68	68	69	67.0
$-$ w/o tie-breaking	63	67	68	68	66.5
$-$ w/o regression	-	67	66	66	65.5
$-$ w/o beam parent	-	66	65	65	64.8
$-$ w/o experience	65	67	63	68	65.8

Appendix

Table 8: TTS component ablation on a 100-instance SWE-bench Verified subset. Entries are resolved RM@8 (%) at each budget

k

; regression and beam parents do not affect the first round (-), so Avg. uses the CER value at @20 for these rows. The best results per model are in bold , and the second-best results are underlined .

Figure 5: Training dynamics of TMax, TMax-40, and CER. From left to right and top to bottom: the fraction of sampled groups filtered, mean reward over all sampled groups, entropy loss, and mean assistant steps in the complete sampled rollouts. Filtered-group rates include zero-variance groups but exclude discarded stale groups. The full-rollout step diagnostic does not represent the prefix length used for policy updates or actual training cost. We only rollout the prefix to terminal for analysis. Curves are means over three repository folds and shading spans the fold-wise minimum and maximum.

Figure 6: (a) AUROC of using the cumulative fraction of golden-patch target files touched to predict terminal verifier reward, computed on one sampled Qwen 3.5 9B rollout for each of the 500 SWE-bench Verified instances. The cumulative fraction of golden-patch target files touched is a weak indicator of terminal reward and can be predictive only after step 20. (b) Mean normalized rubric score and binary cross-entropy against terminal verifier reward, computed on one sampled Nemotron 3 Ultra rollout for each of the 500 SWE-bench Verified instances. Lower binary cross-entropy is better. Nemotron judged scores exhibit strong length bias: longer trajectories receive higher scores which lead to high inaccuracy as the trajectory length increase.

Primary category	Weighted percent	Example
Pre-edit reproduction and runnable testbed	0.65%	For Django-13033, the criterion requires executed evidence that exposes the erroneous SQL join and ordering direction and contrasts it with semantically informative control orderings.
Root-cause localization and causal diagnosis	16.82%	For Astropy-12907, the criterion requires model to identify that recursive separability yields an array for the nested right operand and that _cstack destroys its dependency structure by filling the embedded block with ones.
Fix-design convergence and authoritative grounding	0.36%	For scikit-learn-9288, the criterion goes beyond locating the serial–parallel seed mismatch by requiring a branch-invariant repair that draws one deterministic integer seed per n_init run and passes the corresponding seed to every serial invocation.
Investigation efficiency and environment recovery	3.43%	For pytest-7432, the criterion requires recovery from unusable or contradictory runtime evidence by identifying which pytest installation is executing, avoiding contamination of installed packages, and returning to authoritative repository evidence without overstating validation.
Intermediate-edit safety and workspace hygiene	1.29%	For Django-10914, the criterion preserves independent directory defaults and secure temporary-file behavior while applying mode 0644 only to final stored uploads.

Appendix

Table 9: Semantic taxonomy of rubric criteria, Part I: exploration and diagnosis. Percentages are shares of the total rubric weight over all rubrics. Each example summarizes a real criterion together with its applicability.

Primary category	Weighted percent	Example
Functional implementation correctness	18.07%	For Astropy-13398, the criterion requires a coherent implementation of bidirectional ITRS–AltAz and ITRS–HADec transforms using topocentric geometry while keeping the terrestrial position time-invariant.
Edge-case and implementation completeness	5.81%	For Django-13089, the criterion covers the empty-result edge case in which a culling query returns no row: execution must complete without subscripting None or deleting a valid cache key.
Patch scope and minimality	5.47%	For Astropy-12907, the criterion confines the repair to _cstack : copy the computed right separability matrix into its positioned block rather than overwrite that block with ones.
Compatibility and integration preservation	8.88%	For Astropy-14309, the criterion requires the FITS identifier to return false when no positional object exists without indexing an empty tuple, while preserving signature, extension, and HDU detection.
Final-code quality and maintainability	0.47%	For SymPy-16450, the criterion requires posify to transfer compatible known assumptions through replacement-symbol construction, give positive=True unambiguous precedence, and avoid private-state mutation.
Submission and patch-artifact integrity	9.48%	For Django-14915, the criterion requires a complete, repository-targeted patch in a format that the evaluator can directly recognize and ingest, rather than prose or partial diff fragments that require reconstruction.

Appendix

Table 10: Semantic taxonomy of rubric criteria, Part II: implementation and editing. Percentages and examples follow Table 9 .

Primary category	Weighted percent	Example
Causal before/after verification	1.17%	For Django-11206, the criterion requires executed post-edit evidence for the reported tiny-Decimal behavior plus counterexamples that expose raw-exponent zeroing, lost scientific safeguards, or bypassed formatting arguments.
Targeted behavioral validation	14.43%	For Astropy-13236, the criterion requires evidence that direct multi-field array input reaches ordinary Column construction without automatic NdarrayMixin conversion while explicit columns and legitimate mixins retain established handling.
Edge-case and scenario validation	0.88%	For Django-13028, the criterion distinguishes an ordinary lookup RHS whose application attribute is false from an ORM expression that intentionally prohibits filtering, using source tracing or a task-shaped boundary experiment.
Regression-suite and breadth validation	11.64%	For Astropy-14096, the criterion must establish the corrected nested-property error while protecting ordinary missing attributes and SkyCoord’s dynamic frame and transform behavior.
Execution-environment fidelity	0.57%	For Django-11880, the criterion credits the dictionary-isolation check only when it is executed against the edited checkout, rather than a toy object or a different installed version.
Failure interpretation and claim discipline	0.34%	For Django-12193, the criterion requires a consistent ledger in which a failed property is resolved only after its implementation or oracle is corrected and rerun, and success claims remain limited to demonstrated properties.

Appendix

Table 11: Semantic taxonomy of rubric criteria, Part III: validation and other nonfunctional risks. Percentages and examples follow Table 9 .

Long-Horizon Agent Evaluation

Long-Horizon Agent Evaluation measures whether AI agents can plan, act, remember, and stay reliable across extended tasks and changing environments.

See all 358 papers

The Verification Horizon: No Silver Bullet for Coding Agent Rewards
Jun 24, 2026Binghai Wang, Chenlong Zhang, Dayiheng Liu +10Reward Modeling Reward Hacking
Qwen Team
Opera: A Verbal Critic Framework for Long-horizon Coding Agents
Sep 27, 2026Kai Mei, Zhiyuan Hu, Yutong Dai +7AI Coding Agents AI Agent Auditing
Salesforce AI Research · Rutgers University · Lehigh University
OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents
Oct 15, 2026Yuechen Luo, Zhiyi Lai, Miao Ziqi +9AI Coding Agents LLM Agent Training
Tsinghua University · Independent Researcher · Shanghai Artificial Intelligence Laboratory +1

Jun 24, 2026J/K move · Enter open · S save

The Verification Horizon: No Silver Bullet for Coding Agent Rewards

Binghai Wang, Chenlong Zhang, Dayiheng Liu, Jiajun Zhang+9

Qwen Team

Reward Modeling Reward Hacking AI Coding Agents LLM Agent Verification Reinforcement Learning with Verifiable Rewards Reward Design for RL

Open paper AskPDF arXiv

A classical intuition holds that verifying a solution is easier than producing one. For today's coding agents, this intuition is being inverted: as foundation models develop stronger reasoning capabilities and engineering harnesses grow more sophisticated, generating complex candidate solutions is no longer difficult -- reliably verifying them has become the harder problem. Every verifier we can build is only a proxy for human intent, never the intent itself. This makes verification subject to a twofold difficulty: first, intent is underspecified by nature, making it inherently hard to faithfully check whether it has been fulfilled; second, during model training, optimization widens the gap between proxy and intent -- manifesting as reward hacking or signal saturation. To address this, we characterize the quality of verification signals along three dimensions -- scalability, faithfulness, and robustness -- and argue that achieving all three simultaneously is the central challenge. We further study four reward constructions: a test verifier for general coding tasks, a rubric verifier for frontend tasks, the user as verifier for real-world agent tasks, and an automated agent verifier for long-horizon tasks. Across different task types and policy capability levels, we conduct in-depth analysis and experiments on the core challenges of reward design and how to more effectively leverage reward signals. Experiments show that targeted verification design can effectively suppress reward hacking, improve task completion quality, and achieve significant gains across multiple internal and public benchmarks. These experiences collectively point to a core observation: no fixed reward function can remain effective as policy capability continues to grow; and verification must co-evolve with the generator.

Abstract

Long-Horizon Agent Evaluation

Explore similar work

The Verification Horizon: No Silver Bullet for Coding Agent Rewards

Opera: A Verbal Critic Framework for Long-horizon Coding Agents

OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents