Batched Contextual Reinforcement II: Does RLVR Need Long Rollout Windows?

Bangji Yang; Hongbo Ma; Ruihan Guo; Jiajun Fan; Xi Zhu; Ge Liu

cs.LGOct 16, 2026

Batched Contextual Reinforcement II: Does RLVR Need Long Rollout Windows?

Authors: Bangji Yang, Hongbo Ma, Ruihan Guo, Jiajun Fan, Xi Zhu, Ge Liu

Organizations: University of Illinois at Urbana-Champaign · Tsinghua University · Rutgers University

Abstract

Long rollout windows make reinforcement learning with verifiable rewards (RLVR) expensive, yet simply shortening them can remove the correctness signal: when every single-problem rollout truncates before an answer, group-relative accuracy advantages vanish. As a technical continuation of Batched Contextual Reinforcement (BCR), we study how concurrent rewards relax this dependence on long windows. Concurrent Constraint Reinforcement (CCR) places three independently verifiable problems in one capped training completion. Correct answers completed before truncation retain credit, while a shared budget encourages shorter useful reasoning without an explicit length penalty. We connect this mechanism to conditional reward- and update-variance analyses and study 4B and 8B mathematical reasoning models. Against single-problem GRPO with a 32k window, CCR with an 8,000-token window reduces tokens per training step from 975,464 to 174,957 on 4B and from 793,832 to 165,788 on 8B, saving 82.1% and 79.1%, respectively. An 8B training trace shows increasing accuracy reward alongside shorter completions. Single-problem evaluations provide supporting evidence of retained reasoning quality. These results identify reward-preserving training structure as a practical route to lower per-step cost in short-window RLVR.

Figures & tables

Figure 1: Standard GRPO versus CCR. Top: each candidate solves one problem with a long window and receives a terminal correctness reward. Bottom: each CCR candidate attempts grouped problems sequentially under a shorter shared cap. Correct completed answers retain credit despite suffix truncation; correctness is averaged over all assigned problems for the update. Checks, crosses, and slashes denote correct answers, incorrect answers, and truncation. Shorter ribbons illustrate training-token efficiency, not measured ratios. Reward averaging motivates the conditional stability analysis in Section 4 .

Candidate	Correct prefix	$r_{\mathrm{acc}}$	$a_{j}$
$y_{1}$	0 answers	$0$	$-1/3$
$y_{2}$	1 answer	$1/3$	$0$
$y_{3}$	2 answers	$2/3$	$+1/3$
$y_{4}$	1 answer	$1/3$	$0$

Table 1: Illustrative

N=3

S=4

group. All candidates truncate before completing the group. Their mean accuracy reward is

1/3

; completed prefix answers preserve relative feedback.

Figure 2: CCR reduces tokens per training step by 82.1% on 4B and 79.1% on 8B. Filled berry markers denote CCR with three problems and an 8,000-token shared window; hollow clay markers denote single-problem GRPO with a 32k window. The common linear axis shows the absolute per-step counters. Both comparisons use eight A100 GPUs and matched remaining training parameters.

Figure 3: Accuracy improves as training completions shorten. The 8B CCR run uses

N=3

and an 8,000-token shared cap. Faint traces show raw observations over steps 1–1,000; solid traces show trailing 50-step means of accuracy reward and joint-completion length.

	Qwen3-4B-Thinking				Qwen3-8B-ODA-Math
	Accuracy (%)		Eval. tokens		Accuracy (%)		Eval. tokens
Benchmark	Base	CCR	Base	CCR	Base	CCR	Base	CCR
AIME24	83.3	83.3	18,082	14,539	63.3	70.0	16,549	13,680
AIME25	70.0	83.3	20,773	17,498	66.7	60.0	17,832	13,693
AMC23	97.5	100.0	10,457	7,128	95.0	97.5	6,988	4,516
MATH-500	97.0	97.4	5,136	3,173	95.4	94.4	4,010	2,773

Table 2: Reasoning quality after short-window training. CCR is trained with

N=3

and evaluated on single-problem prompts. Accuracy is final-answer correctness (%); tokens are evaluation lengths. Base denotes the starting checkpoint.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Item	Setting
CCR backbones	4B-Thinking; 8B-ODA-Math
Problems/completion	$N=3$
CCR completion cap	8,000 tokens (both models)
GRPO cost baselines	4B and 8B, one problem, 32k cap
Hardware	8 NVIDIA A100 GPUs
CCR reward weights	Accuracy 2; format 1

Appendix

Table 3: Training and evaluation configuration.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards
Mar 2, 2026Guanzheng Chen, Michael Qizhe Shieh, Lidong BingRL for Language Models Reinforcement Learning with Verifiable Rewards
National University of Singapore · MiroMind AI · absolute AI
Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation
Oct 12, 2026Haifeng Qian, David Mosallanezhad, Christian Munley +2Reinforcement Learning RL for Language Model Reasoning
NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051, USA
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
Oct 14, 2026Wei Chen, Yubing Wu, Zhou Xu +9RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
South China University of Technology · Tsinghua University · The Hong Kong University of Science and Technology +3

Mar 2, 2026J/K move · Enter open · S save

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

Guanzheng Chen, Michael Qizhe Shieh, Lidong Bing

National University of Singapore · MiroMind AI · absolute AI

RL for Language Models Reinforcement Learning with Verifiable Rewards LLM Grounding Reward Shaping Long-Context Language Model Evaluation

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) by optimizing them against factual outcomes. However, this paradigm falters in long-context scenarios, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contextual grounding--the ability to find and reason over externally provided information. We identify a key reason for this failure: a reward based solely on the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, rendering learning intractable. To overcome this bottleneck, we introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the correct grounding information, providing a robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long-context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process is a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is available at https://github.com/real-absolute-AI/LongRLVR.

Abstract

Reinforcement Learning

Explore similar work

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling