LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards

Qize Yu; Zijun Chen; Chen Wu; Yue Chen; Yiyao Yu; Tingfeng Hui; Bowen Ping; Junchi Yan

cs.CLFeb 5, 2026

LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards

Authors: Qize Yu, Zijun Chen, Chen Wu, Yue Chen, Yiyao Yu, Tingfeng Hui, Bowen Ping, Junchi Yan,

Organizations: Shanghai Jiao Tong University · Tsinghua University · Beijing University of Posts and Telecommunications, Beijing, China · Peking University

Abstract

Reinforcement Learning (RL) has emerged as a key driver for LLM reasoning. This capability is equally pivotal in long-context scenarios---such as long-dialogue understanding and structured data analysis---where the challenge extends beyond consuming tokens to performing rigorous deduction. While existing efforts focus on data synthesis or architectural changes, recent work points out that relying solely on sparse, outcome-only rewards yields limited gains, as such coarse signals are often insufficient to effectively guide the complex long-context reasoning. To address this, we propose LongR, a unified framework that enhances long-context performance by integrating a dynamic ``Think-and-Read'' mechanism---which interleaves reasoning with document consultation---with a contextual density reward based on relative information gain to quantify the utility of the relevant documents, without evidence annotations. Empirically, LongR achieves a gain of over 9% on LongBench v2 and consistent improvements on RULER and InfiniteBench, demonstrating robust efficiency in navigating extensive contexts. Furthermore, LongR consistently enhances performance across diverse RL algorithms (e.g., DAPO, GSPO). Finally, we conduct in-depth analyses to investigate the impact of reasoning chain length on efficiency and the model's robustness against distractors.

Figures & tables

Figure 1: Qualitative comparison between Outcome-Driven CoT (Standard RL, left) and Think-and-Read CoT (LongR, right). Left: Without dense supervision, the baseline model tends to rely on superficial query matching while overlooking detailed context reading. Right: LongR incorporates a contextual dense reward. By incentivizing the model to actively consult the context during the reasoning process, it significantly enhances long-context understanding capabilities.

Figure 2: Overview of the LongR Framework. The system operates within a standard reinforcement learning pipeline enhanced by three key mechanisms: (1) Curriculum Learning: through a progressive curriculum, models learn to dynamically interleave consultation with reasoning purely via RL, eliminating the need for costly, long-context-specific SFT data engineering. (2) Interleaved Think-and-Read Policy: generating actions as a dynamic stream where reasoning steps (blue) alternate with grounded evidence extraction (orange), enabling flexible information seeking without rigid chunking. (3) Dense Utility Supervision: providing additional fine-grained feedback by quantifying the utility of each extracted document via relative information gain.

		Difficulty		Length
Model	Overall	Easy	Hard	Short	Medium	Long
Qwen3-8B*	32.99	40.62	28.28	36.15	29.36	34.95
Qwen3-8B-Base	20.58	19.66	21.14	27.85	17.76	14.06
Qwen3-8B-SFT	26.64	28.26	25.64	31.46	24.45	22.97
Qwen3-8B-SFT w/ DAPO	32.84	39.91	28.48	38.06	28.75	32.29
Qwen3-8B-SFT w/ LongR	36.23	41.15	33.20	38.75	32.67	39.12

Table 1: Evaluation results (%) on LongBench v2. Qwen3-8B* and Qwen3-4B* are officially released models available on HuggingFace. Short, Medium, and Long refer to length ranges of ¡32k, 32-128k, and ¿128k, respectively. Qwen3-8B* is evaluated using the official thinking mode, and we have set the random seed to 0 to ensure reproducibility.

Model	RULER-128k				RULER-64k
Model	NIAH	VT	QA	Avg	NIAH	VT	QA	Avg
Qwen3-8B-Base	51.88	47.68	17.60	39.05	61.76	28.08	39.70	43.18
Qwen3-8B-SFT	61.64	51.96	30.50	48.03	62.96	31.00	38.80	44.25
Qwen3-8B-SFT w/ DAPO	57.64	76.56	30.40	54.87	66.13	58.52	40.40	55.02
Qwen3-8B-SFT w/ LongR	58.63	80.20	45.60	61.48	69.88	86.64	59.42	71.98
Qwen3-4B-Base	46.74	45.64	14.00	35.46	60.55	56.40	19.30	45.42

Table 2: Evaluation results (%) on RULER-128K and RULER-64K. NIAH denotes Needle-In-A-Haystack retrieval, VT denotes Variable Tracking, and QA denotes multi-hop question answering.

Methods	Re.Pa	Re.Nu	En.Sum	En.QA	Zh.QA	En.MC	Avg.
Qwen3-8B-SFT	85.76	86.44	14.18	25.09	14.47	54.15	46.68
Qwen3-8B-SFT w/ DAPO	85.93	83.56	11.86	27.45	23.54	56.33	48.11
Qwen3-8B-SFT w/ LongR	87.46	89.79	15.72	27.40	22.31	58.52	50.20
Qwen3-4B-SFT	86.78	83.43	12.76	12.68	9.65	53.28	43.10
Qwen3-4B-SFT w/ DAPO	87.63	84.75	17.78	14.83	10.39	54.59	44.99
Qwen3-4B-SFT w/ LongR	86.95	87.08	16.07	21.85	17.39	56.33	47.61

Table 3: Evaluation results (%) on InfiniteBench. Re.Pa and Re.Nu denote Retrieve.PassKey and Retrieve.Number, En.Sum denotes English Summarization, En.QA and Zh.QA denote English and Chinese question answering, and En.MC denotes English multiple-choice reasoning.

		Difficulty		Length
Model	Overall	Easy	Hard	Short	Medium	Long
Qwen3-8B-SFT	26.64	28.26	25.64	31.46	24.45	22.97
Reinforcement Learning
Outcome-only Reward	32.80	40.10	28.30	38.33	28.37	32.41
w/ Absolute Density	32.41	41.41	26.85	35.97	33.14	25.00
w/ Threshold Density	34.10	35.16	33.44	41.25	26.74	36.81

Table 4: Ablation study on Reward Design. We compare our proposed method against three baselines: (1) Outcome-only, which relies exclusively on sparse final signals (

r_{acc}

and

r_{fmt}

) without dense supervision; (2) w/ Absolute Density, which substitutes the relative metric with the absolute information gain (Eq. 7 ) as the dense signal; and (3) w/ Threshold Density, which applies a binary reward scheme, assigning a discrete value of 1 only if the relative gain (Eq. 9 ) exceeds 0.5, and 0 otherwise. Evaluation results (%) on LongBench v2.

		Difficulty		Length
Model	Overall	Easy	Hard	Short	Medium	Long
Qwen3-8B-SFT	26.64	28.26	25.64	31.46	24.45	22.97
+LongR w/ 1.7B verifier	34.59	39.06	31.83	43.33	26.98	35.19
+LongR w/ 4B verifier	36.23	41.15	33.20	38.75	32.67	39.12
+LongR w/ 32B verifier	36.83	41.67	33.84	38.75	33.14	40.97
Qwen3-4B-SFT	25.65	21.58	28.16	32.95	22.53	19.68

Table 5: Ablation study on Verifier Size. We evaluate LongR performance using varying verifier sizes (1.7B, 4B, and 32B) from the Qwen3 family. For a controlled comparison, all other components remain fixed.

		Difficulty		Length
Model	Overall	Easy	Hard	Short	Medium	Long
Qwen3-8B-SFT	26.64	28.26	25.64	31.46	24.45	22.97
w/ DAPO	36.23	41.15	33.20	38.75	32.67	39.12
w/ CISPO	34.99	35.94	34.41	34.41	36.57	32.81
w/ GSPO	35.39	33.92	36.29	39.17	35.12	29.63
Qwen3-4B-SFT	25.65	21.58	28.16	32.95	22.53	19.68

Table 6: Ablation study on different RL algorithms. Best results within each backbone are highlighted in bold . Evaluation results (%) on LongBench v2.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Average RL training-step time across verifier configurations. The y-axis “Average Time per Training Step (s/step)” reports the average time per RL training step, in seconds per step. The average is computed over the first 100 training steps under the experimental setup described in Appendix B .

		Difficulty		Length
Model	Overall	Easy	Hard	Short	Medium	Long
Qwen3-8B-SFT	26.64	28.26	25.64	31.46	24.45	22.97
W/ Curriculum Learning
+LongR w/ Stage m = 0	32.41	38.54	28.62	35.56	27.91	36.11
+LongR w/ Stage m = 1	36.23	41.15	33.20	38.75	32.67	39.12
Qwen3-4B-SFT	25.65	21.58	28.16	32.95	22.53	19.68

Appendix

Table 7: Detailed contribution of each stage for curriculum learning. Evaluation results (%) on LongBench v2.

Figure 8: Reasoning chain length comparison for 8B models.

Figure 9: Reasoning chain length comparison for 4B models.

Figure 10: Distractor count for 8B models.

Position	Mean reward
Front (5%)	0.86
Middle (50%)	0.86
Tail (95%)	0.85

Appendix

Table 8: Mean contextual rewards across evidence positions.

Figure 11: Mechanism Decomposition of Relative Information Gain. We visualize the internal calculation of

r_{ctx}

on two distinct segments, which are synthesized from RULER. For the Semantic Distractor (blue), the prior loss

\mathcal{L}(s)

is already low due to the text’s high predictability (common sense), resulting in a limited relative gain (

0.38

) despite context consistency. In contrast, the Target Evidence (red) exhibits a high prior loss (unpredictable UUID) which is drastically reduced by the context, yielding a dominant reward (

0.91

). This example illustrates a relative reward difference; the two-UUID query-switching diagnostic in Appendix E.2 further tests whether reward preference follows the query.

Distractor type	Mean margin	Ranking accuracy
Rare person name	0.25	100.00%
Numerical record	0.32	100.00%
Document-specific fact	0.40	100.00%

Appendix

Table 9: Reward comparison between evidence and three types of irrelevant distractors.

		Difficulty		Length
Model	Overall	Easy	Hard	Short	Medium	Long
R-Search-7B	28.43	30.21	27.33	28.89	27.91	28.70
AutoRefine-7B w/ GRPO	28.23	30.21	27.01	29.44	26.05	30.56
AutoRefine-7B w/ PPO	28.43	31.25	26.69	28.89	27.44	29.63
Qwen3-8B*	32.99	40.62	28.28	36.15	29.36	34.95
Qwen3-8B* w/ RAG	34.59	39.06	31.83	38.53	28.84	39.81

Appendix

Table 10: LongBench v2 results (%). * denotes public post-trained Qwen3 models evaluated with non-thinking Naive RAG.

Model		Qwen3-8B-SFT w/ DAPO †	Qwen3-8B-SFT w/ LongR
	Overall	31.81	36.23
Difficulty	Easy	34.90	41.15
Difficulty	Hard	29.90	33.20
Length	Short	33.89	38.75
	Medium	31.16	32.67
	Long	29.63	39.12

Appendix

Table 11: LongBench v2 results (%) with shared training and inference prompts and identical decoding settings. † denotes the outcome-only control using LongR’s quotation prompt. Best scores are bolded.

		Decoding seed
		0	1	42	Mean $\pm$ SD
	Overall	36.23	37.77	35.79	$36.60\pm 1.04$
Difficulty	Easy	41.15	42.71	40.62	$41.49\pm 1.08$
Difficulty	Hard	33.20	34.73	32.80	$33.57\pm 1.02$
Length	Short	38.75	40.00	38.33	$39.03\pm 0.87$
	Medium	32.67	34.42	32.09	$33.06\pm 1.21$

Appendix

Table 12: LongBench v2 results (%) for Qwen3-8B-SFT w/ LongR across three decoding seeds. SD denotes the sample standard deviation across decoding runs with a fixed checkpoint. SD is expressed in percentage points.

Model		Qwen3-8B-SFT w/ LongR	Qwen3-8B-SFT w/ LongR †
	Overall	36.23	36.98
Difficulty	Easy	41.15	42.19
Difficulty	Hard	33.20	33.76
Length	Short	38.75	39.44
	Medium	32.67	33.49
	Long	39.12	39.81

Appendix

Table 13: LongBench v2 results (%) with and without contextual reward clipping. † denotes lower-bound clipping at

-1

. Best scores are bolded.

Long-Context Reasoning in Language Models

See all 41 papers

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards
Mar 2, 2026Guanzheng Chen, Michael Qizhe Shieh, Lidong BingRL for Language Models Reinforcement Learning with Verifiable Rewards
National University of Singapore · MiroMind AI · absolute AI
LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
Apr 16, 2026Bowen Ping, Zijun Chen, Tingfeng Hui +4Reinforcement Learning RL for Language Model Reasoning
1Peking University · 2Shanghai Jiao Tong University · 3Beijing University of Posts and Telecommunications
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Jun 17, 2026Xiaoyue Xu, Sikui Zhang, Xiaorong Wang +2Training Data Curation RL for Language Model Reasoning
OpenBMB · Tsinghua University

Mar 2, 2026J/K move · Enter open · S save

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

Guanzheng Chen, Michael Qizhe Shieh, Lidong Bing

National University of Singapore · MiroMind AI · absolute AI

RL for Language Models Reinforcement Learning with Verifiable Rewards LLM Grounding Reward Shaping Long-Context Language Model Evaluation

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) by optimizing them against factual outcomes. However, this paradigm falters in long-context scenarios, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contextual grounding--the ability to find and reason over externally provided information. We identify a key reason for this failure: a reward based solely on the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, rendering learning intractable. To overcome this bottleneck, we introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the correct grounding information, providing a robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long-context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process is a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is available at https://github.com/real-absolute-AI/LongRLVR.

Abstract

Long-Context Reasoning in Language Models

Explore similar work

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning