Organizations: Shanghai Jiao Tong University · Tsinghua University · Beijing University of Posts and Telecommunications, Beijing, China · Peking University
Reinforcement Learning (RL) has emerged as a key driver for LLM reasoning. This capability is equally pivotal in long-context scenarios---such as long-dialogue understanding and structured data analysis---where the challenge extends beyond consuming tokens to performing rigorous deduction. While existing efforts focus on data synthesis or architectural changes, recent work points out that relying solely on sparse, outcome-only rewards yields limited gains, as such coarse signals are often insufficient to effectively guide the complex long-context reasoning. To address this, we propose LongR, a unified framework that enhances long-context performance by integrating a dynamic ``Think-and-Read'' mechanism---which interleaves reasoning with document consultation---with a contextual density reward based on relative information gain to quantify the utility of the relevant documents, without evidence annotations. Empirically, LongR achieves a gain of over 9% on LongBench v2 and consistent improvements on RULER and InfiniteBench, demonstrating robust efficiency in navigating extensive contexts. Furthermore, LongR consistently enhances performance across diverse RL algorithms (e.g., DAPO, GSPO). Finally, we conduct in-depth analyses to investigate the impact of reasoning chain length on efficiency and the model's robustness against distractors.
Figures & tables
Figure 1: Qualitative comparison between Outcome-Driven CoT (Standard RL, left) and Think-and-Read CoT (LongR, right). Left: Without dense supervision, the baseline model tends to rely on superficial query matching while overlooking detailed context reading. Right: LongR incorporates a contextual dense reward. By incentivizing the model to actively consult the context during the reasoning process, it significantly enhances long-context understanding capabilities.
Figure 2: Overview of the LongR Framework. The system operates within a standard reinforcement learning pipeline enhanced by three key mechanisms: (1) Curriculum Learning: through a progressive curriculum, models learn to dynamically interleave consultation with reasoning purely via RL, eliminating the need for costly, long-context-specific SFT data engineering. (2) Interleaved Think-and-Read Policy: generating actions as a dynamic stream where reasoning steps (blue) alternate with grounded evidence extraction (orange), enabling flexible information seeking without rigid chunking. (3) Dense Utility Supervision: providing additional fine-grained feedback by quantifying the utility of each extracted document via relative information gain.
Difficulty
Length
Model
Overall
Easy
Hard
Short
Medium
Long
Qwen3-8B*
32.99
40.62
28.28
36.15
29.36
34.95
Qwen3-8B-Base
20.58
19.66
21.14
27.85
17.76
14.06
Qwen3-8B-SFT
26.64
28.26
25.64
31.46
24.45
22.97
Qwen3-8B-SFT w/ DAPO
32.84
39.91
28.48
38.06
28.75
32.29
Qwen3-8B-SFT w/ LongR
36.23
41.15
33.20
38.75
32.67
39.12
Table 1: Evaluation results (%) on LongBench v2. Qwen3-8B* and Qwen3-4B* are officially released models available on HuggingFace. Short, Medium, and Long refer to length ranges of ¡32k, 32-128k, and ¿128k, respectively. Qwen3-8B* is evaluated using the official thinking mode, and we have set the random seed to 0 to ensure reproducibility.
Model
RULER-128k
RULER-64k
NIAH
VT
QA
Avg
NIAH
VT
QA
Avg
Qwen3-8B-Base
51.88
47.68
17.60
39.05
61.76
28.08
39.70
43.18
Qwen3-8B-SFT
61.64
51.96
30.50
48.03
62.96
31.00
38.80
44.25
Qwen3-8B-SFT w/ DAPO
57.64
76.56
30.40
54.87
66.13
58.52
40.40
55.02
Qwen3-8B-SFT w/ LongR
58.63
80.20
45.60
61.48
69.88
86.64
59.42
71.98
Qwen3-4B-Base
46.74
45.64
14.00
35.46
60.55
56.40
19.30
45.42
Table 2: Evaluation results (%) on RULER-128K and RULER-64K. NIAH denotes Needle-In-A-Haystack retrieval, VT denotes Variable Tracking, and QA denotes multi-hop question answering.
Methods
Re.Pa
Re.Nu
En.Sum
En.QA
Zh.QA
En.MC
Avg.
Qwen3-8B-SFT
85.76
86.44
14.18
25.09
14.47
54.15
46.68
Qwen3-8B-SFT w/ DAPO
85.93
83.56
11.86
27.45
23.54
56.33
48.11
Qwen3-8B-SFT w/ LongR
87.46
89.79
15.72
27.40
22.31
58.52
50.20
Qwen3-4B-SFT
86.78
83.43
12.76
12.68
9.65
53.28
43.10
Qwen3-4B-SFT w/ DAPO
87.63
84.75
17.78
14.83
10.39
54.59
44.99
Qwen3-4B-SFT w/ LongR
86.95
87.08
16.07
21.85
17.39
56.33
47.61
Table 3: Evaluation results (%) on InfiniteBench. Re.Pa and Re.Nu denote Retrieve.PassKey and Retrieve.Number, En.Sum denotes English Summarization, En.QA and Zh.QA denote English and Chinese question answering, and En.MC denotes English multiple-choice reasoning.
Difficulty
Length
Model
Overall
Easy
Hard
Short
Medium
Long
Qwen3-8B-SFT
26.64
28.26
25.64
31.46
24.45
22.97
Reinforcement Learning
Outcome-only Reward
32.80
40.10
28.30
38.33
28.37
32.41
w/ Absolute Density
32.41
41.41
26.85
35.97
33.14
25.00
w/ Threshold Density
34.10
35.16
33.44
41.25
26.74
36.81
Table 4: Ablation study on Reward Design. We compare our proposed method against three baselines: (1) Outcome-only, which relies exclusively on sparse final signals ( racc and rfmt ) without dense supervision; (2) w/ Absolute Density, which substitutes the relative metric with the absolute information gain (Eq. 7 ) as the dense signal; and (3) w/ Threshold Density, which applies a binary reward scheme, assigning a discrete value of 1 only if the relative gain (Eq. 9 ) exceeds 0.5, and 0 otherwise. Evaluation results (%) on LongBench v2.
Difficulty
Length
Model
Overall
Easy
Hard
Short
Medium
Long
Qwen3-8B-SFT
26.64
28.26
25.64
31.46
24.45
22.97
+LongR w/ 1.7B verifier
34.59
39.06
31.83
43.33
26.98
35.19
+LongR w/ 4B verifier
36.23
41.15
33.20
38.75
32.67
39.12
+LongR w/ 32B verifier
36.83
41.67
33.84
38.75
33.14
40.97
Qwen3-4B-SFT
25.65
21.58
28.16
32.95
22.53
19.68
Table 5: Ablation study on Verifier Size. We evaluate LongR performance using varying verifier sizes (1.7B, 4B, and 32B) from the Qwen3 family. For a controlled comparison, all other components remain fixed.
Difficulty
Length
Model
Overall
Easy
Hard
Short
Medium
Long
Qwen3-8B-SFT
26.64
28.26
25.64
31.46
24.45
22.97
w/ DAPO
36.23
41.15
33.20
38.75
32.67
39.12
w/ CISPO
34.99
35.94
34.41
34.41
36.57
32.81
w/ GSPO
35.39
33.92
36.29
39.17
35.12
29.63
Qwen3-4B-SFT
25.65
21.58
28.16
32.95
22.53
19.68
Table 6: Ablation study on different RL algorithms. Best results within each backbone are highlighted in bold . Evaluation results (%) on LongBench v2.
Figure 9
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Average RL training-step time across verifier configurations. The y-axis “Average Time per Training Step (s/step)” reports the average time per RL training step, in seconds per step. The average is computed over the first 100 training steps under the experimental setup described in Appendix B .
Difficulty
Length
Model
Overall
Easy
Hard
Short
Medium
Long
Qwen3-8B-SFT
26.64
28.26
25.64
31.46
24.45
22.97
W/ Curriculum Learning
+LongR w/ Stage m = 0
32.41
38.54
28.62
35.56
27.91
36.11
+LongR w/ Stage m = 1
36.23
41.15
33.20
38.75
32.67
39.12
Qwen3-4B-SFT
25.65
21.58
28.16
32.95
22.53
19.68
Appendix
Table 7: Detailed contribution of each stage for curriculum learning. Evaluation results (%) on LongBench v2.
Figure 12
Figure 8: Reasoning chain length comparison for 8B models.
Figure 9: Reasoning chain length comparison for 4B models.
Figure 10: Distractor count for 8B models.
Position
Mean reward
Front (5%)
0.86
Middle (50%)
0.86
Tail (95%)
0.85
Appendix
Table 8: Mean contextual rewards across evidence positions.
Figure 11: Mechanism Decomposition of Relative Information Gain. We visualize the internal calculation of rctx on two distinct segments, which are synthesized from RULER. For the Semantic Distractor (blue), the prior loss L(s) is already low due to the text’s high predictability (common sense), resulting in a limited relative gain ( 0.38 ) despite context consistency. In contrast, the Target Evidence (red) exhibits a high prior loss (unpredictable UUID) which is drastically reduced by the context, yielding a dominant reward ( 0.91 ). This example illustrates a relative reward difference; the two-UUID query-switching diagnostic in Appendix E.2 further tests whether reward preference follows the query.
Distractor type
Mean margin
Ranking accuracy
Rare person name
0.25
100.00%
Numerical record
0.32
100.00%
Document-specific fact
0.40
100.00%
Appendix
Table 9: Reward comparison between evidence and three types of irrelevant distractors.
Difficulty
Length
Model
Overall
Easy
Hard
Short
Medium
Long
R-Search-7B
28.43
30.21
27.33
28.89
27.91
28.70
AutoRefine-7B w/ GRPO
28.23
30.21
27.01
29.44
26.05
30.56
AutoRefine-7B w/ PPO
28.43
31.25
26.69
28.89
27.44
29.63
Qwen3-8B*
32.99
40.62
28.28
36.15
29.36
34.95
Qwen3-8B* w/ RAG
34.59
39.06
31.83
38.53
28.84
39.81
Appendix
Table 10: LongBench v2 results (%). * denotes public post-trained Qwen3 models evaluated with non-thinking Naive RAG.
Model
Qwen3-8B-SFT w/ DAPO †
Qwen3-8B-SFT w/ LongR
Overall
31.81
36.23
Difficulty
Easy
34.90
41.15
Hard
29.90
33.20
Length
Short
33.89
38.75
Medium
31.16
32.67
Long
29.63
39.12
Appendix
Table 11: LongBench v2 results (%) with shared training and inference prompts and identical decoding settings. † denotes the outcome-only control using LongR’s quotation prompt. Best scores are bolded.
Decoding seed
0
1
42
Mean ± SD
Overall
36.23
37.77
35.79
36.60±1.04
Difficulty
Easy
41.15
42.71
40.62
41.49±1.08
Hard
33.20
34.73
32.80
33.57±1.02
Length
Short
38.75
40.00
38.33
39.03±0.87
Medium
32.67
34.42
32.09
33.06±1.21
Appendix
Table 12: LongBench v2 results (%) for Qwen3-8B-SFT w/ LongR across three decoding seeds. SD denotes the sample standard deviation across decoding runs with a fixed checkpoint. SD is expressed in percentage points.
Model
Qwen3-8B-SFT w/ LongR
Qwen3-8B-SFT w/ LongR †
Overall
36.23
36.98
Difficulty
Easy
41.15
42.19
Hard
33.20
33.76
Length
Short
38.75
39.44
Medium
32.67
33.49
Long
39.12
39.81
Appendix
Table 13: LongBench v2 results (%) with and without contextual reward clipping. † denotes lower-bound clipping at −1 . Best scores are bolded.
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) by optimizing them against factual outcomes. However, this paradigm falters in long-context scenarios, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contextual grounding--the ability to find and reason over externally provided information. We identify a key reason for this failure: a reward based solely on the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, rendering learning intractable. To overcome this bottleneck, we introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the correct grounding information, providing a robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long-context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process is a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is available at https://github.com/real-absolute-AI/LongRLVR.
Guanzheng Chen, Michael Qizhe Shieh, Lidong Bing
National University of Singapore · MiroMind AI · absolute AI
Reinforcement Learning (RL) has emerged as a critical driver for enhancing the reasoning capabilities of Large Language Models (LLMs). While recent advancements have focused on reward engineering or data synthesis, few studies exploit the model's intrinsic representation characteristics to guide the training process. In this paper, we first observe the presence of high-magnitude activations within the query and key vectors when processing long contexts. Drawing inspiration from model quantization -- which establishes the criticality of such high-magnitude activations -- and the insight that long-context reasoning inherently exhibits a sparse structure, we hypothesize that these weights serve as the pivotal drivers for effective model optimization. Based on this insight, we propose LongAct, a strategy that shifts from uniform to saliency-guided sparse updates. By selectively updating only the weights associated with these significant activations, LongAct achieves an approximate 8% improvement on LongBench v2 and enhances generalization on the RULER benchmark. Furthermore, our method exhibits remarkable universality, consistently boosting performance across diverse RL algorithms such as GRPO and DAPO. Extensive ablation studies suggest that focusing on these salient features is key to unlocking long-context potential.
Bowen Ping, Zijun Chen, Tingfeng Hui +4
1Peking University · 2Shanghai Jiao Tong University · 3Beijing University of Posts and Telecommunications
Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.