Understanding and reasoning over long contexts is a crucial capability for language models (LMs). Although recent models support increasingly long context windows, their accuracy often deteriorates as input length grows. In practice, models often struggle to keep attention aligned with the most relevant context throughout decoding. In this work, we propose DYSCO, a novel decoding algorithm for improving long-context reasoning. DYSCO leverages retrieval heads--a subset of attention heads specialized for longcontext retrieval--to identify task-relevant tokens at each decoding step and explicitly up-weight them. By doing so, DYSCO dynamically adjusts attention during generation to better utilize relevant context. The method is training-free and can be applied directly to any off-the-shelf LMs. Across multiple instruction-tuned and reasoning models, DYSCO consistently improves performance on challenging long-context reasoning benchmarks, yielding relative gains of up to 25% on MRCR and LongBenchV2 at 128K context length with modest additional compute. Further analysis highlights the importance of both dynamic attention rescaling and retrievalhead guided selection for the effectiveness of the method, while providing interpretability insights into decoding-time attention behavior. Our code is available at https://github.com/princeton-pli/DySCO.
Figures & tables
Figure 1: Top: An illustrative Path Traversal task (simplified). Solving the task requires dynamically locating relevant context during decoding. Bottom: Accuracy as a function of context length for Qwen3-8B with and without DySCO . Despite the total context being only 16K tokens, both models exhibit severe performance degradation as context length increases.
Figure 2: Overview of DySCO algorithm. At each decoding step, DySCO consists of three stages: (1) Aggregation : We run a partial forward pass over the input sequence to obtain attentions of retrieval heads, such as QRHead , and use them to assign relevance scores to context tokens; (2) Selection : We use the relevance scores to select the important tokens; (3) Rescaling : We up-weight the important tokens by intervening attention logits of all attention heads and run a full forward pass to sample the next token.
Figure 3: Left : Performance of Qwen3-8B on Path Traversal as context length increases. Middle : Fractions that the gold edge appear among the top-5% edges ranked by attention score (sum of attention over all tokens in the span) from QRHead versus random heads. Right : Attention mass assigned to gold edges by QRHead and random heads. Despite severe performance degradation and a reduction in attention mass on gold edges, QRHead consistently allocates substantially higher attention to the gold edges.
Path Traversal
4K
8K
16K
32K
Qwen3-32B
80
67
32
21
+ UniAttnS
82
68
40
19
+ DySCO
79
71
46
33
Qwen3-8B
57
45
19
1
+ UniAttnS
59
46
25
2
+ DySCO
65
51
33
2
Table 1: Performance across different context lengths with Uniform Attention Scaling ( UniAttnS ) and DySCO .
Figure 4: Performance on MRCR , LongBenchV2 , and Clipper . DySCO substantially outperforms vanilla decoding, and UniAttnS . YaRN is applied to Qwen models at 128K context length, but not to Llama-3.1-8B-Instruct, which natively supports 128K.
MRCR
Hotpot
InfMC
InfQA
Avg
128K
128K
128K
128K
128K
Qwen3-4B YaRN
16.7
37
50
27.1
32.7
+ UniAttnS
18.4
34
39
24.1
28.9
+ DySCO
19.0
39
54
31.3
35.8
Qwen3-8B YaRN
16.7
61
61
39.6
44.6
+ UniAttnS
17.5
58
63
39.7
44.6
Table 2: Performance on long-context recall tasks with 128K input length. We apply YaRN for Qwen models.
MRCR
LongV2
Path
InfMC
64K
64K
16K
128K
Think
Yes
No
Yes
No
No
Vanilla
24.8
24.8
43.4
19
61
DySCO
27.3
26.3
51.2
33
66
w/ RandomHead
26.6
24.2
47.0
20
63
w/ StaticScaling
22.5
23.1
47.5
23
64
Table 3: Ablations of DySCO on head selection and dynamic rescaling with Qwen3-8B. DySCO outperforms its ablations with random attention heads and with static scaling.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
EAGER (Implementation)
Flash (Theoretical)
Vanilla
6.1
17.8
DySCO
3.6
10.5
Appendix
Table 4: Throughput (tokens/s) comparison between vanilla decoding and DySCO , under standard attention (EAGER) and an estimated Flash Attention implementation. Results are obtained on Qwen3-8B with 100K input tokens and 4K output tokens on an H100 GPU.
Figure 5: Comparison between DySCO , RAG (Stella), LongLLMLingua, and vanilla decoding. For RAG and LongLLMLingua, we report the results after reducing the context to different length (4K, 8K, and 16K tokens).
Figure 6: Performance of Qwen3-8B on MRCR and LongBenchV2 at 64K and 128K under four decoding settings: direct answer vs. Think, with vanilla decoding or DySCO . DySCO yields larger gains when combined with CoT.
p
K
β
MRCR
LongBenchV2
Vanilla
–
–
–
16.1
28.6
Default
0.975
8192
2.5
19.6
34.7
Vary p
0.95
8192
2.5
18.4
31.7
Varying K
0.975
2048
2.5
19.6
36.8
Appendix
Table 5: Evaluation of hyperparameter robustness of DySCO with Qwen3-8B. DySCO is robust to modest variations in the rescale strength β , selection constraint K , and selection threshold p .
Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long contexts using gradient updates at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard long-context finetuning, achieving average absolute performance gains of up to 20% for Qwen-2.5 (0.5B & 7B) on synthetic and real-world long-context reasoning. PERK also maintains its advantages across model scales and families. Compared to specialized long-context LLMs, PERK matches or surpasses their performance. Finally, our analyses show PERK is more robust to reasoning complexity, length extrapolation, and the positions of relevant information in contexts. https://perk-long-context.web.app
While Large Language Models (LLMs) advertise million-token context windows, reasoning quality often collapses as inputs grow -- a phenomenon termed context rot. This failure stems from a structural entanglement in monolithic architectures, where the massive search burden of contextual grounding exhausts the representational capacity needed for complex reasoning. To resolve this, we propose Grounding-Reasoning Disaggregation via DIStributed long COntext scaling (DISCO). Inspired by distributed computing frameworks like Apache Spark, DISCO partitions long context across a fleet of Worker LLMs dedicated exclusively to parallel, localized grounding. A central Driver LLM, trained via Reinforcement Learning (GRPO) to optimize planning, orchestrates execution by dynamically mapping queries into atomic extraction tasks and reducing the gathered evidence to synthesize a final answer. By isolating reasoning from raw context noise, DISCO effectively eliminates context rot. On RULER-QA (1M tokens), it maintains 78.4% accuracy where standard baselines collapse. Furthermore, it outperforms full-context models by up to 9.8 points on LongBench v2 and matches frontier models like Gemini-3-Pro-Preview while reducing inference costs by over 80%, establishing a highly efficient paradigm for robust long-context inference.
Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications. Although recent LLMs support increasingly long context windows, they often fail to use relevant evidence that is already present in the input, revealing a gap between context access and effective context utilization. In this work, we propose Recursive Evidence Replay as LLM Harness for Long-Context Reasoning (RECONTEXT), a training-free inference method for improving long-context reasoning. RECONTEXT uses model-internal relevance signals to construct a query-conditioned evidence pool and replays it before final generation while preserving the full original context. This recursive selection process separates evidence organization from answer generation without training, external memory, or context pruning. We also provide a theoretical analysis based on associative memory, which characterizes the context as a memory store, the question as a retrieval cue, attention as cue-trace association, and replay as trace reactivation. Experiments on eight long-context datasets with 128K context length show that RECONTEXT consistently improves evidence utilization across Qwen3-4B, Qwen3-8B, and Llama3-8B, achieving the best average rank on all three backbones. Code is available at https://github.com/Yanjun-Zhao/ReContext.