In this paper, we propose active recap learning (ARL), a framework for enhancing large language model (LLM) in understanding long contexts. ARL enables models to revisit and summarize earlier content through targeted sequence construction during continued pretraining and retrospective summarization at inference. First, we identify key tokens in prepared long context based on loss gaps between long and short forward contexts and find the most relevant preceding paragraphs, then summarize them using an LLM. Second, ARL equips models with the ability to autonomously generate and utilize these retrospective summaries during inference, thereby establishing a recursive memory mechanism across paragraphs. Experimental results show substantial gains, with ARL achieving an improvement of up to 129.7% on RULER and 11--13% on LongBench. Overall, ARL offers a simple yet effective continued pretraining-based approach to strengthen long-context understanding, advancing scalable memory augmentation in LLM. Code is available at CODE
Figures & tables
Figure 1: Overview framework of our proposed ARL with a chunk generating recap example.
Figure 2: Full process of inference-time recap generation and accumulation
Figure 3: Performance comparison of our proposed recap with origin and scaling on RULER bench.
Model (Setting) / Length
Multikey
Query
Value
Single
CWE
FWE
Hotpot
SQuAD
VT
1
2
3
1
2
3
RWKV7 1.5B (recap, 8K)
37.2
4.0
2.0
16.5
32.9
100.0
65.8
38.8
11.1
46.1
15.6
19.5
46.1
RWKV7 1.5B (recap, 16K)
21.8
0.0
0.0
8.1
16.7
100.0
20.8
19.6
8.3
36.4
13.2
15.1
40.9
RWKV7 1.5B (recap, 32K)
5.8
0.0
0.0
2.6
6.7
23.2
13.0
8.2
3.3
32.2
10.8
13.6
12.1
RWKV7 1.5B (origin, 8K)
10.4
1.0
0.4
2.5
1.8
100.0
17.2
23.4
0.1
38.1
20.0
19.1
43.9
RWKV7 1.5B (origin, 16K)
8.4
0.0
0.0
2.7
2.9
100.0
10.4
8.8
0.1
41.9
19.8
17.4
40.0
Table 1: Performance comparison of different models and settings on RULER bench.
Task
Origin
Recap
Improvement
GovReport
14.28
18.20
↑ 27.45%
QMSum
13.37
15.91
↑ 19.00%
MultiNews
10.13
14.91
↑ 47.19%
VCSUM
7.27
9.04
↑ 24.35%
Table 2: Evaluation of several summarization tasks in LongBench on Qwen1.5 1.8B.
Context Length
Sample Size
10k
20k
40k
8k
65.4
71.2
72.9
16k
49.4
52.3
56.9
32k
23.2
25.1
26.8
Table 3: Comparison results of different continued pretraining sample sizes and evaluation context lengths on RULER .
Chunks
3
5
6
7
9
10
Score
13.4
15.3
16.0
16.2
17.5
19.0
Table 4: Comparison score of the number of different chunks on NIAH single2 task.
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.
Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications. Although recent LLMs support increasingly long context windows, they often fail to use relevant evidence that is already present in the input, revealing a gap between context access and effective context utilization. In this work, we propose Recursive Evidence Replay as LLM Harness for Long-Context Reasoning (RECONTEXT), a training-free inference method for improving long-context reasoning. RECONTEXT uses model-internal relevance signals to construct a query-conditioned evidence pool and replays it before final generation while preserving the full original context. This recursive selection process separates evidence organization from answer generation without training, external memory, or context pruning. We also provide a theoretical analysis based on associative memory, which characterizes the context as a memory store, the question as a retrieval cue, attention as cue-trace association, and replay as trace reactivation. Experiments on eight long-context datasets with 128K context length show that RECONTEXT consistently improves evidence utilization across Qwen3-4B, Qwen3-8B, and Llama3-8B, achieving the best average rank on all three backbones. Code is available at https://github.com/Yanjun-Zhao/ReContext.
We propose RecaLLM, a set of reasoning language models post-trained to make effective use of long-context information. In-context retrieval, which identifies relevant evidence from context, and reasoning are deeply intertwined: retrieval supports reasoning, while reasoning often determines what must be retrieved. However, their interaction remains largely underexplored. In preliminary experiments on several open-source LLMs, we observe that in-context retrieval performance substantially degrades even after a short reasoning span, revealing a key bottleneck for test-time scaling that we refer to as lost-in-thought: reasoning steps that improve performance also make subsequent in-context retrieval more challenging. To address this limitation, RecaLLM interleaves reasoning with explicit in-context retrieval, alternating between reasoning and retrieving context information needed to solve intermediate subproblems. We introduce a negligible-overhead constrained decoding mechanism that enables verbatim copying of evidence spans, improving the grounding of subsequent generation. Trained on diverse lexical and semantic retrieval tasks, RecaLLM achieves strong performance on two long-context benchmarks, RULER and HELMET, significantly outperforming baselines. Notably, we observe consistent gains at context windows of up to 128K tokens using training samples of at most 10K tokens, far shorter than those used by existing long-context approaches, highlighting a promising path toward improving long-context performance without expensive long-context training data.
Large language models (LLMs) process long contexts, including long documents and lengthy conversations, but face token-level memory usage that increases proportionally to input length. Although model optimization and lossy prompt compression are widely used, these methods still fail to solve the long-context recall problem beyond pretrained and size-constrained context windows. This paper proposes a long-context recall method that maintains near-constant GPU memory usage as context length increases, without additional training. The main idea is to reconstruct facts using parameter activations in the LLM's feed-forward layers, which store residual vectors representing facts from the source document. Utilizing residual vectors allows the LLM to deterministically reconstruct query relevant facts without referencing the original document, preserving high fidelity and reducing memory usage without fine-tuning weights. Experimental results show that the proposed method enables answering single-fact questions in two-million-token story contexts where previous methods fail.