As LLM capabilities advance, agents are tackling increasingly complex tasks over longer horizons. Their growing interaction histories make memory compaction essential for staying within context windows and reducing prefill cost. Existing methods summarize the history or compress its KV cache, often adding model computation to preserve information for future requests. A new user request can change which history matters, but reassessing that history with the model requires re-encoding it if the KV cache has expired. Past attention provides signals of historical importance and dependencies between messages, while relevance to the current task must be assessed using the new user request. We introduce ReCAP, a memory compaction method that stores attention-derived importance scores and dependency links in a lightweight, persistent context graph. For each new request, ReCAP combines stored importance with relevance cues from the request and follows dependency links to select messages and their supporting context, without additional model calls for selection. Compared with Codex's default summarization-based compaction, ReCAP reduces estimated latency for compaction and cold restoration by approximately 95% on both Qwen3-Coder and gpt-oss. It also roughly halves the historical context per call on SWE-Together at comparable task quality and improves accuracy on the code tasks of Lost-in-Conversation over full history by 19.8 and 41.2 points.
Figures & tables
Figure 1: ReCAP preserves attention-derived importance and dependency links in a persistent context graph. For each new request, it combines stored importance with the request’s relevance cues to select historical blocks and their supporting context, retaining protected contexts.
Figure 2: Attention signals for memory compaction. Left: Attention over history blocks per turn (width: tokens; dashed: user turns; blue: attention density, cube-root scale). Coral: blocks selected under a 10% token budget, concatenated below. Middle: Next-turn identifier coverage versus retained-history budget; Attn. + Query adds query-identifier overlap. Right: Attention redistribution (TV distance) over 32 fixed histories for paraphrases versus changed targets; ticks mark tasks.
Figure 3: ReCAP workflow. 1–2: Sampled attention supplies node importance and support-to-dependent edges. 3: Selection combines stored importance with the query’s identifier overlap, keeps protected records, and adds one-hop predecessors (diagnosis m5 brings in search record m2 ), without a model call. 4–5: Execution produces new messages and attention that update the graph.
Method
Qwen3-Coder-30B-A3B
gpt-oss-20b
Score ↑
Δ
Ctx. ↓
+Pre. ↓
+Dec. ↓
+Lat. ↓
+Mem. ↓
Score ↑
Δ
Ctx. ↓
+Pre. ↓
+Dec. ↓
+Lat. ↓
+Mem. ↓
SWE-Together
No compr.
Full
24.4 ± 1.87
↑ 4.1
28,504
0
0
0
0
8.2 ± 0.15
↑ 0.8
8,508
0
0
0
0
Heuristic
Recency
26.6 ± 0.40
↑ 1.9
10,154
0
0
0
0
8.4 ± 0.08
↑ 0.6
1,264
0
0
0
0
Random
23.8 ± 0.99
↑ 4.7
12,712
0
0
0
0
7.4 ± 0.35
↑ 1.6
1,011
0
0
0
0
Prompt compr.
LLMLingua-2
23.9 ± 1.17
↑ 4.6
11,062
0
0
7.39
2,236
7.8 ± 0.22
↑ 1.2
3,340
0
0
0.43
2,236
Table 1: Main results on SWE-Together (reward) and Lost-in-Conversation (LiC, accuracy), scaled by 100; ± : standard deviation across runs. Δ is ReCAP ’s score minus the row’s ( orange : ReCAP higher; teal : lower). Ctx. : mean tokens per call, excluding fixed system prompts. Compaction adds, per invocation, +Pre. prompt and +Dec. completion tokens on the backbone (means) and +Lat. seconds (one H100; ReCAP on CPU); +Mem. : memory it keeps in MB (GPU for compressors and KV cache, CPU for the context graph). Bold / underline : best/second-best scores; Appendix C gives the accounting.
Figure 4: Estimated cold-restoration latency on SWE-Together. Bars combine TTFT and compaction overhead; arrows show reductions with ReCAP . The two panels use separate scales.
Variant
Acc. ↑
Δ
Ctx. ↓
ReCAP
85.4 ± 1.95
—
94
w/o attention graph
76.6 ± 1.95
↑ 8.8
95
paragraph blocks
83.2 ± 1.92
↑ 2.2
152
line blocks
83.0 ± 2.45
↑ 2.4
168
Table 2: Component ablations on Lost-in-Conversation with Qwen3-Coder-30B-A3B ( β=0.1 ). Δ is ReCAP ’s accuracy minus the variant’s. Ctx. is the mean context size in tokens, excluding the fixed system prompt, as in Table 1 .
Figure 5: ReCAP with prefix caching (gpt-oss-20b, SWE-Together). (a) Idle boundaries prefill the compact context Sn ; warm ones reuse the cache; the graph updates in the background. (b) Prompt tokens per request, cached (hatched) and uncached (solid), with hit rates. (c) Engine TTFT at warm and idle boundaries. (d) Task reward versus warm probability; hollow point: all idle (Table 1 ).
LiC
SWE
Attention
gpt-oss
Qwen
Qwen
agent’s own
67.6 ± 3.58
85.4 ± 1.95
28.5 ± 0.40
Qwen3-0.6B proxy
68.2 ± 3.49
85.2 ± 2.17
29.0 ± 0.64
none
63.4 ± 2.61
76.6 ± 1.95
25.4 ± 0.22
Table 3: ReCAP with attention from the agent or a Qwen3-0.6B proxy ( β=0.1 ). Scores are scaled by 100; ± : standard deviation across runs.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Additional historical-context observations. Left: Aggregate attention concentration under a retained-history budget; the diagonal denotes uniform attention per unit of history. Middle: The full comparison of selection signals, including query identifiers and query embeddings. Right: User-instruction and latest-edit records omitted by attention-ranked selection, together with the fraction of transitions losing any protected record. The selector receives enough budget to fit the complete protected set.
Figure 7: Query effects across request templates. Paraphrases of the same target and requests about different targets are compared on the same 32 histories. Longer framed requests explicitly ask the agent to recall earlier facts and constraints. Both template families show greater attention redistribution under a target change. Curves share a smoothing bandwidth; bottom ticks identify the paired task-level observations, and the dashed line marks zero redistribution.
Figure 8: History retained by ReCAP on SWE-Together with Qwen3-Coder. (a) Retained blocks in each tenth of the session, relative to a uniform spread; counts include protected records. (b) Origin of the blocks retained at each user turn: kept from the previous working context, revived after being omitted from it, or created since.
Benchmark
Backbone
Blocks (median)
Median (ms)
P90 (ms)
Max (ms)
SWE-Together
Qwen3-Coder
132
20.2
45.2
112.6
SWE-Together
gpt-oss
18
1.4
4.4
20.3
LiC
Qwen3-Coder
6
0.30
0.80
1.69
LiC
gpt-oss
6
0.26
0.65
1.51
Appendix
Table 4: Per-request selection time of ReCAP on one CPU core.
Symbol
Meaning
Value
β
nominal history-character retention fraction
0.10
λ
weight of the lexical channel in Eq. 3
0.5
α
importance EMA rate (§ 4.2 )
0.6
ϵ
exploration bonus scale for unobserved blocks
0.1
κ
stale penalty (in standardized-score units)
2.0
δ
allowance for one-hop dependency expansion
0.1
Appendix
Table 5: ReCAP hyperparameters. Selection and update thresholds are shared across tasks and replicates; attention layers depend on the model architecture.
Long-horizon LLM agents accumulate reasoning traces, actions, and tool observations that can eventually exceed a model's fixed context window. Existing compaction methods address this limitation by discarding, summarizing, or retrieving earlier information, but they may remove task-critical details or fail to recover them reliably. We propose ARC (Addressable Recall Compaction), a context-management framework that separates archival storage from active-context presentation. ARC stores tool observations in an append-only, ID-addressable log and replaces older observations with compact citations when compaction is required. The agent can subsequently use these identifiers to request stored content without re-executing the corresponding tools or depending solely on similarity-based retrieval. We evaluate ARC using Qwen3-8B with a 16k context window and Qwen3-32B with a 32k context window. On the Needle-in-a-Haystack evaluation, ARC achieves an average exact-answer accuracy of 99.40%, compared with 88.12% for the best-performing baseline in our evaluation. ARC also reduces estimated serving time and HBM traffic under our hardware-cost model. On the LongBench-v2 Hard subset, ARC obtains an average accuracy of 29.97%, compared with 28.25% for the best-performing baseline. These results indicate that explicit, address-based recall can improve information retention and serving efficiency relative to the evaluated context-management baselines under the tested settings.
Thang Dang, Yuma Ichikawa, Sakina Fatima +1
Fujitsu Research of America · Fujitsu Limited Japan · RIKEN Center for AIP
Long-horizon LLM agents accumulate growing conversation histories that eventually exceed the model's context window. Context compaction via LLM-based summarization keeps the conversation bounded, but summarization is inherently lossy and the blocking call stalls agent inference for tens of seconds. Moreover, the operator has no fine-grained control over summary volume since prompt instructions are largely ignored, and as context grows, both the amount of output tokens the model produces and the information it retains fluctuate substantially from run to run, making the agent's retained knowledge unpredictable across runs. We introduce parallel compaction for long-horizon agentic flows and characterize it against the sequential synchronous baseline across four backbones spanning 8B to 120B parameters, mixing dense and MoE architectures with reasoning and non-reasoning models, on the HotpotQA multi-hop QA and LoCoMo long-context dialogue benchmarks. Parallel compaction gives the operator fine-grained, predictable control over summary volume and enables more targeted prompt engineering per block. At matched compaction decode volume, it reduces end-to-end wall time and improves compaction throughput over the sequential baseline.
Long-term memory enables LLM agents to leverage past interactions, but dialogue histories quickly exceed the context window, forcing agents to retrieve relevant subsets at query time. Because useful evidence is sparse and scattered across verbose conversations, retrieval faces a fundamental tension: broadening recall improves coverage but floods downstream reasoning with noise, while compressing memories at write time eases retrieval but irreversibly discards details that future queries may need. We introduce LazyMem, which resolves this tension by deferring all memory construction to query time. Given a retrieved candidate pool, a lightweight model processes it in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained with supervised fine-tuning followed by reinforcement learning, using a reward that jointly encourages the identification of relevant messages and the generation of compressions that are faithful to the source and useful for answering the query. On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline while using only 213 answer-context memory tokens, 21.0 times fewer than the baseline. It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline. Code is available at https://github.com/allacnobug/LazyMem.
Jing Yu, Yibo Zhao, Jiaming Zhang +1
School of Data Science and Engineering, East China Normal University