Long-horizon LLM agents often exceed their context windows, forcing them to discard information that may be needed later. Existing context management strategies rely on summarization, masking, or structured memory, requiring decisions about what to preserve or how to retrieve it before future needs are known. However, these methods can introduce lossy compression, potentially discarding details needed for future tasks. To address this limitation, we introduce GrepMem, a simple, lossless, and index-free memory mechanism built on two components: an append-only plain-text log of the agent's complete history and a regular-expression search tool for on-demand recall. Unlike structured memory systems, GrepMem requires no embeddings, retrieval index, memory extraction, or additional models. We train Qwen3.5-9B with reinforcement learning under five context management strategies and evaluate on multi-question research sessions, long-stream recall, and agent-trajectory question answering. GrepMem achieves 95-97% accuracy on ContextBench, versus at most 21% for lossy baselines, and outperforms SUPO's summarization by 8.8 percentage points on BCP-Multi. GrepMem scales effectively to increasingly long input streams while remaining on the accuracy-token-cost Pareto frontier across all five evaluation tasks. Our results show that a plain-text log and grep provide a simple but competitive approach to long-horizon agent memory, without requiring complex structured memory infrastructure.
Figures & tables
Figure 1: Mean Accuracy vs. Token Cost After RL. We compare four context management strategies by their mean accuracy and token usage across five evaluation tasks. SUPO compresses context through summarization, while Masking discards older tool outputs to free context space. MemGPT maintains editable working memory, archival memory, and searchable message history. Our method, GrepMem , combines summarization with a lossless, searchable history log, enabling on-demand retrieval through grep without a retrieval index. Token usage includes prompt and generated tokens across all model calls. Figure 5 reports per-task results.
Figure 2: GrepMem Overview. (a) The basic agent loop generates responses and tool calls from its active context, while GrepMem additionally appends these interactions and tool results to a persistent history log. (b) Context compaction generates a summary when the working context reaches the compaction limit, allowing the agent to resume execution with a compacted context. GrepMem builds on SUPO’s summarization loop. Extending this framework, GrepMem enables (c) on-demand retrieval through grep_history and (d) a lossless, append-only full history log containing user messages, tool calls, tool results, summaries, and agent responses. By combining context summarization with a log preserving every message and tool output verbatim, GrepMem enables grep_history to retrieve matching log lines after the first compaction without embeddings, a retrieval index, or an additional model.
Method
How it works
Scales past the window
Keeps everything
Recall
No index
ReAct ( Yao et al., 2023 )
keeps all, stops when full
✗
✓
✗
✓
Masking ( Lindenbauer et al., 2025 )
hides old tool outputs
✓
✗
✗
✓
SUPO ( Lu et al., 2026 )
summarizes and restarts
✓
✗
✗
✓
MemGPT ( Packer et al., 2024 )
summarizes, manages memory
✓
✓
✓
✗
GrepMem (ours)
summarizes, greps raw log
✓
✓
✓
✓
Table 1: Comparison of Context Management Strategies. We compare how five strategies manage context, preserve interaction history, and support retrieval. GrepMem augments SUPO with an automatically appended plain-text history log and a grep tool, preserving the full history for retrieval without an index.
Task
Questions
Timing
BCP-Multi
828
before
AMA-Bench file
2,496
before
AMA-Bench stream
2,268
after
ContextBench KV
576
after
ContextBench Log
480
after
Table 2: Evaluation Task Sizes. Questions are totals per evaluation run. Timing indicates whether questions are revealed before or after the relevant input.
Questions known before the input
Questions revealed after the input
Strategy
BCP-Multi
AMA-Bench file
AMA-Bench stream
ContextBench KV
ContextBench Log
Avg.
After RL
ReAct
0.2
38.3
26.6
0.0
0.0
13.0
SUPO
38.1
45.5
45.2
13.9
6.6
29.8
Masking
34.8
44.5
44.9
20.7
6.6
30.3
MemGPT
39.9
46.4
45.4
99.9
50.0
56.3
Table 3: Accuracy Before and After RL Training. We compare the accuracy (%) of five context management strategies using Qwen3.5-9B before and after training with the same RL recipe (§ 4 ). Within each block, bold marks the best value in each column, and underlining marks the second best. For ContextBench, we report exact-value accuracy on KV Store and exact-answer accuracy on Log Triage. Avg. denotes the mean accuracy across five tasks.
Figure 3: Evaluation Performance of Five Context Management Strategies During RL Training. Scores are averaged over the evaluation samples available at each checkpoint. Evaluation details are provided in Appendix E .
Figure 4: ContextBench Performance Across Stream Lengths. We evaluate how stream length affects ContextBench performance before (dashed) and after RL (solid) training.
Figure 5: Per-Task Accuracy vs. Token Cost After RL. We compare the accuracy and token usage of RL-trained strategies on each evaluation task. Cost is the number of tokens processed per episode (prompt and generated tokens over all LLM calls; ContextBench: per stream), on a log scale. Dashed: the Pareto frontier.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Base model
Qwen3.5-9B (thinking)
Algorithm
GRPO, std-normalized advantages
Prompts per step / rollouts per prompt
8 / 8
Steps
80
Learning rate / weight decay
10−6 / 0.1
Clip range (low / high)
0.2 / 0.28
KL penalty
low-variance KL, β=0.01
Appendix
Table 4: Training Configuration. We report the model, optimization settings, interaction budgets, and hardware used for training. Training hyperparameters are shared across all context management strategies.
Figure 6: Per-Task Accuracy vs. Runtime After RL. We compare the accuracy and wall-clock runtime of RL-trained strategies on each evaluation task. Runtime is measured in minutes per episode, or per stream for ContextBench, and is shown on a logarithmic scale. Dashed lines indicate the Pareto frontier. Runtime measurements are approximate because episodes ran concurrently on shared inference and retrieval servers.
Subset
SUPO
Mask.
MemGPT
GrepMem
≤ 32K tokens (124)
48.2
48.3
47.9
48.3
32K–128K tokens (65)
39.4
38.3
40.6
41.7
A: recall
48.8
48.3
49.2
50.0
B: causal inference
42.2
44.1
45.1
43.1
C: state updating
51.7
49.4
49.8
53.6
D: state abstraction
31.9
32.0
31.1
30.4
Appendix
Table 5: AMA-Bench Stream Accuracy by Length and Question Type. We break down accuracy (%) after RL training by trajectory length and question type. Length groups are defined by whether trajectories fit within the 32K-token context window, with trajectory counts shown in parentheses. Question types cover recall, causal inference, state updating, and state abstraction. Bold text marks the best value in each row.
Benchmark
SUPO
Masking
MemGPT
GrepMem
BCP-Multi
1.15M / 84
949K / 51
1.97M / 101
1.22M / 88
AMA-Bench file
590K / 41
490K / 34
674K / 38
560K / 38
AMA-Bench stream
377K / 24
404K / 23
1.30M / 67
869K / 56
ContextBench
2.11M / 114
2.41M / 112
8.60M / 391
2.51M / 154
Appendix
Table 6: Token Usage and LLM Calls After RL. We report the average token usage and number of LLM calls per episode for each strategy and benchmark. Each entry shows processed tokens / LLM calls, with tokens including prompt and generated tokens summed over all calls. ContextBench costs are measured per stream. ReAct is omitted because its selected checkpoint was not retained.
Large language model (LLM) agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence. Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.
Memory is an indispensable capability for long-horizon LLM agents, enabling them to preserve and utilize information accumulated across extended interactions. Existing memory-agent approaches are typically trained end-to-end with reinforcement learning on downstream tasks. However, collecting high-quality annotated problems for memory-intensive scenarios is costly, and the resulting training data often lack sufficient diversity to cover general memory behaviors. In this work, we propose MemTrain, a self-supervised training framework for generally enhancing the context-memory capability of LLM agents for more effective downstream post-training. MemTrain introduces two coupled proxy tasks over unlabeled Wikipedia corpora: (1) an end-to-end masked reconstruction objective, which requires the model to recover masked entities after multiple rounds of memory updates, thereby encouraging memory maintenance from the final outcome perspective; and (2) an intermediate memory recall objective, which requires the model to reconstruct masked historical information using intermediate memory states, encouraging faithful compression and memory completeness throughout the interaction process. The two objectives are jointly optimized using GRPO. Extensive experiments on long-text QA and search-based QA benchmarks demonstrate that MemTrain consistently improves downstream memory-intensive reasoning performance across different models, achieving gains of up to 17.67 points over direct task-specific post-training.
Ziheng Li, Xingrun Xing, Haoqing Wang +2
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China
LLM agents that operate over long inputs must perform context management: deciding what to retain in a fixed-size context window. A common approach is compaction, where the model replaces its history with a self-generated note. Yet off-the-shelf agentic models, including GPT-5.5 and GLM-5.3, often write ineffective notes, and reinforcement learning (RL) alone rarely discovers useful note-taking behavior. We present MEMENTO, a long-context agent built on Qwen3-8B and trained to manage its context through compaction. We first bootstrap note-taking by fine-tuning on synthetic trajectories annotated with notes generated in hindsight, then reinforce it with RL using a reward that combines answer correctness, note-quality heuristics, and confidence calibration. Although trained only to compact into a fresh window, MEMENTO extends at test time to strategies such as context folding and confidence-based backtracking. On book-level question answering across four benchmarks, with books of up to 2.5M tokens, MEMENTO raises Qwen3-8B's accuracy from 36.9% to 67.8% while never holding more than 20K tokens in context. It outperforms GPT-5.5 agent in the same harness (13.1%) and full book context (56.7%) at less than 1/50 the cost. It also nearly matches a ~100x larger GLM-5.3 agent (69.3%) at 1/9 the cost, and comes within 6.4 points of GLM-5.3 full context at 1/26 the cost.