LLM agents that operate over long inputs must perform context management: deciding what to retain in a fixed-size context window. A common approach is compaction, where the model replaces its history with a self-generated note. Yet off-the-shelf agentic models, including GPT-5.5 and GLM-5.3, often write ineffective notes, and reinforcement learning (RL) alone rarely discovers useful note-taking behavior. We present MEMENTO, a long-context agent built on Qwen3-8B and trained to manage its context through compaction. We first bootstrap note-taking by fine-tuning on synthetic trajectories annotated with notes generated in hindsight, then reinforce it with RL using a reward that combines answer correctness, note-quality heuristics, and confidence calibration. Although trained only to compact into a fresh window, MEMENTO extends at test time to strategies such as context folding and confidence-based backtracking. On book-level question answering across four benchmarks, with books of up to 2.5M tokens, MEMENTO raises Qwen3-8B's accuracy from 36.9% to 67.8% while never holding more than 20K tokens in context. It outperforms GPT-5.5 agent in the same harness (13.1%) and full book context (56.7%) at less than 1/50 the cost. It also nearly matches a ~100x larger GLM-5.3 agent (69.3%) at 1/9 the cost, and comes within 6.4 points of GLM-5.3 full context at 1/26 the cost.
Figures & tables
Figure 1: Context management methods incorporated in this work. (a) Compaction starts a new window containing only the saved notes and discards all prior turns. (b) Context folding additionally retains the turns after the latest note. (c) Confidence-based backtracking resumes from an earlier note checkpoint after a low-confidence answer and generates an alternative continuation.
Figure 2: Method overview. (a) We inject note-taking actions into collected demonstrations. (b) SFT initializes the policy, and GRPO reinforces its actions across context resets with a shared trajectory advantage. (c) At inference, context folding retains notes and recent turns, and confidence guides retries and answer selection.
Accuracy (%)
Efficiency
Model
Category
Harness
LitQA
NHopQA
NovelQA
Saga
Micro Avg
Tokens
Cost ($)
( n = 1,940)
(426)
(2,305)
(37)
(4,708)
Off-the-shelf models, full book in context
Qwen3-8B (40K ctx)
Reference
Full Context
44.8
20.4
29.8
2.7
34.9
33K
11
GPT-5.5 High (232K ctx) †
Reference
Full Context
51.0
54.0
67.0
51.4
56.7
144K
2,577
GLM-5.3 High (1M ctx)
Reference
Full Context
76.6
75.8
71.9
75.7
74.3
167K
1,270
Table 1: Test accuracy (%) on LiteraryQA (LitQA), NovelHopQA (NHopQA), NovelQA, and SagaScale (Saga); n is the number of test questions. Tokens are the mean per rollout, and cost is for the full test set. Bold marks the best result among ablation and proposed methods. Memento is best on every dataset. In the same harness, it nearly matches GLM-5.3 (67.8 vs. 69.3 micro) at 1/9 the cost. It beats GPT-5.5 High in both settings, including full-book context (67.8 vs. 56.7), at ∼1/54 the cost. It is also within 6.5 points of full-context GLM-5.3 at ∼1/26 the cost. † Evaluated on up to 100 random questions per dataset; cost is extrapolated to the full test set.
Figure 3: Validation answer accuracy over RL training for the models in Table 1 . The gray dot marks a single outlier evaluation of the RL (Single Window) run, caused by reward hacking that led us to tighten the answer length limit. While SFT + RL and RL only variants start at similar validation accuracy, SFT + RL outperforms RL only by 15% at the end of training due to more effective compaction behavior instilled by the SFT.
Figure 4: Mean context-window restarts per RL training rollout. Legend names follow Table 1 ; training uses compaction into fresh windows. Faint curves show per-step values; bold curves show five-step trailing means. SFT + RL checkpoints retain compaction ratio robustly during the RL training process, while compaction ratio in RL only checkpoints collapses.
Single
+ Context
+ Back-
Model
window
folding
tracking
Qwen3-8B (untrained)
29.6
32.6
36.9
SFT
34.4
46.6
59.1
Memento (SFT + RL)
58.6
66.9
67.8
Table 2: Ablation of our training and inference components. We report test-set accuracy (%) of the confidence-aware variant of each checkpoint. Rows add training stages (SFT on synthetic data, then RL), and columns add inference-time components cumulatively (context folding, then confidence-aware backtracking). Accuracy improves along both axes: each training stage helps under every inference setup, and each inference component helps for every checkpoint. The best accuracy is shown in bold.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Initial model
Qwen3-8B
Optimizer
AdamW
Peak / minimum learning rate
2×10−5 / 4×10−6
Learning-rate schedule
Single cosine decay
Warmup
0.1 epoch
Weight decay
0.01
Appendix
Table 3: SFT hyperparameters, shared by the variants with and without confidence. Sequences exceeding the limit are truncated from the left, and an assistant turn cut by truncation is excluded from the loss. We use the final checkpoint after two epochs.
Hyperparameter
Value
Advantage estimator
GRPO
RL rollout batch size
256 questions
Policy minibatch size
64 questions
GRPO group size
16
Policy update epochs
1 per rollout batch
Max context window
4K prompt / 12K output
Appendix
Table 4: GRPO hyperparameters, shared by the four RL conditions. The policy and fixed KL reference start from the corresponding SFT checkpoint, or from Qwen3-8B for the RL-only baselines. The confidence reward weight is λ=0.10 through step 160 and 0.25 thereafter, and variants without confidence use λ=0 . Reported RL results use step 222.
Hyperparameter
Value
Policy sampling
Qwen temperature / top- p
1.0 / 1.0
Qwen top- k
Off; untrained test baseline: 20
Validation / test samples
1 per question, before retries
GLM test temperature / top- p
1.0 / 0.95
GLM test reasoning effort
High
Appendix
Table 5: Inference and generation hyperparameters. Qwen sampling applies to GRPO, validation, and test runs. Response budgets include tool observations, except for per-turn generation caps; the folding segment budget is a soft limit. Tool outputs are truncated from the right; full-book inputs retain the beginning. In the retry schedule, n counts note calls accumulated at that retry. Notes and retries do not apply to the no-compaction baseline.
Long-horizon large language model (LLM) agents commonly retain their complete interaction history until compaction is triggered at a predefined threshold. We study Continuous Context Management (CCM), which performs compaction at every turn to prevent interaction history from accumulating in the active prompt. At each turn, a CCM agent emits an updated memory together with an environment action; its next prompt contains the original task, retained memory, and newest observation rather than the complete transcript. We first evaluate CCM without fine-tuning on TerminalBench-2 using Claude Sonnet 4.6, Claude Opus 4.6, GLM-5, and Kimi K3. CCM substantially reduces cumulative input usage and active-prompt size, although it lowers task success for most models while preserving performance for Kimi K3. We use GRPO with privileged full-history distillation to improve CCM in open-weight models. A frozen copy of the student's initial model scores each sampled student action under the complete history reconstructed from that student's rollout, providing dense action-token supervision without a separate teacher rollout or reference solution. On WebShop, this objective substantially improves CCM over GRPO at both evaluated model scales and surpasses full-history GRPO for Qwen3-4B-Instruct, though not for Qwen3-8B. On Endless Terminals, the augmented method provides a modest improvement over GRPO, with both CCM policies outperforming the untrained full-history baseline. These results demonstrate that CCM is a viable inference paradigm for agents operating with substantially reduced retained context and that its performance can be improved through reinforcement learning with privileged full-history distillation.
Memory is an indispensable capability for long-horizon LLM agents, enabling them to preserve and utilize information accumulated across extended interactions. Existing memory-agent approaches are typically trained end-to-end with reinforcement learning on downstream tasks. However, collecting high-quality annotated problems for memory-intensive scenarios is costly, and the resulting training data often lack sufficient diversity to cover general memory behaviors. In this work, we propose MemTrain, a self-supervised training framework for generally enhancing the context-memory capability of LLM agents for more effective downstream post-training. MemTrain introduces two coupled proxy tasks over unlabeled Wikipedia corpora: (1) an end-to-end masked reconstruction objective, which requires the model to recover masked entities after multiple rounds of memory updates, thereby encouraging memory maintenance from the final outcome perspective; and (2) an intermediate memory recall objective, which requires the model to reconstruct masked historical information using intermediate memory states, encouraging faithful compression and memory completeness throughout the interaction process. The two objectives are jointly optimized using GRPO. Extensive experiments on long-text QA and search-based QA benchmarks demonstrate that MemTrain consistently improves downstream memory-intensive reasoning performance across different models, achieving gains of up to 17.67 points over direct task-specific post-training.
Ziheng Li, Xingrun Xing, Haoqing Wang +2
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China
LLM agents increasingly face long-horizon tasks such as web search and deep research in real-world applications, where accumulated context can cause long-context degradation and reasoning failures. Prior work mitigates this through context management with agent-side context control or fixed strategies such as summarization, which require training the agent itself for adaptation - making it impractical for closed-source agents and ignoring that different agents may require different strategies. We introduce Adaptive Context Management (AdaCoM), which trains an external LLM to manage the context of a frozen agent through flexible modification actions and end-to-end reinforcement learning. Across diverse agents on web search and deep research benchmarks, AdaCoM substantially improves performance by preserving task constraints and progress while pruning stale content. The learned strategies reveal a Fidelity-Reliability Trade-off: agents with higher vanilla ReAct performance benefit from higher-fidelity context preservation, whereas lower-performing agents require more aggressive compression to stay within a reliable reasoning regime. Transfer experiments show that AdaCoM generalizes most effectively across agents with similar capability (measured by vanilla ReAct performance), suggesting a practical path toward reusable context managers for agent systems.
Lu Yi, Runlin Lei, Liuyi Yao +6
1Renmin University of China · Work done during internship at Tongyi Lab, Alibaba Group · 2Tongyi Lab, Alibaba Group +2