AI systems must often maintain bounded memory of unbounded interaction histories. However, optimizing the memory representation, and its updates, for downstream task utility is difficult for standard reinforcement learning (RL) due to severe credit assignment bottlenecks induced by sparse, delayed rewards. We introduce MaxIM (Maximally Informative Incremental Memory), a framework that formulates incremental summarization as a capacity-constrained, information-theoretic reward-shaping problem. Grounded in value suboptimality bounds, we show that bounding sequential information loss motivates maximizing the predictive log-likelihood of future, task-relevant targets, a variational lower bound on predictive sufficiency. We introduce a recursive consistency condition to ensure summaries are grounded by maximizing mutual information (MI) with the observed history. We mitigate credit assignment issues with a parameterized asymmetric critic that computes step-wise variational MI bounds for dense reward shaping. Empirical evaluation on long-horizon scientific reading and multi-turn dialogue benchmarks shows that MaxIM outperforms RL, supervised fine-tuning, and prompt-focused baselines on factual grounding, comprehensive autorater metrics, and downstream task utility across strict token budgets and extended horizons, without the need for expensive human preference labels.
Figures & tables
Figure 1: The MaxIM framework. Streaming Inference (Solid Lines): A trained actor policy incrementally compresses prior summaries zt−1 and incoming observations ot into a bounded memory zt . This memory, alongside recent observations, provides the context for a frozen downstream agent interacting with the environment with zero critic calls at test time. Asymmetric Training (Dotted Lines): During training, a pretrained critic qϕ evaluates the actor’s summary using oracle context Yt and the immediate history (zt−1,ot) . It computes dense, information-theoretic shaping rewards (predictive sufficiency and recursive consistency) that step-wise guide the policy, bypassing the credit assignment bottleneck of the sparse terminal task reward Rtask .
arXiv Scientific Reading ( K=500 )
WildChat Dialogue ( Δ vs. No-Belief ↑ )
Method
Abstain MCQ ↑
Abst. (%) ↓
Early Chunk ↑
All T ( N=500 )
T≥10 ( N=158 )
Sat.
Goal
Pref.
Sat.
Pref.
Prompt-Only
31.30±1.28
68.1
17.35±1.05
−0.09
−0.09
+0.01
−0.09
+0.08
Policy SFT
38.76±1.29
59.1
12.47±0.87
−0.04
−0.05
+0.05
−0.21
+0.00
Best-of- N MI SFT
29.05±1.16
70.1
—
—
—
—
—
—
MemAgent RL [ 7 ]
52.90±1.40
46.0
—
−0.28‡
−0.27‡
−0.13‡
−0.15
+0.02
Table 1: Main downstream task utility results (RQ1). Left: arXiv Abstain-MCQ accuracy ( ± SEM; N=338 ), abstention rate, and Early-Section Stress Test ( N=332 , chunks 3 – 7 ). Right: WildChat paired Δ vs. memory-free No-Belief ( k=1 only) on 1 – 10 GPT-4.1 agent ratings over all N=500 dialogues (No-Belief: 8.58 Sat., 8.75 Goal, 8.74 Pref.) and the T≥10 cohort ( N=158 ; No-Belief: 8.08 Sat., 8.28 Pref.; full scores and Coherence in Table 9 ). Best bold ; second-best underlined . †p<0.05 , ‡p<0.01 vs. MaxIM (paired bootstrap).
Figure 2: Budget scaling and long-horizon dynamics. (a) arXiv token-budget Pareto curve ( K∈{100,300,500,800} ; K=500 baselines labeled): MaxIM at K=100 outperforms Prompt-Only at a 5× larger K=500 budget. (b) Chunk claim incorporation (percentage of atomic claims in chunk ot entailed by zt ) across 5-chunk windows ( t=1…25 ): MaxIM resists late-turn memory saturation. (c) WildChat GPT-4.1 pairwise belief-state Usefulness win rate vs. Prompt-Only.
arXiv Scientific Reading ( N=338 )
WildChat Multi-Turn Dialogue ( N=500 )
Method
Entail (%) ↑
Contr (%) ↓
Gnd. Claims ↑
Use Win %
Cons. (1–10)
Entail (%) ↑
Contr (%) ↓
Gnd. Claims ↑
Use Win %
Truth Win %
Prompt-Only
98.04
0.50
33.0
—
6.16
84.38
11.04
18.7
—
—
Policy SFT
75.01
11.62
21.0
13.4
7.12
83.68
9.00
16.5
37.6
36.2
MemAgent RL
98.35
0.62
47.6
80.1
6.07
85.42
9.35
21.0
66.4
61.0
Task-Only RL
98.37
0.65
47.3
62.5
6.44
87.50
7.04
23.1
79.8
69.8
MI-Only RL
98.13
0.78
63.4
47.0
6.04
85.19
10.53
20.5
40.8
40.8
Table 2: Source grounding and qualitative summary evaluation (RQ2; GPT-4.1). Grounding of terminal zT , pairwise Usefulness / Truthfulness win rates vs. Prompt-Only, and 1 – 10 Consistency .
(a) Reward Component Ablation
(b) arXiv Annealing ( K=500 )
Rtask
rpred
rcons
Variant
arXiv MCQ (%) ↑
WC Contr (%) ↓
WC Δ Sat. ↑
Decay (% Steps)
Val MCQ (%)
Test MCQ (%) ↑
✓
Task-Only
54.44±1.45
7.04±0.49
−0.30
None ( λs=1 )
73.18
57.10±1.40
✓
✓
Suff-Only
51.54±1.46
6.48±0.41
−0.18
Early ( 13 – 60% )
73.18
61.18±1.33
✓
✓
MI-Only
50.30±1.50
10.53±0.44
+0.00
Late ( 65 – 90% )
74.55
61.12±1.44
✓
✓
✓
MaxIM
66.39±1.31
4.27±0.27
+0.05
Mid ( 45 – 60% )
75.91
66.39±1.31
vs. 2nd-best:
+11.95 pp
−2.21 pp
+0.05
vs. none:
+2.73 pp
+9.29 pp
Table 3: Reward-component and annealing ablations (RQ3). (a) Isolating Rtask , rpred , and rcons ( WC Δ Sat. vs. No-Belief). (b) Cosine-decay window of λs (% of RL steps; N=338 ).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Setting / Hyperparameter
arXiv Scientific Reading
WildChat Multi-Turn Dialogue
Base Backbone (Policy, Critic, Actor)
Gemini 2.5 Flash
Offline Target / RL Belief Rater
Gemini 3.0 Flash
Atomic-Claim Splitter
Gemini 2.5 Flash-Lite
Prompts per Step × Group Size G
8×8=64 trajectories/step
Offline SFT Batch Size (Critic / Policy SFT)
32 – 64 ( ≤1 epoch over HQ targets)
Discount γ / Step Scale ηshape
0.9 / 0.1 (MemAgent RL: γ=1.0 )
Appendix
Table 4: Summary of training and evaluation settings. Settings shared identically across both benchmarks span a single combined column.
Method
Offline SFT Steps
Online RL Trajectories
Critic Passes per RL Turn
Rel. Train FLOPs
Test Critic
Test Memory Calls / FLOPs
Prompt-Only
0
0
0
0
0
T calls ( 1.00× ) ≤K tok/turn (identical across all methods)
Policy SFT
1,000 (arXiv) 600 (WC)
0
0
≈0.02×
0
Best-of- N MI SFT ( N=8 , arXiv)
350 (+ 8×533 offline rollouts)
0
2 cond. + cached base.
≈0.08×
0
MemAgent RL / Task-Only RL
0
64×Smax ( 64K arXiv / 51.2K WC)
0
1.00×
0
MaxIM (Full, Ours)
1,000 / 900 (Critic SFT)
64×Smax ( 64K arXiv / 51.2K WC)
2 cond. + cached base. ( s<600 )
≈1.25× (arXiv) ≈1.17× (WC)
0
Appendix
Table 5: Training-time forward/generation budget, relative training FLOPs ( 1.00×= sparse-reward RL), and test-time inference cost per T -turn episode. Smax is total RL update steps ( 1,000 on arXiv; 800 on WildChat). Critic passes use teacher-forced log-likelihoods ( 0 generated tokens).
Budget
Method
Abstain- MCQ ↑
Abstain % ↓
Avg. Tokens
Terminal Tokens
arXiv K=100
Prompt-Only
7.04±0.69
92.07±0.70
97.7±0.1
98.1±0.2
arXiv K=100
MaxIM (Ours)
34.26±1.27
64.32±1.28
95.8±0.1
98.7±0.1
arXiv K=300
Prompt-Only
22.43±1.11
76.75±1.13
284.2±0.6
297.7±1.2
arXiv K=300
MaxIM (Ours)
50.06±1.35
48.40±1.36
276.8±0.6
298.3±0.4
arXiv K=500
Prompt-Only
31.30±1.28
68.11±1.29
469.0±0.8
500.3±1.6
arXiv K=500
MaxIM (Ours)
66.39±1.31
32.01±1.31
427.7±1.5
496.7±0.5
Appendix
Table 6: Full arXiv token-budget Pareto scaling ( N=338 ). Average ± SEM across episodes (SentencePiece token counts under Gemini 2.5 Flash tokenizer).
Method
Chunks 1 – 5
Chunks 6 – 10
Chunks 11 – 15
Chunks 16 – 20
Chunks 21 – 25
Prompt-Only
49.8%
31.7%
19.5%
15.9%
15.4%
Policy SFT
48.5%
31.4%
22.6%
18.0%
17.6%
MemAgent RL ( γ=1.0 )
44.9%
38.3%
25.7%
19.2%
18.6%
Task-Only RL ( γ=0.9 )
50.2%
40.0%
27.0%
21.0%
20.4%
MI-Only RL ( Rtask=0 )
56.5%
44.2%
30.5%
24.0%
23.3%
MaxIM (Full, Ours)
50.2%
44.2%
33.9%
27.3%
26.8%
Appendix
Table 7: arXiv turn-wise chunk claim incorporation (%) across 5-chunk streaming windows ( N=338 , K=500 ; supporting Figure 2 (b)). Entries measure the percentage of atomic claims in incoming chunk ot (extracted by Gemini 2.5 Flash-Lite) entailed by updated summary zt (verified by Gemini 3.0 Flash).
Table 8: Offline critic validation comparing Raw VPL vs. PMI on held-out arXiv trajectories/triplets and held-out WildChat validation triplets.
Method
User Sat. ↑
Goal Comp. ↑
Pref. Ret. ↑
Coherence ↑
Avg. Turns
(i) All Test Dialogues ( N=500 )
No-Belief ( k=1 )
8.58±0.11
8.75±0.10
8.74±0.10
8.70±0.09
8.75
Prompt-Only
8.49±0.11 ( −0.09 )
8.66±0.10 ( −0.09 )
8.75±0.09 ( +0.01 )
8.62±0.10 ( −0.08 )
8.74
Policy SFT
8.54±0.11 ( −0.04 )
8.70±0.10 ( −0.05 )
8.79±0.09 ( +0.05 )
8.72±0.09 ( +0.02 )
8.61
MemAgent [ 7 ]
8.30±0.11 ( −0.28 )
8.48±0.10 ( −0.27 )
8.60±0.10 ( −0.13 )
8.44±0.11 ( −0.26 )
8.84
Task-Only RL
8.28±0.12 ( −0.30 )
8.46±0.11 ( −0.29 )
8.58±0.10 ( −0.15 )
8.47±0.10 ( −0.23 )
8.58
Appendix
Table 9: Full WildChat bounded-memory ( k=1 , T≤15 ) downstream chat-agent evaluation (GPT-4.1 judge without summary access, 1 – 10 scale). Average ± SEM (with paired Δ vs. No-Belief in parentheses, computed prior to rounding) on all N=500 test seeds and the fixed No-Belief T≥10 long-horizon cohort ( N=158 ).
Method
Test ( N=500 )
Unseen-Sim OOD ( N=200 )
Turn 15 Extrap.
Prompt-Only
8.32±0.07
8.44±0.11
7.11
Policy SFT
8.31±0.06
8.29±0.09
6.96
MI-Only RL ( Rtask=0 )
8.33±0.07
8.32±0.10
6.85
Suff-Only RL ( wc=0 )
8.76±0.06
8.76±0.09
7.50
Task-Only RL ( γ=0.9 )
8.79±0.06
8.90±0.08
7.79
MemAgent RL ( γ=1.0 ) [ 7 ]
8.98±0.05
8.92±0.09
8.18
Appendix
Table 10: WildChat 1 – 10 causal belief-state tracking fidelity (training reward metric) across held-out Test ( N=500 ), unseen-simulator OOD ( N=200 ), and T=15 horizon extrapolation.
Table 11: Turn-stratified GPT-4.1 qualitative evaluation. (a) arXiv pairwise Usefulness win rate (%) vs. Prompt-Only across chunks t (plus standalone 1 – 10 Consistency at t=20 , and direct MaxIM vs. MemAgent judging). (b) WildChat pairwise Usefulness / Truthfulness win rates (%) vs. Prompt-Only across turns (active conversation counts N shown per column on a 100 -seed replay subset alongside all N=500 terminal states).
In-context reinforcement learning (ICRL) enables agents to adapt to new tasks without parameter updates, relying on memory to retain what they learn from experience. For agents operating continually, existing approaches using memory trade off bounded cost against access to past experience. Approaches using growing memories, such as full-history attention and accumulating summaries, incur storage and per-decision costs that increase with task length. Fixed-size memories bound storage but restrict access: sliding windows discard older steps, while recurrent networks compress even recent interactions into a hidden state. We introduce Recurrent Summary Memory (RSM), a Transformer policy that attends exactly to a short working buffer of recent interactions and carries a fixed-capacity summary of earlier experience. When the buffer fills, RSM rewrites the summary from its previous contents and the buffer, then clears the buffer, so deployment memory and per-decision cost stay O(1) in task length. On long-running partially observable tasks, frozen RSM policies sustain their adapted performance for eight times the training horizon with about 1% of full-history memory, ending ahead of every evaluated baseline while growing memories deteriorate. When the task configuration changes mid-run, RSM re-adapts better than growing and recurrent memories. Clearing the summary or blocking the policy's access to it removes its benefit, while swapping in another task's summary steers behaviour towards that task. Together, these results show that our fixed-capacity summary can sustain in-context adaptation far beyond the training horizon.
Persistent textual memory allows language models to carry information across long interactions, but learning what to remember is fundamentally a credit-assignment problem. A memory rewrite may only become useful many steps later, while much of the observed utility may be inherited from information already stored before the rewrite. We introduce Memory Gain Policy Optimization (MGPO), which isolates the incremental value of each memory rewrite by crediting it for its marginal contribution to current and future downstream utility. This turns delayed memory utility into a direct learning signal for optimizing what information should persist. We study MGPO on document-level information extraction, where structured supervision makes the effects of individual memory updates directly measurable. MGPO improves extraction while reducing average memory length by nearly 80% relative to the initial memory policy before optimization. The learned memory policy also supports reuse and transfer across domains, downstream models without further training. These results show that effective memory learning depends not only on preserving useful information, but on identifying which memory updates create lasting incremental value.
Jiaming Tang, Mingyan Liu, Armin Sarabi
Department of Electrical Engineering and Computer Science, University of Michigan
Reinforcement Learning with Verifiable Rewards (RLVR) is a central technique for improving long-horizon reasoning in Large Language Models (LLMs). However, existing RLVR methods often encourage unnecessarily long reasoning rollouts, which can degrade reasoning coherence and exhaust the available context budget. Existing approaches to long-context organization often depend on external mechanisms to organize rollouts, rather than enabling the model to manage its own reasoning trajectory. To address this limitation, we propose ReSum, a novel RLVR framework that enables LLMs to compress and organize their reasoning trajectories through self-summarization. Our pilot studies show that self-summarization stabilizes generation by lowering token-level entropy, and that introducing a ``summarization'' phrase can substantially mitigate errors propagated from an incorrect rollout prefix. Motivated by these findings, ReSum adopts a summarization-aware adaptive rollout mechanism that contrastively evaluates whether self-summarization benefits the ongoing reasoning process. Specifically, when the model spontaneously triggers self-summarization, ReSum masks the summarization phrase to create a contrastive branch; for non-summarization positions, it instead randomly injects the phrase to create a matched branch. We further design a summarization-aware advantage to enable finer-grained comparison between contrastive rollout trajectories. Extensive experiments show that ReSum improves performance at an average of 4% while reducing rollout length by 18.6%.
Xucong Wang, Ziyu Ma, Yong Wang +5
University of Science and Technology of China · 2AMAP, Alibaba Group