MaxIM: Maximally Informative Incremental Summarization via Reinforcement Learning
Organizations: Google Research · Noetive · Google DeepMind
Abstract
AI systems must often maintain bounded memory of unbounded interaction histories. However, optimizing the memory representation, and its updates, for downstream task utility is difficult for standard reinforcement learning (RL) due to severe credit assignment bottlenecks induced by sparse, delayed rewards. We introduce MaxIM (Maximally Informative Incremental Memory), a framework that formulates incremental summarization as a capacity-constrained, information-theoretic reward-shaping problem. Grounded in value suboptimality bounds, we show that bounding sequential information loss motivates maximizing the predictive log-likelihood of future, task-relevant targets, a variational lower bound on predictive sufficiency. We introduce a recursive consistency condition to ensure summaries are grounded by maximizing mutual information (MI) with the observed history. We mitigate credit assignment issues with a parameterized asymmetric critic that computes step-wise variational MI bounds for dense reward shaping. Empirical evaluation on long-horizon scientific reading and multi-turn dialogue benchmarks shows that MaxIM outperforms RL, supervised fine-tuning, and prompt-focused baselines on factual grounding, comprehensive autorater metrics, and downstream task utility across strict token budgets and extended horizons, without the need for expensive human preference labels.
Figures & tables
| arXiv Scientific Reading ( ) | WildChat Dialogue ( vs. No-Belief ) | |||||||
| Method | Abstain MCQ | Abst. (%) | Early Chunk | All ( ) | ( ) | |||
| Sat. | Goal | Pref. | Sat. | Pref. | ||||
| Prompt-Only | ||||||||
| Policy SFT | ||||||||
| Best-of- MI SFT | — | — | — | — | — | — | ||
| MemAgent RL [ 7 ] | — | |||||||
| arXiv Scientific Reading ( ) | WildChat Multi-Turn Dialogue ( ) | |||||||||
| Method | Entail (%) | Contr (%) | Gnd. Claims | Use Win % | Cons. (1–10) | Entail (%) | Contr (%) | Gnd. Claims | Use Win % | Truth Win % |
| Prompt-Only | — | — | — | |||||||
| Policy SFT | ||||||||||
| MemAgent RL | ||||||||||
| Task-Only RL | ||||||||||
| MI-Only RL | ||||||||||
| (a) Reward Component Ablation | (b) arXiv Annealing ( ) | ||||||||
| Variant | arXiv MCQ (%) | WC Contr (%) | WC Sat. | Decay (% Steps) | Val MCQ (%) | Test MCQ (%) | |||
| ✓ | Task-Only | None ( ) | |||||||
| ✓ | ✓ | Suff-Only | Early ( – ) | ||||||
| ✓ | ✓ | MI-Only | Late ( – ) | ||||||
| ✓ | ✓ | ✓ | MaxIM | Mid ( – ) | |||||
| vs. 2nd-best: | pp | pp | vs. none: | pp | pp | ||||
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting / Hyperparameter | arXiv Scientific Reading | WildChat Multi-Turn Dialogue |
|---|---|---|
| Base Backbone (Policy, Critic, Actor) | Gemini 2.5 Flash | |
| Offline Target / RL Belief Rater | Gemini 3.0 Flash | |
| Atomic-Claim Splitter | Gemini 2.5 Flash-Lite | |
| Prompts per Step Group Size | trajectories/step | |
| Offline SFT Batch Size (Critic / Policy SFT) | – ( epoch over HQ targets) | |
| Discount / Step Scale | / (MemAgent RL: ) | |
| Method | Offline SFT Steps | Online RL Trajectories | Critic Passes per RL Turn | Rel. Train FLOPs | Test Critic | Test Memory Calls / FLOPs |
|---|---|---|---|---|---|---|
| Prompt-Only | calls ( ) tok/turn (identical across all methods) | |||||
| Policy SFT | (arXiv) (WC) | |||||
| Best-of- MI SFT ( , arXiv) | (+ offline rollouts) | cond. + cached base. | ||||
| MemAgent RL / Task-Only RL | ( arXiv / WC) | |||||
| MaxIM (Full, Ours) | / (Critic SFT) | ( arXiv / WC) | cond. + cached base. ( ) | (arXiv) (WC) |
| Budget | Method | Abstain- MCQ | Abstain % | Avg. Tokens | Terminal Tokens |
|---|---|---|---|---|---|
| arXiv | Prompt-Only | ||||
| arXiv | MaxIM (Ours) | ||||
| arXiv | Prompt-Only | ||||
| arXiv | MaxIM (Ours) | ||||
| arXiv | Prompt-Only | ||||
| arXiv | MaxIM (Ours) |
| Method | Chunks – | Chunks – | Chunks – | Chunks – | Chunks – |
|---|---|---|---|---|---|
| Prompt-Only | |||||
| Policy SFT | |||||
| MemAgent RL ( ) | |||||
| Task-Only RL ( ) | |||||
| MI-Only RL ( ) | |||||
| MaxIM (Full, Ours) |
| Validation Metric | Raw VPL Critic | PMI Critic (Ours) |
|---|---|---|
| arXiv Within-Paper Sufficiency Spread ( , nats ) | ( ) | |
| arXiv Correlation with Downstream Abstain-MCQ ( ) | ( ) | |
| arXiv Hallucination Discrimination AUROC (HQ vs. Halluc. ) | ( pts) | |
| arXiv Grounding Length Correlation ( , closer to ) | ||
| WildChat Sufficiency Pairwise Acc. (HQ Vague / Halluc.) | — | / |
| WildChat Recursive Consistency Pairwise Acc. (HQ Halluc.) | — | ( nats) |
| Method | User Sat. | Goal Comp. | Pref. Ret. | Coherence | Avg. Turns |
| (i) All Test Dialogues ( ) | |||||
| No-Belief ( ) | |||||
| Prompt-Only | ( ) | ( ) | ( ) | ( ) | |
| Policy SFT | ( ) | ( ) | ( ) | ( ) | |
| MemAgent [ 7 ] | ( ) | ( ) | ( ) | ( ) | |
| Task-Only RL | ( ) | ( ) | ( ) | ( ) | |
| Method | Test ( ) | Unseen-Sim OOD ( ) | Turn Extrap. |
|---|---|---|---|
| Prompt-Only | |||
| Policy SFT | |||
| MI-Only RL ( ) | |||
| Suff-Only RL ( ) | |||
| Task-Only RL ( ) | |||
| MemAgent RL ( ) [ 7 ] |
| (a) arXiv Pairwise Summary Comparison ( ; vs. Prompt-Only unless noted) | |||||||
|---|---|---|---|---|---|---|---|
| Method | ( Use % / Cons. ) | ||||||
| Policy SFT | / | ||||||
| MI-Only RL | / | ||||||
| Task-Only RL | / | ||||||
| Suff-Only RL | / | ||||||
| MemAgent RL | / | ||||||
Text Summarization
Text summarization develops methods for condensing documents while preserving the information, perspective, and evidence readers need.
See all 151 papers