In-context reinforcement learning (ICRL) enables agents to adapt to new tasks without parameter updates, relying on memory to retain what they learn from experience. For agents operating continually, existing approaches using memory trade off bounded cost against access to past experience. Approaches using growing memories, such as full-history attention and accumulating summaries, incur storage and per-decision costs that increase with task length. Fixed-size memories bound storage but restrict access: sliding windows discard older steps, while recurrent networks compress even recent interactions into a hidden state. We introduce Recurrent Summary Memory (RSM), a Transformer policy that attends exactly to a short working buffer of recent interactions and carries a fixed-capacity summary of earlier experience. When the buffer fills, RSM rewrites the summary from its previous contents and the buffer, then clears the buffer, so deployment memory and per-decision cost stay O(1) in task length. On long-running partially observable tasks, frozen RSM policies sustain their adapted performance for eight times the training horizon with about 1% of full-history memory, ending ahead of every evaluated baseline while growing memories deteriorate. When the task configuration changes mid-run, RSM re-adapts better than growing and recurrent memories. Clearing the summary or blocking the policy's access to it removes its benefit, while swapping in another task's summary steers behaviour towards that task. Together, these results show that our fixed-capacity summary can sustain in-context adaptation far beyond the training horizon.
Figures & tables
Figure 1: Recurrent Summary Memory . (a) The stateless encoder fϕ maps each public record to et ; RSM supplies mt to the actor and critics. (b) Xb=(eb,1,…,eb,nb) contains the segment’s records. Each RECORD output mb,i uses Zb and the causal prefix Xb,1:i ; normalised WRITE outputs Wb yield Zb+1=Pθ(Wb) for the next segment. Parameters are shared across segments; token embeddings are omitted. (c) Successive snapshots of the same bounded deployment state: the summary is replaced and the segment KV cache cleared (crosses). Counters are also retained; full-task training memory grows with task length.
Table 1: Performance at the training horizon and over long tasks (settings in Section 4.1 ). Training horizon: Key-to-Door (KD) doors per 500 steps, CountRecall (CR) exact accuracy, MazeRunner (MR) fraction of goals reached, T-Maze (TM) success. Long-running: doors or goals per 500 steps, in the last 500 of 4,000 steps (same task), per later layout or on the fourth map (new task). CR evicted: accuracy on evidence that left the buffer. State: peak per agent. Bold: best compared memory.
Figure 3: Per-agent state against task length.
Training horizon
Last 500 of 4,000
KD
MR
KD
MR
RSM
29.1
0.804
46.0
26.6
window
24.1
0.750
4.9
2.6
Table 2: A state-matched sliding window (the last 40 records per layer, 246.1 KB per agent).
Figure 4: Interventions on the frozen RSM policy. (a–e) The summary kept or cleared at every boundary, beside RSM w/o memory, and in (a,b) the summary or the earlier buffer unreadable to the decisions ( β=∞ ); (e) MazeRunner goals in the last 500 of 4,000 steps on one map. (f) Key-to-Door tasks whose next attempt visits the donor’s key first, after a transplant. Dots: seeds.
Figure 5: CountRecall retention : accuracy by the number of summary rewrites since the queried evidence was dealt.
M=1
M=4 (RSM)
M=8
M=16
State per agent
209.9 KB
249.9 KB
303.1 KB
409.6 KB
Doors, steps 1–500
29.6 [26.5, 33.0]
29.1 [25.9, 32.4]
30.8 [28.5, 33.1]
30.1 [27.8, 32.4]
Doors, steps 3,501–4,000
45.3 [42.2, 48.2]
46.0 [43.3, 48.6]
45.3 [42.5, 48.1]
45.9 [43.1, 48.7]
Table 3: Summary length on Key-to-Door (4,000-step tasks; 95% intervals).
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Memory
Code identifier
Carried forward
Parameters
RSM (ours)
raw_summary
Pθ(W) at each boundary
3.18–3.39M
Full-history Transformer
full_context
Every record’s keys and values
3.11–3.32M
GRU
full_gru
Recurrent hidden state
1.82–2.03M
Memo
memo
Every summary block; jittered segments
3.11–3.32M
Memo, fixed segments
memo_fixed
Every summary block
3.11–3.32M
Sliding window
raw_window
The last W=40 records per layer
3.12M
Appendix
Table 4: Compared memories, their code identifiers, what each carries forward and its parameter count over the benchmarks it runs on.
Key-to-Door
CountRecall
MazeRunner
T-Maze
Optimiser
AdamW; learning rate 10−4 ; weight decay 10−3
Warm-up; gradient-norm clip
128 updates; 1.0
Epochs × steps × agents
1,000×500×16=8 M environment steps
Updates per epoch; batch
128; 16 complete tasks
Replay capacity
10,000 complete tasks
Longest training task (steps)
500
103
500
257
Appendix
Table 5: Training settings, shared by every memory within a benchmark.
Memory
State kept between decisions
Peak, T=4,000
RSM, RSM-R, RSM w/o memory
Summary, buffer caches and counters (Equation 7 )
249.9 KB
Sliding window
Keys and values of W records per layer, their positions and a length: 8LWd+8W+4 bytes
246.1 KB
GRU
Hidden state of three layers: 12d bytes
3.1 KB
Memo, both variants
Keys and values of four tokens per completed segment and of the open segment
3.24 MB
Full-history Transformer
Keys and values of every record and a length: 8L(T+1)d+4 bytes
24.58 MB
Appendix
Table 6: Per-agent state of each memory ( L=3 , d=256 , FP32).
RSM minus
Mean
95% interval
Seeds
Verdict
Key-to-Door, training horizon ( δ=1 )
Full-history Transformer
+1.6
[−1.3,+4.7]
−0.5 / +3.8 / +1.6
no consistent difference
GRU
−0.5
[−5.0,+3.3]
−4.9 / +0.4 / +2.8
no consistent difference
Memo
−0.5
[−2.8,+1.7]
+0.5 / −0.3 / −1.8
no consistent difference
Memo, fixed segments
+2.3
[−1.4,+5.5]
−1.1 / +3.9 / +4.1
no consistent difference
Sliding window
+4.9
[+1.4,+9.0]
+2.6 / +3.1 / +9.1
consistent
Appendix
Table 7: Paired differences at the training horizon , RSM minus each memory on the same tasks and seeds: mean, 95% joint seed/task interval, the value on each seed (42 / 100 / 2026) and the verdict of Appendix E.3 . T-Maze: every memory reaches 1.000 on every seed (practically equivalent under our rule), except RSM w/o memory ( +0.486[+0.410,+0.536] ; consistent). At the Key-to-Door and CountRecall training horizon, RSM-R differs from RSM by less than δ in mean (Table 2 ), so neither difference is consistent.
RSM minus
Mean
95% interval
Seeds
Verdict
Key-to-Door, last 500 of 4,000 steps ( δ=1 )
Full-history Transformer
+17.8
[+13.1,+21.5]
+20.8 / +19.8 / +12.8
consistent
GRU
+5.4
[+1.9,+8.5]
+1.6 / +6.8 / +7.7
consistent
Memo
+35.4
[+27.0,+42.3]
+37.9 / +26.0 / +42.2
consistent
Memo, fixed segments
+14.6
[+10.0,+18.8]
+17.4 / +10.0 / +16.4
consistent
Sliding window
+41.1
[+38.6,+43.8]
+42.4 / +41.6 / +39.3
consistent
Appendix
Table 8: Paired differences on long-running tasks and summary-length ablations , reported as in Table 7 .
Figure 7: Within-task adaptation. (a) Key-to-Door door success by attempt. (b) CountRecall cumulative accuracy; dotted: rewrites after queries 32, 64, 96. (c) Fraction of MazeRunner maps reaching each ordered goal. (d) T-Maze success at corridor 128 over all tasks (95% interval) and by cue side (triangles; faint: seeds); shaded: the success of a fixed turn. Bands in (a–c): RSM and RSM w/o memory.
Memory kept
Memory wiped
1–500
3,501–4,000
1–500
3,501–4,000
RSM (ours)
25.0
26.6
21.4
22.2
Full-history Transformer
8.4
0.8
3.6
0.7
GRU
17.6
19.1
14.9
15.7
Memo
19.5
1.8
21.1
22.2
Memo, fixed segments
18.5
2.8
18.2
18.7
Appendix
Table 9: Memory kept or wiped at every attempt on one MazeRunner map: goals in the first and last 500 of 4,000 steps (confirmation maps, three seeds).
Figure 8: Re-adaptation without memory resets (4,000 steps). New Key-to-Door layouts every 500 steps (top); new MazeRunner mazes/goals every 1,000 (bottom). (a,d) Successes per 100 steps; changes dotted. (b,e) Doors per layout/goals per 500 steps. (c,f) Steps to first door (censored at 500)/completed attempt. Band: RSM.
Figure 9: Summary transplants. At one boundary, the summary is replaced by a donor task’s (blue) or the initial summary (dotted). (a) Key-to-Door doors per 50 steps; transplant before step 257. (b) Fraction of tasks whose next attempt visits the donor’s key first. (c) CountRecall accuracy after query 64, scored against true (solid) or donor-consistent (dashed) counts.
Figure 10: Summary representations. (a–c,e–g) First two principal components, fitted per seed on development tasks; colour: key column (Key-to-Door, step 257)/first-category count (CountRecall, query 65). (d,h) Ridge R2 ; bars: seed means; dots: seeds. (i,j) Centred cosine similarity of the summaries over repeated MazeRunner attempts on one map; (k) centred norms. Panels (i–k): first 64 confirmation maps.
Figure 11: Summary attention and read interventions. (a,b) Attention to four summary tokens by segment position/layer; grey band: RSM w/o memory (separately trained fixed-READ baseline); dotted: uniform M/(M+i) . (c,d) Performance under attention-score bias −β on summary (solid)/earlier buffer keys (dashed); β=∞ blocks access; grey dashed: RSM w/o memory.
Figure 12: Carry and rewrite ablations (Section 4.4 ; 500-step windows). (a) Key-to-Door doors. (b) MazeRunner goals on one repeated map; crosses: RSM-R with memory wiped each attempt. (c) Goals on four maps, changed every 1,000 steps. Dotted: training horizon.
Figure 13: Learning at matched experience. Collection return (top)/greedy development performance (bottom) versus training steps for every memory on all four benchmarks. Lines: seed means; band: RSM’s seed range; dotted: development thresholds. For display only, collection curves use a centred 25-epoch window and development curves a centred three-evaluation window, both with edge padding; threshold crossings use the raw series.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
In-context reinforcement learning (ICRL) enables large language model agents to adapt to new environments using their interaction history without updating model parameters. However, repeatedly conditioning on growing histories can lead to substantial token cost. We propose a bounded-history context-management framework that predicts the task-dependent downstream effect of removing historical interactions to guide history selection and determine a decision-dependent context budget. Formally, our framework uses the full rolling history as a reference. The predictor evaluates removal effects, defines a deletion ordering, and applies a shared selection criterion to determine how much history to retain at each decision. We evaluate the method in closed-loop SUMO driving under held-out in-distribution, unseen-domain, and unseen-route settings, and in ScienceWorld under a continual ICRL protocol. Relative to a baseline using the full context, our method reduces total token usage by 25.7%, 25.8%, and 23.2% across the three driving settings while maintaining comparable closed-loop driving performance. In ScienceWorld, it reduces total token usage by 52.1% compared to full context and uses 30.2% and 37.8% fewer tokens than the Recent and Similarity baselines, respectively, while maintaining performance.
Ruihan A. Li, Shangtong Zhang, Rohan Chandra
University of Illinois Urbana-Champaign · University of Virginia
Algorithm Distillation (AD) has demonstrated the remarkable ability of Transformers to perform in-context reinforcement learning without explicit weight updates. However, capturing long-term learning progress necessitates expansive context windows, which incur prohibitive memory costs and limit scalability in complex, long-horizon tasks. To address this bottleneck, we propose Recurrent Algorithm Distillation (RAD). RAD employs a dual-component architecture: a Compression Transformer that distills extended interaction histories into compact latent tokens, and an AD Transformer that auto-regressively generates actions using a hybrid context of these compressed memories and recent transitions. By maintaining a fixed-size latent buffer, RAD decouples the effective history length from computational complexity, functionally providing the model with a long-horizon memory. Empirical evaluations across diverse environments demonstrate that RAD matches the asymptotic performance of standard AD with significantly reduced context window sizes, offering a scalable solution for efficient in-context decision-making.
Large language model (LLM) agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence. Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.