Remember Less, Adapt Longer: Bounded-State In-Context Reinforcement Learning with Recurrent Summary Memory

Jiawen Wang; Cheng Xue; Peng Zhang; Jochen Renz

cs.LGOct 19, 2026

Remember Less, Adapt Longer: Bounded-State In-Context Reinforcement Learning with Recurrent Summary Memory

Authors: Jiawen Wang, Cheng Xue, Peng Zhang, Jochen Renz

Organizations: The Australian National University

Abstract

In-context reinforcement learning (ICRL) enables agents to adapt to new tasks without parameter updates, relying on memory to retain what they learn from experience. For agents operating continually, existing approaches using memory trade off bounded cost against access to past experience. Approaches using growing memories, such as full-history attention and accumulating summaries, incur storage and per-decision costs that increase with task length. Fixed-size memories bound storage but restrict access: sliding windows discard older steps, while recurrent networks compress even recent interactions into a hidden state. We introduce Recurrent Summary Memory (RSM), a Transformer policy that attends exactly to a short working buffer of recent interactions and carries a fixed-capacity summary of earlier experience. When the buffer fills, RSM rewrites the summary from its previous contents and the buffer, then clears the buffer, so deployment memory and per-decision cost stay $O(1)$ in task length. On long-running partially observable tasks, frozen RSM policies sustain their adapted performance for eight times the training horizon with about 1% of full-history memory, ending ahead of every evaluated baseline while growing memories deteriorate. When the task configuration changes mid-run, RSM re-adapts better than growing and recurrent memories. Clearing the summary or blocking the policy's access to it removes its benefit, while swapping in another task's summary steers behaviour towards that task. Together, these results show that our fixed-capacity summary can sustain in-context adaptation far beyond the training horizon.

Figures & tables

Figure 1: Recurrent Summary Memory . (a) The stateless encoder

f_{\phi}

maps each public record to

e_{t}

; RSM supplies

m_{t}

to the actor and critics. (b)

X_{b}=(e_{b,1},\ldots,e_{b,n_{b}})

contains the segment’s records. Each RECORD output

m_{b,i}

uses

Z_{b}

and the causal prefix

X_{b,1:i}

; normalised WRITE outputs

W_{b}

yield

Z_{b+1}=P_{\theta}(W_{b})

for the next segment. Parameters are shared across segments; token embeddings are omitted. (c) Successive snapshots of the same bounded deployment state: the summary is replaced and the segment KV cache cleared (crosses). Counters are also retained; full-task training memory grows with task length.

Table 1: Performance at the training horizon and over long tasks (settings in Section 4.1 ). Training horizon: Key-to-Door (KD) doors per 500 steps, CountRecall (CR) exact accuracy, MazeRunner (MR) fraction of goals reached, T-Maze (TM) success. Long-running: doors or goals per 500 steps, in the last 500 of 4,000 steps (same task), per later layout or on the fourth map (new task). CR evicted: accuracy on evidence that left the buffer. State: peak per agent. Bold: best compared memory.

Figure 3: Per-agent state against task length.

	Training horizon		Last 500 of 4,000
	KD	MR	KD	MR
RSM	29.1	0.804	46.0	26.6
window	24.1	0.750	4.9	2.6

Table 2: A state-matched sliding window (the last 40 records per layer, 246.1 KB per agent).

Figure 4: Interventions on the frozen RSM policy. (a–e) The summary kept or cleared at every boundary, beside RSM w/o memory, and in (a,b) the summary or the earlier buffer unreadable to the decisions (

\beta=\infty

); (e) MazeRunner goals in the last 500 of 4,000 steps on one map. (f) Key-to-Door tasks whose next attempt visits the donor’s key first, after a transplant. Dots: seeds.

Panel of Figure 5 — Figure 5: CountRecall retention : accuracy by the number of summary rewrites since the queried evidence was dealt.

	$M=1$	$M=4$ (RSM)	$M=8$	$M=16$
State per agent	209.9 KB	249.9 KB	303.1 KB	409.6 KB
Doors, steps 1–500	29.6 [26.5, 33.0]	29.1 [25.9, 32.4]	30.8 [28.5, 33.1]	30.1 [27.8, 32.4]
Doors, steps 3,501–4,000	45.3 [42.2, 48.2]	46.0 [43.3, 48.6]	45.3 [42.5, 48.1]	45.9 [43.1, 48.7]

Table 3: Summary length on Key-to-Door (4,000-step tasks; 95% intervals).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Memory	Code identifier	Carried forward	Parameters
RSM (ours)	raw_summary	$P_{\theta}(W)$ at each boundary	3.18–3.39M
Full-history Transformer	full_context	Every record’s keys and values	3.11–3.32M
GRU	full_gru	Recurrent hidden state	1.82–2.03M
Memo	memo	Every summary block; jittered segments	3.11–3.32M
Memo, fixed segments	memo_fixed	Every summary block	3.11–3.32M
Sliding window	raw_window	The last $W=40$ records per layer	3.12M

Appendix

Table 4: Compared memories, their code identifiers, what each carries forward and its parameter count over the benchmarks it runs on.

	Key-to-Door	CountRecall	MazeRunner	T-Maze
Optimiser	AdamW; learning rate $10^{-4}$ ; weight decay $10^{-3}$
Warm-up; gradient-norm clip	128 updates; 1.0
Epochs $\times$ steps $\times$ agents	$1{,}000\times 500\times 16=8$ M environment steps
Updates per epoch; batch	128; 16 complete tasks
Replay capacity	10,000 complete tasks
Longest training task (steps)	500	103	500	257

Appendix

Table 5: Training settings, shared by every memory within a benchmark.

Memory	State kept between decisions	Peak, $T=4{,}000$
RSM, RSM-R, RSM w/o memory	Summary, buffer caches and counters (Equation 7 )	249.9 KB
Sliding window	Keys and values of $W$ records per layer, their positions and a length: $8LWd+8W+4$ bytes	246.1 KB
GRU	Hidden state of three layers: $12d$ bytes	3.1 KB
Memo, both variants	Keys and values of four tokens per completed segment and of the open segment	3.24 MB
Full-history Transformer	Keys and values of every record and a length: $8L(T+1)d+4$ bytes	24.58 MB

Appendix

Table 6: Per-agent state of each memory (

L=3

d=256

, FP32).

RSM minus	Mean	95% interval	Seeds	Verdict
Key-to-Door, training horizon ( $\delta=1$ )
Full-history Transformer	$+1.6$	$[-1.3,+4.7]$	$-0.5$ / $+3.8$ / $+1.6$	no consistent difference
GRU	$-0.5$	$[-5.0,+3.3]$	$-4.9$ / $+0.4$ / $+2.8$	no consistent difference
Memo	$-0.5$	$[-2.8,+1.7]$	$+0.5$ / $-0.3$ / $-1.8$	no consistent difference
Memo, fixed segments	$+2.3$	$[-1.4,+5.5]$	$-1.1$ / $+3.9$ / $+4.1$	no consistent difference
Sliding window	$+4.9$	$[+1.4,+9.0]$	$+2.6$ / $+3.1$ / $+9.1$	consistent

Appendix

Table 7: Paired differences at the training horizon , RSM minus each memory on the same tasks and seeds: mean, 95% joint seed/task interval, the value on each seed (42 / 100 / 2026) and the verdict of Appendix E.3 . T-Maze: every memory reaches 1.000 on every seed (practically equivalent under our rule), except RSM w/o memory (

+0.486

[+0.410,+0.536]

; consistent). At the Key-to-Door and CountRecall training horizon, RSM-R differs from RSM by less than

\delta

in mean (Table 2 ), so neither difference is consistent.

RSM minus	Mean	95% interval	Seeds	Verdict
Key-to-Door, last 500 of 4,000 steps ( $\delta=1$ )
Full-history Transformer	$+17.8$	$[+13.1,+21.5]$	$+20.8$ / $+19.8$ / $+12.8$	consistent
GRU	$+5.4$	$[+1.9,+8.5]$	$+1.6$ / $+6.8$ / $+7.7$	consistent
Memo	$+35.4$	$[+27.0,+42.3]$	$+37.9$ / $+26.0$ / $+42.2$	consistent
Memo, fixed segments	$+14.6$	$[+10.0,+18.8]$	$+17.4$ / $+10.0$ / $+16.4$	consistent
Sliding window	$+41.1$	$[+38.6,+43.8]$	$+42.4$ / $+41.6$ / $+39.3$	consistent

Appendix

Table 8: Paired differences on long-running tasks and summary-length ablations , reported as in Table 7 .

Figure 7: Within-task adaptation. (a) Key-to-Door door success by attempt. (b) CountRecall cumulative accuracy; dotted: rewrites after queries 32, 64, 96. (c) Fraction of MazeRunner maps reaching each ordered goal. (d) T-Maze success at corridor 128 over all tasks (95% interval) and by cue side (triangles; faint: seeds); shaded: the success of a fixed turn. Bands in (a–c): RSM and RSM w/o memory.

	Memory kept		Memory wiped
	1–500	3,501–4,000	1–500	3,501–4,000
RSM (ours)	25.0	26.6	21.4	22.2
Full-history Transformer	8.4	0.8	3.6	0.7
GRU	17.6	19.1	14.9	15.7
Memo	19.5	1.8	21.1	22.2
Memo, fixed segments	18.5	2.8	18.2	18.7

Appendix

Table 9: Memory kept or wiped at every attempt on one MazeRunner map: goals in the first and last 500 of 4,000 steps (confirmation maps, three seeds).

Figure 8: Re-adaptation without memory resets (4,000 steps). New Key-to-Door layouts every 500 steps (top); new MazeRunner mazes/goals every 1,000 (bottom). (a,d) Successes per 100 steps; changes dotted. (b,e) Doors per layout/goals per 500 steps. (c,f) Steps to first door (censored at 500)/completed attempt. Band: RSM.

Figure 9: Summary transplants. At one boundary, the summary is replaced by a donor task’s (blue) or the initial summary (dotted). (a) Key-to-Door doors per 50 steps; transplant before step 257. (b) Fraction of tasks whose next attempt visits the donor’s key first. (c) CountRecall accuracy after query 64, scored against true (solid) or donor-consistent (dashed) counts.

Figure 10: Summary representations. (a–c,e–g) First two principal components, fitted per seed on development tasks; colour: key column (Key-to-Door, step 257)/first-category count (CountRecall, query 65). (d,h) Ridge $R^{2}$ ; bars: seed means; dots: seeds. (i,j) Centred cosine similarity of the summaries over repeated MazeRunner attempts on one map; (k) centred norms. Panels (i–k): first 64 confirmation maps.

Figure 11: Summary attention and read interventions. (a,b) Attention to four summary tokens by segment position/layer; grey band: RSM w/o memory (separately trained fixed-READ baseline); dotted: uniform

M/(M+i)

. (c,d) Performance under attention-score bias

-\beta

on summary (solid)/earlier buffer keys (dashed);

\beta=\infty

blocks access; grey dashed: RSM w/o memory.

Figure 12: Carry and rewrite ablations (Section 4.4 ; 500-step windows). (a) Key-to-Door doors. (b) MazeRunner goals on one repeated map; crosses: RSM-R with memory wiped each attempt. (c) Goals on four maps, changed every 1,000 steps. Dotted: training horizon.

Figure 13: Learning at matched experience. Collection return (top)/greedy development performance (bottom) versus training steps for every memory on all four benchmarks. Lines: seed means; band: RSM’s seed range; dotted: development thresholds. For display only, collection curves use a centred 25-epoch window and development curves a centred three-evaluation window, both with edge padding; threshold crossings use the raw series.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Downstream-Aware Context Selection for Online In-Context Reinforcement Learning
Sep 27, 2026Ruihan A. Li, Shangtong Zhang, Rohan ChandraIn-Context RL In-Context Learning
University of Illinois Urbana-Champaign · University of Virginia
Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation
Sep 28, 2026Yuanqing Ma, Zhenrui Zheng, Chenjun XiaoTransformer In-Context RL
The Chinese University of Hong Kong, Shenzhen
Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory
Mar 4, 2026Zhenting Wang, Huancheng Chen, Jiayun Wang +1LLM Agent Memory Memory Management for LLM Agents
Center for Advanced AI, Accenture

Sep 27, 2026J/K move · Enter open · S save

Downstream-Aware Context Selection for Online In-Context Reinforcement Learning

Ruihan A. Li, Shangtong Zhang, Rohan Chandra

University of Illinois Urbana-Champaign · University of Virginia

In-Context RL In-Context Learning Contextual RL Context Compression

Open paper AskPDF arXiv

In-context reinforcement learning (ICRL) enables large language model agents to adapt to new environments using their interaction history without updating model parameters. However, repeatedly conditioning on growing histories can lead to substantial token cost. We propose a bounded-history context-management framework that predicts the task-dependent downstream effect of removing historical interactions to guide history selection and determine a decision-dependent context budget. Formally, our framework uses the full rolling history as a reference. The predictor evaluates removal effects, defines a deletion ordering, and applies a shared selection criterion to determine how much history to retain at each decision. We evaluate the method in closed-loop SUMO driving under held-out in-distribution, unseen-domain, and unseen-route settings, and in ScienceWorld under a continual ICRL protocol. Relative to a baseline using the full context, our method reduces total token usage by 25.7%, 25.8%, and 23.2% across the three driving settings while maintaining comparable closed-loop driving performance. In ScienceWorld, it reduces total token usage by 52.1% compared to full context and uses 30.2% and 37.8% fewer tokens than the Recent and Similarity baselines, respectively, while maintaining performance.

Abstract

Reinforcement Learning

Explore similar work

Downstream-Aware Context Selection for Online In-Context Reinforcement Learning

Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation

Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory