Context-dependent sequential decision making is commonly addressed either by providing context explicitly as an input or by increasing recurrent memory so that contextual information can be represented internally. We study a third alternative: realizing contextual dependence by intervening on a shared recurrent latent state, without enlarging recurrent dimensionality. To this end, we introduce an intervention-based recurrent architecture in which a recurrent core first constructs a shared pre-intervention latent state, and context then acts through an additive, context-indexed operator. We evaluate this idea on a context-switching sequential decision task under partial observability. We compare three model families: a label-assisted baseline with direct context access, a memory baseline with enlarged recurrent state, and the proposed intervention model, which uses no direct context input to the recurrent core and no memory growth. On the main benchmark, the intervention model performs strongly without additional recurrent dimensions. We also evaluate the models using the conditional mutual information (I(C;O | S)) as a theorem-motivated operational probe of contextual dependence at fixed latent state. For task-relevant phase-1 outcomes, the intervention model exhibits positive conditional contextual information. Together, these results suggest that intervention on a shared recurrent state provides a viable alternative to recurrent memory growth for contextual control in this setting.
Figures & tables
Figure 1: Problem setting of the context-switching sequential decision task. The agent acts in a 9×9 maze and observes a local 3×3 view together with a context token. The task contains two phases within each episode. In the AB order, the target is G1 in phase 0 and G2 in phase 1; in the BA order, the target is G2 in phase 0 and G1 in phase 1. The main benchmark conditions AB25 and BA30. Phase-1 success and reward are counted only after phase-0 success.
Figure 2: Overview of the benchmark conditions and model families. The upper row provides a compact timeline view of the two representative benchmark conditions, AB25 and BA30. The lower row compares the three model families. L directly receives the context token together with the spatial observation. M removes direct context input and instead enlarges the recurrent state from d to d+m . I also removes direct context input from the recurrent core, but applies a context-indexed residual intervention to a shared pre-intervention latent state, yielding zt′=zt+αDct(zt) . This comparison isolates three distinct implementations of contextual dependence: explicit conditioning, memory expansion, and intervention on a shared pre-intervention latent state.
Model
Context access
Extra memory growth
Update rule / interpretation
L
direct label input
no
zt=LSTM(ϕ([xt,ct]),ht−1)
oracle-style reference baseline
M
no direct label
yes
zt=LSTM(ϕ(xt),ht−1),dim(zt)=d+m
memory baseline with enlarged hidden state
I
operator only
no
zt=LSTM(ϕ(xt),ht−1),zt′=zt+αDct(zt)
intervention-based recurrent model
Table 1: Summary of the three model families. xt denotes the local observation, ct the context token, ht−1 the recurrent state, and m the extra memory size used only in M.
Figure 3: Main performance on AB25 and BA30. Bars show the fraction of seeds (out of 10) that solved both phases for each model family. L solves both tasks perfectly. I achieves strong performance without additional recurrent dimensions. Among the memory baselines, performance is non-monotonic in memory size: M16 is strongest on BA30 and among the strongest settings on AB25, while both smaller and larger memory settings underperform it on BA30.
Model
AB25 success
BA30 success
AB25 phase 1
BA30 phase 1
L
10/10
10/10
1.00
1.00
I
9/10
7/10
0.90
0.70
M8
8/10
4/10
0.80
0.40
M16
10/10
9/10
1.00
0.90
M32
10/10
6/10
1.00
0.60
M64
10/10
8/10
1.00
0.80
Table 2: Main performance on the context-switching tasks after 300k training steps. “Success” reports the number of seeds (out of 10) that solved both phases. “Phase 1” reports the mean phase-1 success rate across the same 10 seeds.
Outcome O
Model
AB25
BA30
target_hit
L
0.0372±0.0125
0.0370±0.0078
target_hit
I
0.0341±0.0066
0.0462±0.0194
target_hit
M16
0.0236±0.0019
0.0419±0.0157
goal3
L
0.0410±0.0143
0.0427±0.0104
goal3
I
0.0402±0.0097
0.0550±0.0240
goal3
M16
0.0254±0.0025
0.0440±0.0170
Table 3: Estimated conditional contextual information I(C;O∣S) (bits) in phase 1 under a uniform prior over contexts. We report mean ± standard error across 10 seeds. Goal-related outcomes reveal positive conditional contextual information for all three representative models, including the intervention model without memory growth.
Context engineering decides what information a model carries forward, and current designs meter it in tokens: compressing the past into a bounded recurrent state, keeping a key-value entry for every token, or imposing a fixed budget through a window or eviction rule. All three make the token the unit of memory even when the stream is redundant and the task depends on the distinct information it carries. Building on a companion mechanism paper that opens a cache slot only when an incoming key is novel, so memory scales with the number of distinct items rather than tokens, we develop that allocate-on-novelty cache as a working-memory component and organize context by how a task depends on the past: recall-carried information belongs in a content-addressed novelty cache, summary-carried information in a recurrent state, and locality-carried information in a recency window. The claim is empirical and bounded. On a matched character-level control, novelty-gated attention reaches full-attention performance while attending to about half the tokens, and coupling the cache with a state-space summary matches full-attention coupling at that reduced cost; the advantage grows as context lengthens, while a sliding window is preferable on short, locality-dominated spans. On next-code prediction over synthetic Medicare claims the coupled component leads full attention and every fixed-budget eviction policy at a thousand-event horizon, whereas cost forecasting over the same stream is summary-carried and the cache is neutral. The retained memory is an inspectable table of templates, codes, drugs, or places rather than an opaque state. The experiments are small-scale and use only public data; they establish the primitive that context can scale with distinct information rather than tokens, in a working memory that is content-addressable and auditable.
In-context reinforcement learning (ICRL) enables agents to adapt to new tasks without parameter updates, relying on memory to retain what they learn from experience. For agents operating continually, existing approaches using memory trade off bounded cost against access to past experience. Approaches using growing memories, such as full-history attention and accumulating summaries, incur storage and per-decision costs that increase with task length. Fixed-size memories bound storage but restrict access: sliding windows discard older steps, while recurrent networks compress even recent interactions into a hidden state. We introduce Recurrent Summary Memory (RSM), a Transformer policy that attends exactly to a short working buffer of recent interactions and carries a fixed-capacity summary of earlier experience. When the buffer fills, RSM rewrites the summary from its previous contents and the buffer, then clears the buffer, so deployment memory and per-decision cost stay O(1) in task length. On long-running partially observable tasks, frozen RSM policies sustain their adapted performance for eight times the training horizon with about 1% of full-history memory, ending ahead of every evaluated baseline while growing memories deteriorate. When the task configuration changes mid-run, RSM re-adapts better than growing and recurrent memories. Clearing the summary or blocking the policy's access to it removes its benefit, while swapping in another task's summary steers behaviour towards that task. Together, these results show that our fixed-capacity summary can sustain in-context adaptation far beyond the training horizon.
We propose MATE, a simple yet effective memory architecture for solving Contextual Markov Decision Processes (CMDPs), a family of MDPs parameterized by an unobserved context. In CMDPs, an optimal agent can adapt online by maintaining the posterior belief over contexts. MATE replaces this intractable posterior with a sum-aggregated memory, leveraging the posterior's permutation invariance to retain provably sufficient expressiveness. Compared to prior memory architectures, MATE avoids the growing per-step rollout cost of Transformers and the gradient issues commonly associated with Recurrent Neural Networks (RNNs). Extensive evaluations across diverse benchmarks demonstrate that MATE provides clear computational advantages while achieving performance comparable to standard sequence-model baselines.
Himchan Hwang, Hyeokju Jeong, Gene Chung +3
Seoul National University · Saige · Ulsan National Institute of Science and Technology (UNIST)