Memory in reinforcement learning raises two problems. Attention finds the relevant item by scoring the whole store, which is costly for robots under latency and energy limits. When a retrieved memory pays off, credit for acquiring it must cross the whole sequence. We argue that both are governed by one quantity, the value of information, and propose to estimate it from archived rollouts with a learned world model. We obtain that, when read at the observation, it yields a provable lower bound on the optimal value of the acquiring action, which floors its TD target and attributes this floor to the observation that earned it. Read at the decision, it defines the optimal retrieval set. A pilot retrieval system, to be trained and improved with this signal, solves a memory task with reduced context.
Figures & tables
Figure 1: Information value on a recorded POMDP trajectory. Decision t needs x1 (at s ) and x2 (at u ) jointly; the look at k−1 would repeat x1 but is not taken, so R=∅ attains the sup in Eq. ( 7 ). Bellman backups carry the reward back step by step; from t , where the information changes the action, the dual arrow jumps back. Along orange it credits the acquiring actions, flooring their TD targets; x1 is credited since x2 is in its base set. Along blue it values recalling s and u jointly.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Baseline
Transformer-XL without retrieval head
Retrieval heads
Memory Navigation
Absolute position, categorical head
Absolute position, Gaussian head
Cue stored in the temporal embedding (no retrieval head, context 5)
Appendix
Table 1: Rollout videos on MiniGrid-Memory-S17Random. Purple denotes attention (scaled in the first panel), the red triangle the agent, and the orange rectangle the Memory Ghost. Panels from left to right: environment state with the agent’s 3×3 observation grid; Transformer activations with temporal and spatial positional encodings; memory states (single: observed, double: visited); token consumption timeline.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.