Memory is essential for long-horizon robotic manipulation, where successful actions may depend on past events that are no longer recoverable from the current observation. As episodes grow longer, however, retaining the full history becomes increasingly costly, creating a fundamental scalability challenge for memory-augmented policies. Existing approaches address this challenge by either storing selected past observations in a bounded memory bank or compressing interaction history into a fixed-size parametric state through Test-Time Training (TTT). Yet these formulations do not explicitly distinguish between historical information that can already be recovered from the policy's current context and information that must persist beyond it. We introduce TTT-RM, which repurposes TTT as Residual Memory, using fast weights not to generically compress history but to complement a bounded memory bank by preserving task-relevant historical information that cannot be recovered from the policy's current context. Concretely, TTT-RM learns a history decoder that reconstructs historical representations from the current observation and retrieved memory. The resulting reconstruction residual captures what this context fails to explain and serves as the learning target for TTT. The TTT slow weights are optimized against this residual target so that online fast-weight updates learn to encode complementary historical information over time. The fast-weight state is then queried to produce a residual memory representation that conditions action generation. Extensive experiments on memory-intensive simulation benchmarks and real-world tasks show that TTT-RM consistently improves across multiple memory designs, outperforms diverse baselines, and supports sustained execution on a three-minute, eight-stage stowing task.
Figures & tables
Figure 1: Comparison of memory paradigms. Incoming observations are represented as tokens along the timeline. (a) Explicit Memory retains selected historical information in a bounded memory bank, giving direct access to only a sparse subset of the preceding history. (b) Test-Time Training instead compresses history into a fixed-size fast-weight state whose size does not grow with episode length. (c) TTT as Residual Memory. TTT is repurposed to preserve only the historical information not recoverable from the current observation and retrieved memory.
Figure 2: Overview of TTT-RM. Frozen visual features are selectively stored in Explicit Memory and written into TTT fast weights through the visual-stream writer. During training, the history decoder estimates what past information is recoverable from the existing policy context, and its reconstruction residual supervises Residual Memory. At inference, the decoder is removed, while Explicit Memory and Residual Memory jointly condition action generation.
Method
Counting
Permanence
Reference
Imitation
AVG
Base ( 9 )
23.3 ± 0.7
17.1 ± 2.9
20.0 ± 2.5
13.3 ± 1.4
18.4 ± 0.8
SAM2Act+ ( 7 )
35.3 ± 1.4
26.0 ± 1.4
16.8 ± 1.4
7.3 ± 0.6
21.4 ± 0.6
MemER ( 22 )
48.8 ± 2.5
53.2 ± 1.4
38.0 ± 2.1
29.5 ± 1.2
42.4 ± 0.9
Test-Time Training ( 6 )
w/ Expert
22.1 ± 6.9
20.8 ± 1.9
20.0 ± 2.5
7.1 ± 0.7
17.5 ± 1.1
w/ Modul
34.5 ± 3.4
22.2 ± 4.1
20.4 ± 1.8
10.7 ± 1.6
21.9 ± 1.5
Table 1: Results on RoboMME. Performance is measured as average success rate (%). Each TTT-RM variant augments the corresponding Explicit Memory baseline with Residual Memory.
Figure 3: Ablation studies. (a) shows the effect of the residual-loss weight λres . Disabling residual supervision ( λres=0 ) can degrade performance, while moderate values yield the strongest results. (b) compares TTT-RM with a naive EM+TTT combination, showing that simply adding TTT to an EM-equipped policy provides limited benefit. (c) examines the effect of our three-stage training pipeline, with the full procedure achieving the best performance across both EM variants.
Figure 4: Real-world results. Qualitative rollouts across the three manipulation tasks are paired with quantitative task progress profiles, which report the fraction of trials reaching or exceeding each progress threshold. Our method consistently achieves better performance across all task stages.
Memory-dependent robotic manipulation requires policies to use information that is no longer available in the current observation. Retaining history alone is insufficient: memory must preserve information that supports future actions. One challenge is whether a memory-free foundation model can learn to retain and use historical information from action demonstrations alone, without external memory support. We introduce T2Mem, a framework that develops this capability within a pretrained vision-language-action policy, without external reasoning models or memory-specific annotations. T2Mem uses test-time training to encode observation history into compact fast weights through online self-supervised updates, avoiding repeated processing of the full history. An observation-grounded interface extracts vision-language information for memory formation and supplies retrieved context to the action expert. Action supervision shapes what the memory learns to retain and use, while alternating memory-policy learning gives each component a fixed counterpart during optimization. Across 16 RoboMME tasks, T2Mem improves average success from 17.93% to 56.83% over the memory-free base policy and outperforms the recurrent-memory methods reported in the benchmark, while controlled profiling indicates at least 3x inference speedup over explicit methods. Project website: https://yzliu84.github.io/T2MEM-project/
Yize Liu, Huang Huang, Yining Hong +4
Stanford University · NVIDIA · University of Michigan, Ann Arbor
Memory-augmented robotic policies are essential in handling memory-dependent tasks. However, existing approaches typically rely on simply extending the observation window, struggling to simultaneously achieve precise task-state tracking and robust long-horizon retention. To overcome these challenges, inspired by the Atkinson--Shiffrin memory model, we propose MemoAct, a hierarchical memory-augmented policy that leverages distinct memory tiers to tackle specific bottlenecks. Specifically, sensory memory filters immediate perceptual inputs, lossless short-term memory supports precise task-state tracking, and compressed long-term memory facilitates robust long-horizon retention. To enrich the evaluation landscape, we construct MemoryRTBench based on RoboTwin 2.0, comprising 6 manipulation tasks that systematically evaluate policy memory capabilities across three dimensions: sequential, spatial, and episodic memory. Extensive experiments across simulated and real-world scenarios demonstrate that MemoAct achieves superior performance compared to both existing Markovian baselines and history-aware policies. The project page is available at https://tlf-tlf.github.io/MemoActPage/.
Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.
Marcel Torne, Karl Pertsch, Homer Walke +14
Physical Intelligence · Stanford University · UC Berkeley +1