cs.ROSep 29, 2026
SaveTMem: Learning Test-Time Memory for Robotics
Organizations: Stanford University · NVIDIA · University of Michigan, Ann Arbor
Abstract
Memory-dependent robotic manipulation requires policies to use information that is no longer available in the current observation. Retaining history alone is insufficient: memory must preserve information that supports future actions. One challenge is whether a memory-free foundation model can learn to retain and use historical information from action demonstrations alone, without external memory support. We introduce TMem, a framework that develops this capability within a pretrained vision-language-action policy, without external reasoning models or memory-specific annotations. TMem uses test-time training to encode observation history into compact fast weights through online self-supervised updates, avoiding repeated processing of the full history. An observation-grounded interface extracts vision-language information for memory formation and supplies retrieved context to the action expert. Action supervision shapes what the memory learns to retain and use, while alternating memory-policy learning gives each component a fixed counterpart during optimization. Across 16 RoboMME tasks, TMem improves average success from 17.93% to 56.83% over the memory-free base policy and outperforms the recurrent-memory methods reported in the benchmark, while controlled profiling indicates at least 3x inference speedup over explicit methods. Project website: https://yzliu84.github.io/T2MEM-project/
Figures & tables
Figure 2: Architecture of T²Mem. (a) Memory connects the VLM and AE alongside the direct vision-language pathway. (b) The observation-grounded interface extracts vision-language features, fuses memory readouts, and supplies attention context to the AE. (c) Self-supervised updates encode history in fast weights.
Figure 3: Alternating memory–policy learning. Stage 0 initializes a pretrained VLA, and Stage 1 adapts the policy without memory. Stage 2A learns memory with the policy fixed; Stage 2B learns a memory-conditioned policy with the memory mechanism fixed. Red arrows indicate repeated alternation; freezing refers to slow parameters, not episode-specific memory updates.
| Method | Counting | Permanence | Reference | Imitation | AVG | |||||||||||||
| Bin Fill | Pick Xtimes | Swing Xtimes | Stop Cube | Video Umsk | Button Umsk | Video UmskS | Button UmskS | Pick HighL | Video Repick | Video PlcBtn | Video PlcOrd | Move Cube | Insert Peg | Pattern Lock | Route Stick | |||
| Human Performance | 96.00 | 100.0 | 80.00 | 78.00 | 90.00 | 92.00 | 92.00 | 90.00 | 92.00 | 92.00 | 98.00 | 90.00 | 90.00 | 98.00 | 84.00 | 86.00 | 90.50 | |
| MME-VLA w/ Symbolic Memory | ||||||||||||||||||
| SimpleSG | Oracle | 85.78 | 99.78 | 100.0 | 44.67 | 33.11 | 22.00 | 15.56 | 15.56 | 44.00 | 27.78 | 31.33 | 26.00 | 87.33 | 10.00 | 95.33 | 55.11 | 49.58 |
| GroundSG | Oracle | 85.78 | 100.0 | 100.0 | 49.67 | 98.78 | 95.00 | 99.22 | 80.22 | 83.33 | 97.33 | 100.0 | 100.0 | 87.78 | 15.56 | 97.00 | 55.56 | 84.08 |
| SimpleSG | Gemini | 46.00 | 63.00 | 45.00 | 2.00 | 29.00 | 9.00 | 14.00 | 2.00 | 20.00 | 15.00 | 26.00 | 29.00 | 61.00 | 4.00 | 7.00 | 0.00 | 23.25 |
Table 1: Evaluation success rates (%) on the 16 RoboMME tasks. Red denotes the highest score, blue the second-best, and green the third-best among non-privileged methods. Our method ranks among the top three on most tasks.
Figure 4: Ablation studies of T²Mem. (a) Memory architecture and (b) training schedule: success rates on VideoUnmask and MoveCube, with a memory-free reference. Mean denotes the average across the two tasks. (c) Adaptive writing: reconstruction NMSE of early-memory K/V bindings over subsequent observations, averaged over 20 VideoUnmask Hard episodes. Lower is better; the label denotes at every write, disabling adaptive scaling but retaining step-size calibration. Training configurations and comparison scope are detailed in Appendix D .
Figure 5: Online memory supports history-dependent decisions. (a) Disabling fast-weight writes reduces success across three tasks. (b) Correct, conflicting, and empty memories produce markedly different outcomes on VideoUnmask. (c) Pre-occlusion writes match full-demonstration performance with half the updates, whereas post-occlusion writes perform substantially worse. Success rates (%) are pooled over three seeds under each panel’s evaluation protocol.
Figure 6: Controlled-workload inference efficiency. Left: latency introduced by memory only. Right: end to end latency for an inference.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Reference configuration |
| VLM / action expert | 18 layers each; widths 2,048 / 1,024 |
| Interface | 16 tokens of width 1,024, initialized from ; rank-16 attention adapters |
| Fast-weight memory | 16 heads; per-head MLP , exact GeLU, biases |
| Learned initialization | : matrix standard deviation 0.02, zero biases |
| Read/write projections | Separate biased Q/K/V projections |
| Normalization / position | Q/K/V RMS normalization ( ); Q/K interleaved RoPE, base |
Table 2: Core architecture settings. RoPE uses interface token positions, not elapsed episode time.
| Stage 1 | Stage 2A | Stage 2B | |
| Purpose | Memory-free adaptation | Memory learning | Memory-conditioned policy learning |
| Initialization | Pretrained | Stage-1 step 20,000; new memory | Preceding memory phase |
| Updated group | VLM LoRA, complete AE, action input/output and time projections | Interface, memory projections and initialization, step multipliers, gates | VLM LoRA, AE, action/time and proprioceptive projections |
| Budget | step 20,000 | 500 updates per cycle | 500 updates per cycle |
| Learning rate | ; gate | ||
| Effective batch | 64 frames (8 accumulation 8) | 32 sequences | 32 sequences |
Table 3: Released training recipe.
| Intervention | Protocol |
| Disable writing | Zero effective updates from reset, including demonstration and execution. Retain , reads, and normal inference-call/RNG cadence. |
| Replace content | Build memory from the target demonstration, a conflicting donor demonstration, or no demonstration. Keep the target environment, instruction, and sampling seed fixed across conditions. Empty means learned , not all-zero weights. |
| Restrict write times | On a 66-frame prefix, permit all writes, only , or only . The write-timing protocol restricts writes to a 66-frame prefix, which truncates longer Hard demonstrations; the full-prefix condition therefore differs from the unrestricted setting in panel (a). |
Table 4: Protocols for the three panels of Figure 5 .
Figure 8: VideoUnmask: recalling information after occlusion. The robot observes colored cubes, then lifts the container hiding the instructed color. In this training trajectory, color–location bindings are visible in frames – and occluded from frame (the 33rd frame). Choosing the correct container therefore requires recalling the early cue. This localized evidence window enables targeted interventions on memory acquisition. Selected frames follow temporal order from left to right, then top to bottom. The blue border marks the video demonstration, during which the policy observes without executing actions.
Figure 9: MoveCube: remembering how an action was demonstrated. The robot watches a demonstration and moves a cube to the target using the same method: pushing, pick-and-place, or tool-mediated hooking. This training example shows hooking in the demonstration and subsequent execution. Because the instruction does not specify the method, the policy must remember how the cube was moved, not merely its destination. Frames are ordered from left to right, then top to bottom. The blue border marks the video demonstration, during which the policy observes without executing actions.
Figure 10: SwingXtimes: remembering progress through repeated actions. The robot moves the red cube between two targets for two right-to-left cycles, then puts it down and presses the stop button. Similar configurations recur across cycles, so the policy must track completed visits to stop after the instructed count. Frames are ordered from left to right, then top to bottom; left and right follow the robot’s coordinate frame.
Figure 11: Counterfactual histories in VideoUnmask. The two rows show different color assignments before occlusion but identical observations afterward. Given the same instruction to select the container hiding the green cube, the correct choice depends on the observed history rather than the current image alone.
Figure 12: Sensitivity to the external planner’s generation budget.