cs.RONov 10, 2026

Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies

Authors: Chubin Zhang, Bo An

Organizations: Nanyang Technological University

Abstract

Memory-augmented vision-language-action (VLA) policies are meant to remember what happened. Many of them, however, fill memory by time: they sample frames uniformly across the episode and stamp each frame with its time step. Because time passes even when nothing happens, such a memory may not tell waiting from working. We show that it does not. On the counting tasks of RoboMME, we make the robot briefly pause or fidget its wrist. Such a null event changes nothing the task depends on, yet a released memory VLA counts the interval as work done and stops early. We call this failure phantom progress. It cuts success from 47 to 16 of 50 on one task and from 44 to 10 on the other. When the same frames are withheld from memory, success returns, so the cause lies in memory rather than in the interruption itself. We trace the failure to a temporal shortcut: because demonstrations proceed at a steady pace, elapsed time predicts progress. Indeed, rescaling the time stamps alone makes the policy stop early when the clock runs fast and repeat when it runs slow. Surprisingly, removing the time encoding does not cure the failure, because the frames of a null event still occupy slots of the time-sampled memory. Across RoboMME's released designs, vulnerability follows how memory is indexed. Of the fixes the diagnosis suggests, training with injected pauses does not remove the failure and impairs counting, whereas indexing memory by events recovers half or more of the loss without retraining. A memory meant to remember what happened should count events rather than clock ticks.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

    Sep 28, 2026Tanguy Dieudonné, Jack B. Jedlicki, Heng YangMemory-Augmented VLMsLong-Horizon Robotic Manipulation

  2. When Plans Change, Remember: Replanning-Guided Visual Memory for Vision-Language-Action Models

    Oct 15, 2026Victoria Smelova, Daniil Kazachkov, Georgiy Kazantsev +6Memory-Augmented VLMsVision-Language-Action Models

  3. ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

    Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Memory-Augmented VLMsLong-Horizon Robotic Manipulation