Physical Time as a Missing Dimension of Robot Policy Memory Dynamics

Yushen Liang; Yue Peng; Baosheng Jin; Tianluo Zhang; Xinyu Zhang; Shuyi Zhou; Zhuoran Chen; Xinqi Liu

cs.ROOct 14, 2026

Physical Time as a Missing Dimension of Robot Policy Memory Dynamics

Authors: Yushen Liang, Yue Peng, Baosheng Jin, Tianluo Zhang, Xinyu Zhang, Shuyi Zhou, Zhuoran Chen, Xinqi Liu,

Organizations: New York University Shanghai, Shanghai, China · University of Electronic Science and Technology of China, Chengdu, China

Abstract

Existing robot policy memories typically evolve with observation sequences, implicitly treating each update as the same amount of physical time. Yet identical observation histories can span different durations and require different actions, exposing physical time as a missing dimension of policy memory. We introduce Temporal Flow Policy (TFP), which learns memory dynamics to retain past events and evolve the memory state over measured time intervals. Observations and prior memory determine the content and time scales of each transition; elapsed time determines how far the state advances. TFP conditions diffusion and flow-matching policies through this memory state and is trained with their native imitation objectives. We evaluate TFP with Diffusion Policy and $π_{0.5}$ through controlled comparisons of memory designs. These experiments show that successful timing depends on both measured intervals and retained history. TFP achieves higher average success than the recurrent and state-space memories we equip with the same intervals and than a variant that feeds the interval to a learned gate. Memory readouts recover the time since an event and motion information after occlusion. Changing the supplied intervals systematically delays or advances actions, linking time inputs to behavior. Real-robot tasks requiring timing or memory, together with LIBERO evaluations, demonstrate benefits in physical and general manipulation. These results suggest viewing robot memory not as stored observation history, but as a state that retains past events and evolves with physical time.

Figures & tables

Figure 1: Physical time gives remembered observations meaning for action. (a) The same tea-steeping history and next image require waiting or lifting depending on elapsed time. (b) The same displacement over different intervals implies different velocities, which must be remembered for interception after occlusion. TFP uses measured intervals to evolve memory and guide action. Dashed paths indicate hidden motion; timings, trajectories, and memory states are schematic.

Figure 2: Memory evolution and action generation in TFP. Observation features and previous memory determine candidate content and adaptive time scales. The measured interval controls how far memory moves toward the candidate. The updated state conditions the backbone’s action generator alongside its current context. Repeated previous-memory symbols denote the same state. Retention curves and command traces are schematic.

Figure 3: Task overview. Five real-robot tasks, three controlled simulation tasks, and two LIBERO ( Liu et al., 2023 ) tasks selected for illustration. LIBERO evaluation covers all 40 tasks across the four suites in Table 4 . Appendix F shows demonstration frames.

Diffusion Policy
	Button Press				Passage				Interception
Memory	10 Hz	5 Hz	Jit.	Drop.	10 Hz	5 Hz	Jit.	Drop.	10 Hz	5 Hz	Jit.	Drop.
Reactive	0.0	0.0	0.0	0.0	0.0	0.7	5.3	3.3	8.7	10.3	11.7	11.7
History	0.0	0.0	0.0	0.0	1.3	2.7	2.7	2.0	30.0	34.3	29.0	23.0
GRU	42.7	4.7	28.0	3.3	92.0	4.0	50.0	5.3	40.3	10.3	32.3	18.3
GRU+ $\Delta t$	44.0	5.3	30.0	4.7	90.0	5.3	50.0	6.0	41.3	11.0	33.0	19.0

Table 1: Temporal control across observation schedules. Mean success over three training seeds (%,

\uparrow

). Fixed-

\Delta t

uses

0.1

s at inference. Bold: column maxima by backbone, including ties.

Figure 4: Scaling the supplied clock shifts action timing. Median physical durations under jitter versus supplied interval multiplier

\alpha

(inputs

\alpha\Delta_{n}

); weights and physical query schedules stay fixed. Shading marks the required timing windows. Button measures contact until release; Passage measures first attempted entry since cue onset. Medians use observed events: DP Button

n=20/38/36

of 50 paired trials for

\alpha=0.5/1/2

; all other points

n=50

	Diffusion Policy			$\pi_{0.5}$
Variant	Button	Passage	Intercept.	Button	Passage	Intercept.
Full TFP (ours)	81.2	97.3	53.9	93.5	86.7	39.7
Candidate w/o history	42.7 [-1pt] $(-38.5)$	8.0 [-1pt] $(-89.3)$	44.2 [-1pt] $(-9.7)$	12.0 [-1pt] $(-81.5)$	65.7 [-1pt] $(-21.0)$	31.7 [-1pt] $(-8.0)$
Static time constants	58.5 [-1pt] $(-22.7)$	82.7 [-1pt] $(-14.6)$	44.8 [-1pt] $(-9.1)$	64.3 [-1pt] $(-29.2)$	13.3 [-1pt] $(-73.4)$	31.4 [-1pt] $(-8.3)$
Learned $\Delta t$ gate	14.5 [-1pt] $(-66.7)$	15.0 [-1pt] $(-82.3)$	30.5 [-1pt] $(-23.4)$	27.0 [-1pt] $(-66.5)$	37.0 [-1pt] $(-49.7)$	30.1 [-1pt] $(-9.6)$

Table 2: Which memory components matter? Mean success over three training seeds (%,

\uparrow

), averaged equally over four schedules. Parentheses show changes from Full TFP (pp).

Figure 5: Memory dynamics during Delayed Passage. Observations, channel-wise write gains, and two channel time constants share a physical time axis. Heatmap cells and step traces start at their query timestamps. Shading marks the cue and permitted crossing window. A demonstration is replayed through frozen $\pi_{0.5}$ with TFP; Appendix D.6 details channel and trajectory selection.

Diffusion Policy
Method	Object swapping	Counting pick-place	Water pouring	Tea steeping	Staged heating	Avg.
Reactive	10.0	30.0	20.0	15.0	10.0	17.0
TFP (ours)	60.0	75.0	60.0	65.0	60.0	64.0
Gain (pp)	$+50.0$	$+45.0$	$+40.0$	$+50.0$	$+50.0$	$+47.0$
$\pi_{0.5}$
Reactive	15.0	40.0	30.0	25.0	20.0	26.0

Table 3: Real-robot manipulation on Galaxea A1. Success (%,

\uparrow

); Avg. weights all five tasks equally. Reactive denotes the standard backbone policy without persistent memory. Gains are TFP minus Reactive, in percentage points (pp).

Published results
Policy	LIBERO					LIBERO-plus
	Spatial	Object	Goal	Long-10	Avg.	Noise	Light	Backgr.	Avg.
Reactive $\pi_{0.5}$ (OpenPI)	98.8	98.2	98.0	92.4	96.9	–	–	–	–
HAMLET (3B)	99.0	100.0	99.2	92.2	97.6	–	–	–	–
AVA-VLA ( Xiao et al., 2026 ) (7.3B)	97.4	99.4	97.4	97.6	98.0	78.0	95.5	88.9	87.5
MemoryVLA (7.3B)	98.4	98.4	96.4	93.4	96.7	–	–	–	–

Table 4: General manipulation and visual robustness. Success (%,

\uparrow

); reproduced variants report means over three training seeds. Bold marks column maxima across listed methods, including ties. Gains compare TFP with the corresponding reactive

\pi_{0.5}

baseline (pp).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Diffusion Policy	$\pi_{0.5}$
RGB resolution	$96\times 96$	$224\times 224$
Observation summary $d_{x}$	$2\times 128$ (vision, state)	$3\times 256$ (vision, state, text)
Memory width $d_{h}$	128	256
Input to each memory head	$384=256+128$	$1024=768+256$
Candidate head	Linear + $\tanh$	Linear + RMS norm + $\tanh$
Time-constant head	Linear + softplus + $10^{-6}$	Linear + bounded sigmoid

Appendix

Table 5: TFP architecture and sampling settings for the controlled timing tasks.

Variant	Button	Passage
Full TFP	$\mathbf{81.2\pm 1.2}$	$\mathbf{97.3\pm 1.0}$
$\tau$ : observation	$68.3\pm 1.5$	$88.5\pm 1.0$
$\tau$ : memory	$49.3\pm 1.5$	$94.0\pm 1.0$
$\tau$ : static	$58.5\pm 1.3$	$82.7\pm 1.3$
No memory condition	$0.5\pm 0.5$	$3.7\pm 0.8$
Reset each observation	$0.3\pm 0.6$	$6.2\pm 0.8$

Appendix

Table 6: Additional Diffusion Policy ablations. Success (%,

\uparrow

), reported as mean

\pm

sample SD over three training seeds. Each seed’s result averages the four observation schedules equally.

(a) Fixed time input: $\Delta t\rightarrow 0.1$ s
		Button Press		Passage		Interception
Backbone	Memory	Normal	Fixed	Normal	Fixed	Normal	Fixed
DP	TFP	81.2	25.5	97.3	32.8	53.9	31.9
$\pi_{0.5}$	TFP	93.5	27.3	86.7	35.7	39.7	32.2
(b) Scaled time input: $\Delta t\rightarrow\alpha\Delta t$
Backbone	Memory	$1.0$	$0.5\,/\,2.0$	$1.0$	$0.5\,/\,2.0$	$1.0$	$0.5\,/\,2.0$

Appendix

Table 7: Time-input and memory-reset interventions. Success (%,

\uparrow

). Each intervention uses the same trained weights as its corresponding unmodified policy. Fixed-input results average four query schedules; clock scaling uses jitter; memory resets use 10 Hz.

(a) Event duration (s)
Features	Button	Passage
Current observation	0.3336	1.3073
Episode clock	0.6178	0.8334
TFP memory	0.0140	0.0373

Appendix

Table 8: Event time and motion in memory. Linear-probe MAE (

\downarrow

) from frozen Diffusion Policy.

(a) DP: duration MAE (s) $\downarrow$
Features	Button	Passage
Observation + memory	0.0121	0.0355
Episode clock	0.6178	0.8334

Appendix

Table 9: Additional linear-readout comparisons using frozen policy representations, supplementing Table 8 . MAE and RMSE are reported separately; direction is classification accuracy.

Figure 6: Memory dynamics during Timed Button Press. Simulation frames, write gains, and memory channel 106 are aligned in physical time for a successful sequence. Shading marks contact from 3.09 to 6.33 s. The displayed channel rises during the hold and drops around release.

Figure 7: Memory evolution under occlusion with a stationary robot. The target moves at 0.16 m/s and becomes invisible to the policy at 0.35 s; shading marks the hidden interval. The robot remains stationary throughout this 1.3 s observation sequence. Channels 1 and 115 evolve after occlusion while the visible observation remains unchanged. The Interception task-evaluation protocol is specified in Appendix D.1 .

Figure 8: Memory updates during LIBERO manipulation. Observations (top), channel-wise write gains (middle), and gain summaries (bottom) show changes around grasping, release, and door interaction in a successful rollout. The horizontal axis indexes policy queries (denoted $t$ in the plot labels). Gains depend on both measured intervals and predicted time constants.

	Diffusion Policy			$\pi_{0.5}$
Memory	Button	Passage	Intercept.	Button	Passage	Intercept.
Reactive	$0.0\pm 0.0$	$2.3\pm 0.3$	$10.6\pm 0.1$	$0.0\pm 0.0$	$9.3\pm 0.6$	$18.3\pm 0.3$
History	$0.0\pm 0.0$	$2.2\pm 0.3$	$29.1\pm 0.5$	$5.5\pm 0.0$	$2.8\pm 0.6$	$35.1\pm 0.2$
GRU	$19.7\pm 1.3$	$37.8\pm 0.3$	$25.3\pm 1.5$	$10.5\pm 1.5$	$36.8\pm 1.5$	$27.1\pm 1.5$
GRU+ $\Delta t$	$21.0\pm 0.0$	$37.8\pm 0.3$	$26.1\pm 0.6$	$11.7\pm 0.3$	$37.3\pm 0.3$	$27.8\pm 1.1$
S4D	$25.7\pm 2.1$	$40.5\pm 1.8$	$24.8\pm 1.5$	$32.0\pm 2.0$	$24.7\pm 2.3$	$18.8\pm 1.5$

Appendix

Table 10: Variation across training seeds. Task-average success (%), reported as mean

\pm

sample SD over three independent training seeds.

LTC+ $\Delta t$ : results by observation schedule
		Diffusion Policy		$\pi_{0.5}$
Task	Schedule	Mean $\pm$ SD	95% CI	Mean $\pm$ SD	95% CI
Button	10 Hz	$18.7\pm 3.1$	$[13.2,25.7]$	$23.3\pm 3.1$	$[17.3,30.7]$
	5 Hz	$13.3\pm 3.1$	$[8.8,19.7]$	$18.7\pm 3.1$	$[13.2,25.7]$
	Jitter	$20.7\pm 3.1$	$[15.0,27.8]$	$22.7\pm 3.1$	$[16.7,30.0]$
	Dropout	$11.3\pm 3.1$	$[7.2,17.4]$	$16.0\pm 2.0$	$[11.0,22.7]$

Appendix

Table 11: Variation across training seeds (continued). LTC+

\Delta t

results by observation schedule, with mean

\pm

sample SD and rollout 95% confidence intervals.

LIBERO
Policy	Spatial	Object	Goal	Long-10	Avg.
GRU	$99.0\pm 0.2$	$98.4\pm 0.2$	$95.8\pm 0.3$	$89.1\pm 0.4$	$95.6\pm 0.1$
S4D	$98.5\pm 0.1$	$98.4\pm 0.2$	$97.0\pm 0.2$	$92.5\pm 0.3$	$96.6\pm 0.0$
TFP (ours)	$99.5\pm 0.1$	$99.0\pm 0.2$	$99.3\pm 0.1$	$96.9\pm 0.3$	$98.7\pm 0.2$
TFP 95% CI	$[99.0,99.8]$	$[98.4,99.4]$	$[98.8,99.6]$	$[95.9,97.7]$	–

Appendix

Table 12: LIBERO and LIBERO-plus success (%): mean

\pm

sample SD across three training seeds. The TFP CI row reports rollout 95% confidence intervals on standard LIBERO.

Figure 9: Qualitative task sequences in simulation (a–c) and real-robot demonstrations (d–h, continued on the next page). Frames show action order from left to right. In the Interception expert demonstration (c), the target becomes invisible to the policy before the robot starts moving. Dashed outlines show the hidden target’s true position for visualization only; they are absent from policy inputs. For real-robot tasks, each column pairs Agent view (top) and Wrist camera (bottom), preserving the model’s camera input framing.

Figure 10: Qualitative task sequences (continued). Paired Agent view (top) and Wrist camera (bottom) for Water pouring, Tea steeping, and Staged heating.

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 377 papers

T $^2$ Mem: Learning Test-Time Memory for Robotics
Sep 29, 2026Yize Liu, Huang Huang, Yining Hong +4Robot Policy Learning Test-Time Adaptation of VLA Models
Stanford University · NVIDIA · University of Michigan, Ann Arbor
Test-Time Training as Residual Memory for Robot Policies
Oct 3, 2026Haoxuan Wang, Gengyu Zhang, Ramana Rao Kompella +2Memory-Augmented Agents Robot Policy Learning
University of Illinois Chicago · Cisco Research
Chronos Policy: Learning Robot Policies on Physical Time
Oct 9, 2026Changhao Wang, Jaden Clark, Shuran SongRobot Policy Learning Robot Learning
Stanford University

Sep 29, 2026J/K move · Enter open · S save

T $^2$ Mem: Learning Test-Time Memory for Robotics

Yize Liu, Huang Huang, Yining Hong, Zijian Du+3

Stanford University · NVIDIA · University of Michigan, Ann Arbor

Robot Policy Learning Test-Time Adaptation of VLA Models Policy Learning Memory-Augmented Agents Long-Term Memory for Embodied Agents

Open paper AskPDF arXiv

Memory-dependent robotic manipulation requires policies to use information that is no longer available in the current observation. Retaining history alone is insufficient: memory must preserve information that supports future actions. One challenge is whether a memory-free foundation model can learn to retain and use historical information from action demonstrations alone, without external memory support. We introduce T

^2

Mem, a framework that develops this capability within a pretrained vision-language-action policy, without external reasoning models or memory-specific annotations. T

^2

Mem uses test-time training to encode observation history into compact fast weights through online self-supervised updates, avoiding repeated processing of the full history. An observation-grounded interface extracts vision-language information for memory formation and supplies retrieved context to the action expert. Action supervision shapes what the memory learns to retain and use, while alternating memory-policy learning gives each component a fixed counterpart during optimization. Across 16 RoboMME tasks, T

^2

Mem improves average success from 17.93% to 56.83% over the memory-free base policy and outperforms the recurrent-memory methods reported in the benchmark, while controlled profiling indicates at least 3x inference speedup over explicit methods. Project website: https://yzliu84.github.io/T2MEM-project/

Abstract

Robot Policy Learning

Explore similar work

T2^22Mem: Learning Test-Time Memory for Robotics

Test-Time Training as Residual Memory for Robot Policies

Chronos Policy: Learning Robot Policies on Physical Time

T $^2$ Mem: Learning Test-Time Memory for Robotics