A robot's next action can depend on recent motion or earlier visual evidence absent from its current observation. A recurrent visual-memory architecture can retain this history in a visual state, but its usefulness depends on what the state represents and how it is updated with new observations. We introduce Prediction-Regulated Memory (PREM), a framework that uses action-conditioned future prediction to guide both memory learning and updating in vision-language-action policies. During training, predicting future visual features provides temporal supervision for memory learning. During execution, PREM compares current visual features with the prediction made at the preceding decision and uses the spatially normalized prediction discrepancy to regulate updates to the visual state at each spatial location. The updated visual state supplies temporal context for action generation alongside current observations. Built on π0.5, PREM improves average success rate from 9.63% to 20.74% on DOMINO dynamic manipulation tasks and from 14.4% to 83.8% on the RMBench M(1) memory-dependent setting. Further evaluations on LIBERO and LIBERO-Pro show improvements in general manipulation and robustness to spatial perturbations. PREM incurs a mean inference overhead of 7.7 ms on a single NVIDIA H800 GPU.
Figures & tables
Figure 1: Motivation and overview of PREM. (A) Current observations alone can miss recent motion and earlier task-relevant visual evidence. (B) PREM uses future prediction to supervise memory learning and prediction discrepancy to regulate visual-state updating. (C) PREM improves general manipulation, dynamic manipulation, and memory-dependent manipulation.
Figure 2: PREM architecture. (A) PREM compares the cached prediction with observed visual features at each spatial location to compute prediction discrepancy. (B) The spatially normalized discrepancy regulates how much of the candidate visual-state update is incorporated. (C) An action-conditioned predictor predicts the next visual features from the incoming visual state, observed visual features, and latent action representation. The prediction objective supervises memory learning. The cached prediction provides a reference for subsequent visual-state updating.
Table 3: Success rates on the RMBench M(1) setting (%).
Table 7
Figure 4: Prediction discrepancy and visual-state updating. A temporal prediction-discrepancy curve and visual-state update maps are shown for a successful LIBERO rollout. Spatial responses concentrate around the gripper and manipulated object near bowl grasp ( c ) and bowl placement ( f ).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Raw hidden shape
Hact
Hcond
LIBERO
B×10×1024
10
5
DOMINO
B×50×1024
50
16
RMBench
B×50×1024
50
50
Appendix
Table 6: Action-token selection for future prediction. B denotes batch size. Only the first Hcond action tokens are used.
Benchmark
Warm-up
Ramp
K
Stride
LIBERO
2,000
2,000
16
5
DOMINO
10,000
10,000
16
16
RMBench
1,000
2,000
task-specific
50
Appendix
Table 7: Memory-modulation and temporal-training settings. Warm-up and ramp lengths count optimizer steps; K counts replanning observations. Stride is measured in environment steps.
Task
π0.5 (Baseline)
PUMA
PREM (Ours)
Dynamic Interception
Adjust Bottle
52
65
74 (+22)
Dump Bin Bigbin
0
0
37 (+37)
Grab Roller
10
33
26 (+16)
Handover Block
1
17
20 (+19)
Handover Mic
5
35
26 (+21)
Appendix
Table 8: Complete per-task success rates (%) on DOMINO Level-1.
Figure 5: Representative successful rollouts on LIBERO. The rows show LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-10, respectively.
Figure 6: Successful Dynamic Interception rollouts on DOMINO. The five examples visualize Shake Bottle, Handover Mic, Dump Bin into Big Bin, Adjust Bottle, and Move Can to Pot. These tasks require the policy to react to objects or interaction states that evolve during execution.
Figure 7: Successful Dynamic Tracking rollouts on DOMINO. The five examples visualize Place Shoe, Place Bread in Basket, Beat Block Hammer, Click Alarm Clock, and Place Container on Plate. Successful execution requires continuously tracking the task-relevant object as its spatial relationship with the robot changes.
Figure 8: Successful rollouts across the five RMBench M(1) task families. The rows visualize Observe and Pick Up, Put Back Block, Rearrange Blocks, Swap-T, and Swap Blocks. These tasks require retaining object identities, initial spatial configurations, or earlier interaction states that cannot always be recovered from the current observation.