Remembering What Matters for Long-Horizon Robotic World Models
Authors: Hao Wu, Tianyu Shao, Shengju Qian, Yizhou Zhao, Fan Zhang, Penghao Zhao, Weiyan Wang, Yufei Huang, Fan Xu, Kun Wang, Xin Wang, Ming-Hsuan Yang, Yuxuan Liang
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · North China Electric Power University · LIGHTSPEED · Carnegie Mellon University · The Chinese University of Hong Kong · Tencent · University of Science and Technology of China · Nanyang Technological University · University of California, Merced
Autoregressive robot video world models based on discrete visual tokens remain unreliable over long rollout horizons. Retaining the entire generated history does not necessarily improve prediction quality. We investigate a possible memory-role conflict in Full Context decoding: persistent scene information, recent visual-action states, and stale generated history compete for attention despite their different predictive roles. This perspective motivates ReCAP (Receding Context with Anchor Preservation), a training-free context-scheduling method for long-horizon robot video world models. ReCAP keeps the pretrained tokenizer, world model, and visual decoder frozen. It preserves the initial scene anchor and a contiguous window of recent complete visual-action blocks while evicting older generated history. The schedule preserves within-block structure and temporal order, requires no additional modules or parameter updates, and bounds active-context growth independently of rollout length. We evaluate ReCAP on RT-1, BridgeV2, LIBERO-90, and CALVIN using frozen autoregressive world models. Across evaluated environments and horizons, ReCAP improves visual fidelity, action-conditioned consistency, and robot structure preservation relative to Full Context. On BridgeV2, it reduces LPIPS by 63.3%. In the system benchmark at H=64, it reduces the logical KV footprint by 73.3% and p50 latency by 6.51%. Attention analyses show increased attention to retained recent states, while history interventions associate larger amounts of stale context with higher prediction error. These observations are consistent with the memory-role-conflict hypothesis, although they do not isolate it from context-length and position-handling effects. Our results support inference-time context scheduling as a practical approach to improving long-horizon robot video prediction and computational efficiency without retraining.
Figures & tables
Figure 1: ReCAP improves long-horizon prediction and memory efficiency. (a) ReCAP preserves attention to recent visual states. (b) It improves prediction quality on BridgeV2. (c) Three VLM judges score ReCAP higher across semantic criteria. (d) It reduces KV-cache and GPU-memory usage. Panels use dedicated samples and extrapolation horizons and are therefore not directly comparable to the fixed-horizon evaluations in Section 5 .
Figure 2: Overview of ReCAP . Full Context retains all generated states, with growing context cost and increasing prediction drift in the evaluated rollouts. ReCAP groups history into complete visual–action blocks, preserves the initial scene anchor and the most recent blocks, and evicts stale middle history. This training-free schedule keeps the active KV cache bounded while maintaining stable autoregressive rollouts.
Figure 3: Qualitative case study. ReCAP preserves the robot arm throughout the rollout, whereas Full Context collapses; the blind VLM judge prefers ReCAP . The displayed A/B prompt differs from the per-video RT-1 protocol in Appendix D .
Figure 4: ReCAP improves long-horizon robustness. (a–c) LPIPS across prediction horizons on three datasets; shaded bands denote 95% confidence intervals. (d) Paired LPIPS AUC gains on independent BridgeV2 and RT-1 evaluation sets, where positive values favor ReCAP .
Figure 5: Qualitative comparisons across four datasets. Selected cases from Appendix C , with three prediction times per dataset: t=16,32,64 for RT-1, CALVIN, and LIBERO-90; t=8,16,32 for the shorter BridgeV2 protocol. Rows show ground truth, Full Context, and ReCAP .
Figure 6: Mechanism analysis of ReCAP . (a) Component ablation with 95% bootstrap confidence intervals. (b) H=64 LPIPS after reinserting k stale blocks. (c) Layer-averaged semantic attention and (d) recent-state attention by layer on CALVIN C06 at H=64 .
Metric
Full
ReCAP
Change
Tokens ↓
7,232
1,931
−73.3%
Logical KV (MiB) ↓
254.25
67.89
−73.3%
p50 (ms) ↓
312.95
292.57
−6.51%
C=256 throughput ↑
1,596
6,834
4.28×
Table 3: System efficiency summary. H=64 values are macro averages; throughput uses C=256 .
Figure 7: ReCAP bounds context cost and improves concurrent throughput. (a) Prompt tokens, (b) logical BF16 KV, (c) p50 world-model latency, and (d) concurrent throughput. Shading marks Full Context truncation; bands in (c) show the range across three datasets.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Token organization for the 1,931-token ReCAP configuration. (a) The fixed prefix comprises 1,280 scene-context tokens, 80 initial dynamics tokens, and 13 supplied action tokens, totaling 1,373 tokens. Each context-conditioned dynamics representation uses one-sixteenth as many visual tokens as the context representation. (b) Six recent complete visual–action blocks contribute 558 tokens, giving a 1,931-token input before next-frame generation. The next 80 visual tokens are outputs and are excluded from this input budget. (c) Before a further prediction, the generated visual tokens are paired with the next supplied action to complete a 93-token block, and the oldest recent block is evicted. Here, action ui conditions the transition from frame i to frame i+1 . Segment widths are schematic.
Component
Version
Python
3.13.11
PyTorch / torchvision
2.9.1 / 0.24.1+d801a34
CUDA (PyTorch build) / cuDNN
12.9 / 9.1.6.0
NCCL
2.27.3
Transformers
4.57.1; 4.57.1.post1 (metrics)
vLLM
0.13.0
Appendix
Table 4: Software environment used for model execution and evaluation. Versions marked “metrics” refer to the separate metric-evaluation environment.
Figure 9: CALVIN tokenizer adaptation. (a) Reconstruction PSNR, LPIPS, and dynamics-code utilization measured on the same fixed set of 32 validation clips (256 frames) for the released RT-1 tokenizer and the 5k, 10k, and 20k checkpoints. (b) Same-frame reconstructions at t∈{0,2,4,7} . The adapted tokenizer progressively removes the color shift and object disappearance observed with the unadapted RT-1 tokenizer.
Figure 10: CALVIN world-model adaptation. (a) Raw training loss and its nine-point moving average. (b) Validation negative log-likelihood evaluated every 1,000 updates; the star marks the minimum near 19k and the square marks the pre-specified final 20k checkpoint. (c) Validation perplexity. (d) A 16-frame action-conditioned open-loop rollout on held-out CALVIN validation data. The model receives the first frame and ground-truth actions, then autoregressively feeds back predicted visual tokens. It achieves 21.18 dB mean future PSNR, compared with 16.43 dB for repeating the first frame.
Setting
Tokenizer adaptation
World-model adaptation
Initialization
RLVR-World RT-1 tokenizer
RLVR-World RT-1 world model
Trainable component
Compressive tokenizer
Autoregressive transformer
Training updates
20,000
20,000
Segment / context frames
8 / 1
8 / 1
Global batch size
8
8
Optimizer
AdamW
AdamW
Appendix
Table 5: CALVIN adaptation and training hyperparameters. Batch size denotes the global batch across eight H20 GPUs.
Figure 11: LIBERO-90 tokenizer adaptation. (a) Reconstruction PSNR, LPIPS, and dynamics-code utilization for the released RT-1 tokenizer and the final 20k LIBERO tokenizer, evaluated on the identical fixed set of 32 held-out clips (256 frames). (b) Same-frame reconstructions at t∈{0,2,4,7} . Adaptation substantially reduces color shift and preserves robot and object structure.
Figure 12: LIBERO-90 world-model adaptation. (a) Raw training loss and its nine-point moving average. (b) Validation negative log-likelihood evaluated every 1,000 updates; the star marks the minimum near 17k and the square marks the pre-specified final 20k checkpoint. (c) Validation perplexity. (d) A 16-frame action-conditioned visual open-loop rollout on a held-out high-motion LIBERO-90 episode. The model observes the first frame, receives ground-truth future actions, and autoregressively feeds back its visual predictions.
Setting
Tokenizer adaptation
World-model adaptation
Initialization
RLVR-World RT-1 tokenizer
RLVR-World RT-1 world model
Trainable component
Compressive tokenizer
Autoregressive transformer
Training updates
20,000
20,000
Segment / context frames
8 / 1
8 / 1
Global batch size
8
8
Optimizer
AdamW
AdamW
Appendix
Table 6: LIBERO-90 adaptation and training hyperparameters. Batch size denotes the global batch across eight H20 GPUs.
Figure 13: BridgeV2 tokenizer training. (a) Total generator loss; (b–c) context and dynamics reconstruction and perceptual losses; (d–e) online validation reconstruction and perceptual losses; (f) discriminator hinge loss plus gradient penalty (logarithmic vertical axis). Dashed lines mark adversarial training activation at shared step 10k. Faint training traces show raw logged values; solid training traces use a nine-point moving average. Validation points are unsmoothed and end at step 59,001. The shared counter includes alternating generator/discriminator updates after GAN activation.
Figure 14: BridgeV2 world-model training. (a) Raw causal cross-entropy and its nine-point moving average over 100k transformer updates. (b) Validation negative log-likelihood and (c) validation perplexity. Stars indicate the lowest monitored validation loss near 70k; squares mark the final 100k checkpoint used for the rollout comparisons. All validation points are plotted without smoothing, including the final evaluation at 100,000.
Setting
Tokenizer adaptation
World-model adaptation
Initialization
RT-1 tokenizer
RT-1 world model
Checkpoint counter
60,000 (shared)
100,000
Segment / context frames
8 / 1
8 / 1
Per-device batch / accumulation
1 / 4
1 / 1
Global batch per optimizer update
32
8
Optimizer / weight decay
AdamW / 0
AdamW / 0
Appendix
Table 7: BridgeV2 adaptation hyperparameters. Tokenizer steps use the shared generator/discriminator counter; world-model steps are transformer updates.
Setting
Value
Tokenizer / world-model checkpoint
60k / 100k
World-model parameters
127.1M
Training GPUs / global batch / segment
8 H20 / 8 / 8 frames
World-model learning rate / precision
10−5 / BF16
Observed / predicted frames
1 / 32
Inference precision
FP32, TF32 disabled
Appendix
Table 8: BridgeV2 settings for the additional H=32 comparisons.
Figure 15: Additional RT-1 qualitative comparisons. Nine time points are shown for each rollout. Across these three instruction-conditioned examples, ReCAP improves mean PSNR over Full Context by +2.41 , +2.12 , and +1.54 dB, respectively.
Figure 16: Additional CALVIN validation comparisons. The examples are drawn from annotated validation trajectories. ReCAP improves mean PSNR over Full Context by +5.23 , +4.75 , and +4.61 dB, respectively.
Figure 17: Additional LIBERO-90 held-out-task comparisons. The three examples use distinct language instructions from the task-disjoint validation split. ReCAP improves mean PSNR over Full Context by +6.04 , +3.92 , and +2.61 dB, respectively. Frames are vertically flipped for canonical display orientation only.
Figure 18: Additional BridgeV2 validation comparisons. Three high-gain examples selected from a fixed pool of 24 candidates, with nine displayed time points ( t=0,4,8,12,16,20,24,28,32 ) over 32 predicted frames. Rows show ground truth, Full Context, and ReCAP . Mean future PSNR improves by +0.78 , +4.02 , and +2.30 dB; mean LPIPS decreases by 26.7% , 32.5% , and 23.3% , respectively. Metrics use all 32 predictions, excluding t=0 . Task text identifies the episodes and is not a model input.
Score
Interaction quality
Instruction following
Task progress
Motion naturalness
1–2
Penetration or absent proper contact
Wrong action, object, or scene
No visible progress
Severe jumps or jitter; or a frozen arm
3–4
Contact with implausible physical response
Related behavior with major errors
Motion without meaningful progress
Frequent irregular motion; or tiny, aimless movements
5–6
Mostly plausible, with minor interaction errors
General intent followed with execution errors
Partial completion
Meaningful motion with occasional stutter or kinematic issues
7–8
Realistic contact and object response
Mostly correct, with minor deviations
Mostly completed
Meaningful, smooth motion with minor imperfections
9–10
Visually indistinguishable from real interaction
All instructed elements correctly executed
Fully completed
Meaningful, smooth, physically plausible motion
Appendix
Table 9: Shared VLM evaluation rubric. Score bands summarize the prompt; the mandatory score caps also apply. Higher values indicate better observed behavior.
Figure 19: Human–VLM alignment under two judging protocols. Bars show pooled fractions on the same 80 pairs: (a) three-way agreement with human-majority labels and (b) tie rates. Gray denotes the original protocol and blue the combined revision. The dashed line in (b) marks the human tie rate of 17.5%. Judge order is fixed across panels.
Protocol
Evaluator
ReCAP wins
Ties
Full Context wins
Agreement with humans
κ
Reference
Human majority
42
14
24
–
–
Original
Qwen3-VL
47
8
25
55/80 (68.8%)
0.46
InternVL3
45
10
25
53/80 (66.3%)
0.43
GPT-5.5
49
7
24
58/80 (72.5%)
0.52
Combined
Qwen3-VL
43
14
23
60/80 (75.0%)
0.58
InternVL3
42
15
23
58/80 (72.5%)
0.55
Appendix
Table 10: Human–VLM agreement on 80 paired rollouts. Agreement counts all three labels, including ties. Unweighted Cohen’s κ is calculated from the pooled preference counts and observed agreement. The combined revision is described in Appendix D.7 .
Protocol
Qwen3-VL
InternVL3
GPT-5.5
Original
47 / 8 / 25
45 / 10 / 25
49 / 7 / 24
Explicitly allow equal scores
43 / 14 / 23
42 / 15 / 23
45 / 12 / 23
Half-strength static penalty
45 / 11 / 24
43 / 13 / 24
47 / 10 / 23
Remove static penalty
42 / 13 / 25
40 / 15 / 25
44 / 12 / 24
Include first and last frames
46 / 11 / 23
44 / 13 / 23
48 / 9 / 23
Combined revision
43 / 14 / 23
42 / 15 / 23
45 / 12 / 23
Appendix
Table 11: Sensitivity of VLM preferences to the scoring protocol. Each cell gives ReCAP wins / ties / Full Context wins and sums to 80. The original protocol remains the basis of the main VLM results.
Scene prefix (tokens)
Initial block (tokens)
Recent blocks
Recent tokens
Effective tokens
Unused budget
Late LPIPS ↓
nDTW ↑
4
93
19
1,767
1,864
67
0.208±0.003
0.619±0.006
16
93
19
1,767
1,876
55
0.207±0.003
0.621±0.006
64
93
19
1,767
1,924
7
0.205±0.003
0.624±0.005
1,280
93
6
558
1,931
0
0.203±0.002
0.628±0.005
Appendix
Table 12: Scene-prefix allocation under a 1,931-token budget cap. Results on 450 held-out LIBERO-90 episodes with five generation seeds are reported as mean ± standard deviation. Token counts describe the steady-state prompt; every row includes the same initial 93-token block. The full scene anchor is highlighted.
Comparison
Δ Late LPIPS ↓
95% CI
Δ nDTW ↑
95% CI
Full anchor − 4 tokens
−0.005
[−0.007,−0.003]
+0.009
[+0.005,+0.013]
Full anchor − 16 tokens
−0.004
[−0.006,−0.002]
+0.007
[+0.003,+0.011]
Full anchor − 64 tokens
−0.002
[−0.004,−0.001]
+0.004
[+0.001,+0.007]
Appendix
Table 13: Episode-level paired comparisons. Differences are full scene anchor minus the shorter scene prefix; brackets give 95% confidence intervals. Lower Late LPIPS and higher nDTW favor the full anchor.
Real-time robot control demands enough visual history to infer motion and task progress, but processing that history can delay action. We present Long-WAM, a model-system framework for scaling the context of causal world-action models under real-time control constraints. Our central finding is that access to history is not the same as using it: longer histories pay off far more when the video foundation is pretrained autoregressively (AR). We first learn causal prediction from robot and egocentric videos without action labels, then preserve this history-to-future structure during world-action adaptation. On RoboCasa GR-1, increasing context from 0.0 to 19.2 seconds raises success from 63.3% to 78.7%, whereas a bidirectionally pretrained initialization shows no net gain; robot-domain AR pretraining further raises peak success on GR-1 and LIBERO-Long. Long-WAM also achieves the best results among compared methods on LIBERO-Long, RoboTwin 2.0, and DOMINO. Streaming observation encoding, asynchronous execution, and hardware-specific acceleration enable deployment on RTX 5090, DGX Spark, and Jetson AGX Thor without dropping future prediction; on RTX 5090, each action chunk, including future-video latent prediction, takes 107.4 ms. Real-time deployment on Unitree G1 and YAM supports dynamic and long-horizon manipulation, including 95% success on dynamic cup stacking, where Pi0.5 and Fast-WAM succeed in none of 20 trials. As a memory-informed executor, Long-WAM also complements higher-level planning in composite tasks.
Existing robot video world models are typically trained with low-level objectives such as reconstruction and perceptual similarity, which are poorly aligned with the capabilities that matter most for robot decision making, including instruction following, manipulation success, and physical plausibility. They also suffer from error accumulation in long-horizon autoregressive prediction. We present RoboAlign-R1, a framework that combines reward-aligned post-training with stabilized long-horizon inference for robot video world models. We construct RobotWorldBench, a benchmark of 10,000 annotated video-instruction pairs collected from four robot data sources, and train a multimodal teacher judge, RoboAlign-Judge, to provide fine-grained six-dimensional evaluation of generated videos. We then distill the teacher into a lightweight student reward model for efficient reinforcement-learning-based post-training. To reduce long-horizon rollout drift, we further introduce Sliding Window Re-encoding (SWR), a training-free inference strategy that periodically refreshes the generation context. Under our in-domain evaluation protocol, RoboAlign-R1 improves the aggregate six-dimension score by 10.1% over the strongest baseline, including gains of 7.5% on Manipulation Accuracy and 4.6% on Instruction Following; these ranking improvements are further supported by an external VLM-based cross-check and a blinded human study. Meanwhile, SWR improves long-horizon prediction quality with only about 1% additional latency, yielding a 2.8% gain in SSIM and a 9.8% reduction in LPIPS. Together, these results show that reward-aligned post-training and stabilized long-horizon decoding improve task consistency, physical realism, and long-horizon prediction quality in robot video world models.
Hao Wu, Yuqi Li, Yuan Gao +10
Tencent · Tsinghua University · Department of Computer Science, City University of New York +3
Action-conditioned robot world models generate future video frames of the manipulated scene given a robot action sequence, offering a promising alternative for simulating tasks that are difficult to model with traditional physics engines. However, these models are optimized for short-term prediction and break down when deployed autoregressively: each predicted clip feeds back as context for the next, causing errors to compound and visual quality to rapidly degrade. We address this through the following contributions. First, we introduce a reinforcement learning (RL) post-training scheme that trains the world model on its own autoregressive rollouts rather than on ground-truth histories. We achieve this by adapting a recent contrastive RL objective for diffusion models to our setting and show that its convergence guarantees carry over exactly. Second, we design a training protocol that generates and compares multiple candidate variable-length futures from the same rollout state, reinforcing higher-fidelity predictions over lower-fidelity ones. Third, we develop efficient, multi-view visual fidelity rewards that combine complementary perceptual metrics across camera views and are aggregated at the clip level for dense, low-variance training signal. Fourth, we show that our approach establishes a new state-of-the-art for rollout fidelity on the DROID dataset, outperforming the strongest baseline on all metrics (e.g., LPIPS reduced by 14% on external cameras, SSIM improved by 9.1% on the wrist camera), winning 98% of paired comparisons, and achieving an 80% preference rate in a blind human study.
Jai Bardhan, Patrik Drozdik, Josef Sivic +1
Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague