Remembering What Matters for Long-Horizon Robotic World Models

Hao Wu; Tianyu Shao; Shengju Qian; Yizhou Zhao; Fan Zhang; Penghao Zhao; Weiyan Wang; Yufei Huang

cs.ROOct 10, 2026

Remembering What Matters for Long-Horizon Robotic World Models

Authors: Hao Wu, Tianyu Shao, Shengju Qian, Yizhou Zhao, Fan Zhang, Penghao Zhao, Weiyan Wang, Yufei Huang,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · North China Electric Power University · LIGHTSPEED · Carnegie Mellon University ·

Abstract

Autoregressive robot video world models based on discrete visual tokens remain unreliable over long rollout horizons. Retaining the entire generated history does not necessarily improve prediction quality. We investigate a possible memory-role conflict in Full Context decoding: persistent scene information, recent visual-action states, and stale generated history compete for attention despite their different predictive roles. This perspective motivates ReCAP (Receding Context with Anchor Preservation), a training-free context-scheduling method for long-horizon robot video world models. ReCAP keeps the pretrained tokenizer, world model, and visual decoder frozen. It preserves the initial scene anchor and a contiguous window of recent complete visual-action blocks while evicting older generated history. The schedule preserves within-block structure and temporal order, requires no additional modules or parameter updates, and bounds active-context growth independently of rollout length. We evaluate ReCAP on RT-1, BridgeV2, LIBERO-90, and CALVIN using frozen autoregressive world models. Across evaluated environments and horizons, ReCAP improves visual fidelity, action-conditioned consistency, and robot structure preservation relative to Full Context. On BridgeV2, it reduces LPIPS by 63.3%. In the system benchmark at H=64, it reduces the logical KV footprint by 73.3% and p50 latency by 6.51%. Attention analyses show increased attention to retained recent states, while history interventions associate larger amounts of stale context with higher prediction error. These observations are consistent with the memory-role-conflict hypothesis, although they do not isolate it from context-length and position-handling effects. Our results support inference-time context scheduling as a practical approach to improving long-horizon robot video prediction and computational efficiency without retraining.

Figures & tables

Figure 1: ReCAP improves long-horizon prediction and memory efficiency. (a) ReCAP preserves attention to recent visual states. (b) It improves prediction quality on BridgeV2. (c) Three VLM judges score ReCAP higher across semantic criteria. (d) It reduces KV-cache and GPU-memory usage. Panels use dedicated samples and extrapolation horizons and are therefore not directly comparable to the fixed-horizon evaluations in Section 5 .

Figure 2: Overview of ReCAP . Full Context retains all generated states, with growing context cost and increasing prediction drift in the evaluated rollouts. ReCAP groups history into complete visual–action blocks, preserves the initial scene anchor and the most recent blocks, and evicts stale middle history. This training-free schedule keeps the active KV cache bounded while maintaining stable autoregressive rollouts.

Figure 3: Qualitative case study. ReCAP preserves the robot arm throughout the rollout, whereas Full Context collapses; the blind VLM judge prefers ReCAP . The displayed A/B prompt differs from the per-video RT-1 protocol in Appendix D .

Figure 4: ReCAP improves long-horizon robustness. (a–c) LPIPS across prediction horizons on three datasets; shaded bands denote 95% confidence intervals. (d) Paired LPIPS AUC gains on independent BridgeV2 and RT-1 evaluation sets, where positive values favor ReCAP .

Figure 5: Qualitative comparisons across four datasets. Selected cases from Appendix C , with three prediction times per dataset: $t=16,32,64$ for RT-1, CALVIN, and LIBERO-90; $t=8,16,32$ for the shorter BridgeV2 protocol. Rows show ground truth, Full Context, and ReCAP .

Figure 6: Mechanism analysis of ReCAP . (a) Component ablation with 95% bootstrap confidence intervals. (b)

H{=}64

LPIPS after reinserting

k

stale blocks. (c) Layer-averaged semantic attention and (d) recent-state attention by layer on CALVIN C06 at

H{=}64

Metric	Full	ReCAP	Change
Tokens $\downarrow$	7,232	1,931	$-73.3\%$
Logical KV (MiB) $\downarrow$	254.25	67.89	$-73.3\%$
p50 (ms) $\downarrow$	312.95	292.57	$-6.51\%$
$C{=}256$ throughput $\uparrow$	1,596	6,834	$4.28\times$

Table 3: System efficiency summary.

H{=}64

values are macro averages; throughput uses

C{=}256

Figure 7: ReCAP bounds context cost and improves concurrent throughput. (a) Prompt tokens, (b) logical BF16 KV, (c) p50 world-model latency, and (d) concurrent throughput. Shading marks Full Context truncation; bands in (c) show the range across three datasets.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Token organization for the 1,931-token ReCAP configuration. (a) The fixed prefix comprises 1,280 scene-context tokens, 80 initial dynamics tokens, and 13 supplied action tokens, totaling 1,373 tokens. Each context-conditioned dynamics representation uses one-sixteenth as many visual tokens as the context representation. (b) Six recent complete visual–action blocks contribute 558 tokens, giving a 1,931-token input before next-frame generation. The next 80 visual tokens are outputs and are excluded from this input budget. (c) Before a further prediction, the generated visual tokens are paired with the next supplied action to complete a 93-token block, and the oldest recent block is evicted. Here, action

\mathbf{u}_{i}

conditions the transition from frame

i

to frame

i+1

. Segment widths are schematic.

Component	Version
Python	3.13.11
PyTorch / torchvision	2.9.1 / 0.24.1+d801a34
CUDA (PyTorch build) / cuDNN	12.9 / 9.1.6.0
NCCL	2.27.3
Transformers	4.57.1; 4.57.1.post1 (metrics)
vLLM	0.13.0

Appendix

Table 4: Software environment used for model execution and evaluation. Versions marked “metrics” refer to the separate metric-evaluation environment.

Figure 9: CALVIN tokenizer adaptation. (a) Reconstruction PSNR, LPIPS, and dynamics-code utilization measured on the same fixed set of 32 validation clips (256 frames) for the released RT-1 tokenizer and the 5k, 10k, and 20k checkpoints. (b) Same-frame reconstructions at $t\in\{0,2,4,7\}$ . The adapted tokenizer progressively removes the color shift and object disappearance observed with the unadapted RT-1 tokenizer.

Figure 10: CALVIN world-model adaptation. (a) Raw training loss and its nine-point moving average. (b) Validation negative log-likelihood evaluated every 1,000 updates; the star marks the minimum near 19k and the square marks the pre-specified final 20k checkpoint. (c) Validation perplexity. (d) A 16-frame action-conditioned open-loop rollout on held-out CALVIN validation data. The model receives the first frame and ground-truth actions, then autoregressively feeds back predicted visual tokens. It achieves 21.18 dB mean future PSNR, compared with 16.43 dB for repeating the first frame.

Setting	Tokenizer adaptation	World-model adaptation
Initialization	RLVR-World RT-1 tokenizer	RLVR-World RT-1 world model
Trainable component	Compressive tokenizer	Autoregressive transformer
Training updates	20,000	20,000
Segment / context frames	8 / 1	8 / 1
Global batch size	8	8
Optimizer	AdamW	AdamW

Appendix

Table 5: CALVIN adaptation and training hyperparameters. Batch size denotes the global batch across eight H20 GPUs.

Figure 11: LIBERO-90 tokenizer adaptation. (a) Reconstruction PSNR, LPIPS, and dynamics-code utilization for the released RT-1 tokenizer and the final 20k LIBERO tokenizer, evaluated on the identical fixed set of 32 held-out clips (256 frames). (b) Same-frame reconstructions at $t\in\{0,2,4,7\}$ . Adaptation substantially reduces color shift and preserves robot and object structure.

Figure 12: LIBERO-90 world-model adaptation. (a) Raw training loss and its nine-point moving average. (b) Validation negative log-likelihood evaluated every 1,000 updates; the star marks the minimum near 17k and the square marks the pre-specified final 20k checkpoint. (c) Validation perplexity. (d) A 16-frame action-conditioned visual open-loop rollout on a held-out high-motion LIBERO-90 episode. The model observes the first frame, receives ground-truth future actions, and autoregressively feeds back its visual predictions.

Setting	Tokenizer adaptation	World-model adaptation
Initialization	RLVR-World RT-1 tokenizer	RLVR-World RT-1 world model
Trainable component	Compressive tokenizer	Autoregressive transformer
Training updates	20,000	20,000
Segment / context frames	8 / 1	8 / 1
Global batch size	8	8
Optimizer	AdamW	AdamW

Appendix

Table 6: LIBERO-90 adaptation and training hyperparameters. Batch size denotes the global batch across eight H20 GPUs.

Figure 13: BridgeV2 tokenizer training. (a) Total generator loss; (b–c) context and dynamics reconstruction and perceptual losses; (d–e) online validation reconstruction and perceptual losses; (f) discriminator hinge loss plus gradient penalty (logarithmic vertical axis). Dashed lines mark adversarial training activation at shared step 10k. Faint training traces show raw logged values; solid training traces use a nine-point moving average. Validation points are unsmoothed and end at step 59,001. The shared counter includes alternating generator/discriminator updates after GAN activation.

Figure 14: BridgeV2 world-model training. (a) Raw causal cross-entropy and its nine-point moving average over 100k transformer updates. (b) Validation negative log-likelihood and (c) validation perplexity. Stars indicate the lowest monitored validation loss near 70k; squares mark the final 100k checkpoint used for the rollout comparisons. All validation points are plotted without smoothing, including the final evaluation at 100,000.

Setting	Tokenizer adaptation	World-model adaptation
Initialization	RT-1 tokenizer	RT-1 world model
Checkpoint counter	60,000 (shared)	100,000
Segment / context frames	8 / 1	8 / 1
Per-device batch / accumulation	1 / 4	1 / 1
Global batch per optimizer update	32	8
Optimizer / weight decay	AdamW / 0	AdamW / 0

Appendix

Table 7: BridgeV2 adaptation hyperparameters. Tokenizer steps use the shared generator/discriminator counter; world-model steps are transformer updates.

Setting	Value
Tokenizer / world-model checkpoint	60k / 100k
World-model parameters	127.1M
Training GPUs / global batch / segment	8 H20 / 8 / 8 frames
World-model learning rate / precision	$10^{-5}$ / BF16
Observed / predicted frames	1 / 32
Inference precision	FP32, TF32 disabled

Appendix

Table 8: BridgeV2 settings for the additional

H=32

comparisons.

Figure 17: Additional LIBERO-90 held-out-task comparisons. The three examples use distinct language instructions from the task-disjoint validation split. ReCAP improves mean PSNR over Full Context by $+6.04$ , $+3.92$ , and $+2.61$ dB, respectively. Frames are vertically flipped for canonical display orientation only.

Figure 18: Additional BridgeV2 validation comparisons. Three high-gain examples selected from a fixed pool of 24 candidates, with nine displayed time points ( $t=0,4,8,12,16,20,24,28,32$ ) over 32 predicted frames. Rows show ground truth, Full Context, and ReCAP . Mean future PSNR improves by $+0.78$ , $+4.02$ , and $+2.30$ dB; mean LPIPS decreases by $26.7\%$ , $32.5\%$ , and $23.3\%$ , respectively. Metrics use all 32 predictions, excluding $t=0$ . Task text identifies the episodes and is not a model input.

Score	Interaction quality	Instruction following	Task progress	Motion naturalness
1–2	Penetration or absent proper contact	Wrong action, object, or scene	No visible progress	Severe jumps or jitter; or a frozen arm
3–4	Contact with implausible physical response	Related behavior with major errors	Motion without meaningful progress	Frequent irregular motion; or tiny, aimless movements
5–6	Mostly plausible, with minor interaction errors	General intent followed with execution errors	Partial completion	Meaningful motion with occasional stutter or kinematic issues
7–8	Realistic contact and object response	Mostly correct, with minor deviations	Mostly completed	Meaningful, smooth motion with minor imperfections
9–10	Visually indistinguishable from real interaction	All instructed elements correctly executed	Fully completed	Meaningful, smooth, physically plausible motion

Appendix

Table 9: Shared VLM evaluation rubric. Score bands summarize the prompt; the mandatory score caps also apply. Higher values indicate better observed behavior.

Figure 19: Human–VLM alignment under two judging protocols. Bars show pooled fractions on the same 80 pairs: (a) three-way agreement with human-majority labels and (b) tie rates. Gray denotes the original protocol and blue the combined revision. The dashed line in (b) marks the human tie rate of 17.5%. Judge order is fixed across panels.

Protocol	Evaluator	ReCAP wins	Ties	Full Context wins	Agreement with humans	$\kappa$
Reference	Human majority	42	14	24	–	–
Original	Qwen3-VL	47	8	25	55/80 (68.8%)	0.46
	InternVL3	45	10	25	53/80 (66.3%)	0.43
	GPT-5.5	49	7	24	58/80 (72.5%)	0.52
Combined	Qwen3-VL	43	14	23	60/80 (75.0%)	0.58
	InternVL3	42	15	23	58/80 (72.5%)	0.55

Appendix

Table 10: Human–VLM agreement on 80 paired rollouts. Agreement counts all three labels, including ties. Unweighted Cohen’s

\kappa

is calculated from the pooled preference counts and observed agreement. The combined revision is described in Appendix D.7 .

Protocol	Qwen3-VL	InternVL3	GPT-5.5
Original	47 / 8 / 25	45 / 10 / 25	49 / 7 / 24
Explicitly allow equal scores	43 / 14 / 23	42 / 15 / 23	45 / 12 / 23
Half-strength static penalty	45 / 11 / 24	43 / 13 / 24	47 / 10 / 23
Remove static penalty	42 / 13 / 25	40 / 15 / 25	44 / 12 / 24
Include first and last frames	46 / 11 / 23	44 / 13 / 23	48 / 9 / 23
Combined revision	43 / 14 / 23	42 / 15 / 23	45 / 12 / 23

Appendix

Table 11: Sensitivity of VLM preferences to the scoring protocol. Each cell gives ReCAP wins / ties / Full Context wins and sums to 80. The original protocol remains the basis of the main VLM results.

Scene prefix (tokens)	Initial block (tokens)	Recent blocks	Recent tokens	Effective tokens	Unused budget	Late LPIPS $\downarrow$	nDTW $\uparrow$
4	93	19	1,767	1,864	67	$0.208\pm 0.003$	$0.619\pm 0.006$
16	93	19	1,767	1,876	55	$0.207\pm 0.003$	$0.621\pm 0.006$
64	93	19	1,767	1,924	7	$0.205\pm 0.003$	$0.624\pm 0.005$
1,280	93	6	558	1,931	0	$\mathbf{0.203\pm 0.002}$	$\mathbf{0.628\pm 0.005}$

Appendix

Table 12: Scene-prefix allocation under a 1,931-token budget cap. Results on 450 held-out LIBERO-90 episodes with five generation seeds are reported as mean

\pm

standard deviation. Token counts describe the steady-state prompt; every row includes the same initial 93-token block. The full scene anchor is highlighted.

Comparison	$\Delta$ Late LPIPS $\downarrow$	95% CI	$\Delta$ nDTW $\uparrow$	95% CI
Full anchor $-$ 4 tokens	$-0.005$	$[-0.007,-0.003]$	$+0.009$	$[+0.005,+0.013]$
Full anchor $-$ 16 tokens	$-0.004$	$[-0.006,-0.002]$	$+0.007$	$[+0.003,+0.011]$
Full anchor $-$ 64 tokens	$-0.002$	$[-0.004,-0.001]$	$+0.004$	$[+0.001,+0.007]$

Appendix

Table 13: Episode-level paired comparisons. Differences are full scene anchor minus the shorter scene prefix; brackets give 95% confidence intervals. Lower Late LPIPS and higher nDTW favor the full anchor.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 257 papers

Long-WAM: Scaling the Context of World-Action Models
Oct 7, 2026Wei Huang, Bohan Zhang, Chenzhi Liu +13Long-Horizon Robotic Manipulation Action-Conditioned World Models
NVIDIA · HKU · MIT +1
RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models
May 5, 2026Hao Wu, Yuqi Li, Yuan Gao +10Multimodal Reward Modeling World Model Evaluation
Tencent · Tsinghua University · Department of Computer Science, City University of New York +3
Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning
Mar 26, 2026Jai Bardhan, Patrik Drozdik, Josef Sivic +1World Models for Robotics World Models
Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague

Oct 7, 2026J/K move · Enter open · S save

Long-WAM: Scaling the Context of World-Action Models

Wei Huang, Bohan Zhang, Chenzhi Liu, Isabella Liu+12

NVIDIA · HKU · MIT+1

Long-Horizon Robotic Manipulation Action-Conditioned World Models Efficient Inference for World Action Models Video Prediction

Open paper AskPDF arXiv

Real-time robot control demands enough visual history to infer motion and task progress, but processing that history can delay action. We present Long-WAM, a model-system framework for scaling the context of causal world-action models under real-time control constraints. Our central finding is that access to history is not the same as using it: longer histories pay off far more when the video foundation is pretrained autoregressively (AR). We first learn causal prediction from robot and egocentric videos without action labels, then preserve this history-to-future structure during world-action adaptation. On RoboCasa GR-1, increasing context from 0.0 to 19.2 seconds raises success from 63.3% to 78.7%, whereas a bidirectionally pretrained initialization shows no net gain; robot-domain AR pretraining further raises peak success on GR-1 and LIBERO-Long. Long-WAM also achieves the best results among compared methods on LIBERO-Long, RoboTwin 2.0, and DOMINO. Streaming observation encoding, asynchronous execution, and hardware-specific acceleration enable deployment on RTX 5090, DGX Spark, and Jetson AGX Thor without dropping future prediction; on RTX 5090, each action chunk, including future-video latent prediction, takes 107.4 ms. Real-time deployment on Unitree G1 and YAM supports dynamic and long-horizon manipulation, including 95% success on dynamic cup stacking, where Pi0.5 and Fast-WAM succeed in none of 20 trials. As a memory-informed executor, Long-WAM also complements higher-level planning in composite tasks.

Abstract

Video World Models

Explore similar work

Long-WAM: Scaling the Context of World-Action Models

RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models

Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning