cs.ROAug 26, 2026

WALL-SS: Scaling Long-horizon World Models via Next-Scale Autoregression

Authors: Maeve Zhang, Rain Sun, Xiang Wang, Cyril Zhang, Shalfun Li, Meng Cao, Howard Lu, Ethan Chen, 

Organizations: X Square Robot Team

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Generative world models provide robots with predictive models of how the world evolves under interaction, with growing potential for simulation, planning, policy evaluation, and robot learning. Beyond clip-level future prediction, a unified generative formulation should relate actions to consequences, support flexible horizons and continuous interaction, and enable reward-driven optimization. We introduce WALL-SS, a world model that generates visual futures through Scale-wise autoregressive Scaling, enabling action-controllable and long-horizon robotic simulation. WALL-SS represents embodied trajectories as causal sequences of temporally interleaved observations and actions, making action-dependent state transitions explicit while naturally supporting variable-length generation, streaming extension through reusable causal states, and direct optimization through sequence probabilities. To make this formulation effective over long horizons, we generate each future observation in a coarse-to-fine manner and develop three complementary components within the same hierarchy. Action-conditioned next-scale prediction injects scale-aligned action representations to improve action-future coupling and model both successful and failed behaviors. Scale-compressed long-horizon memory retains recent interactions at fine resolution while compressing distant observations and actions, with scale-wise dream forcing enhancing robustness to self-generated context. Finally, on-policy alignment optimizes autoregressive visual dynamics with action-following and long-term consistency rewards while preserving the pretrained visual distribution. Experiments show that WALL-SS improves action following and trajectory accuracy, supports coherent minute-long streaming rollout under bounded memory, and consistently benefits from on-policy alignment in reducing action drift and long-horizon inconsistency.

Figures & tables

Figure 1: Overview of WALL-SS . WALL-SS learns action-grounded visual dynamics from heterogeneous robot and UMI demonstrations. Given a task instruction, an initial multi-view observation, and prescribed robot actions, the model generates visual futures through coarse-to-fine next-scale autoregression and propagates state over streaming time using bounded time–scale memory. Generated observations can be recursively consumed by an external robot policy for closed-loop rollout. Bottom panels show representative long-horizon visual futures and quantitative evaluations of generation capability and real-robot calibration.
Figure 2: Comparison of generative world-model formulations. (a) Conventional video diffusion predicts a future clip from visual history through iterative denoising. (b) Action-conditioned diffusion additionally conditions generation on robot actions, but is typically organized as a clip-level prediction process. (c) In contrast, WALL-SS causally aligns observations, actions, and task context, generates visual futures through coarse-to-fine next-scale autoregression, and propagates bounded time–scale memory across clips for action-faithful streaming rollout.
Figure 3: Overall framework of WALL-SS . WALL-SS unifies action-conditioned next-scale visual prediction, streaming long-horizon state propagation, and on-policy alignment of autoregressive visual dynamics.
Figure 4: Overview of the training-data composition and curation pipeline. Public robot data, private robot and non-embodiment UMI recordings, and intervention/recovery trajectories are processed through video-integrity, temporal-alignment, calibration-and-kinematics, and outcome-aware quality checks, and are then standardized into semantic clips at 10 FPS. Samples with reliable action geometry are routed to the action-conditioned pool Dac\mathcal{D}_{\mathrm{ac}} , which supports action-conditioned future prediction, video generation, and value estimation. Samples with valid visual streams but unavailable or unreliable action geometry are retained in the video-only pool Dvid\mathcal{D}_{\mathrm{vid}} for video generation.
Figure 5: On-policy visual alignment and closed-loop evaluation. (a) Under a fixed initial observation, instruction, and prescribed action sequence, a newly frozen behavior model samples fresh long-horizon rollouts. (b) At evaluation time, the same frozen external policy interacts independently with WALL-SS and the physical robot from matched tasks and initial states; their outcomes are compared without updating either system.
ModelVisual QualityMotion QualityTemporal & Semantic ConsistencyPhysical & Conditional Fidelity
Image Quality ↑\uparrowAesthetic Quality ↑\uparrowDynamic Degree ↑\uparrowFlow Score ↑\uparrowMotion Smooth. ↑\uparrowSubject Consist. ↑\uparrowBackground Consist. ↑\uparrowSemantic Align. ↑\uparrowInteraction Quality ↑\uparrowPerspectivity ↑\uparrowInstruction Following ↑\uparrowTrajectory Acc. ↑\uparrowDepth Acc. ↑\uparrowAction Following ↑\uparrow
InfinityStar ( 34 )0.6020.3210.2530.0440.6010.8030.8790.8600.4840.7680.4060.2510.749–
Wan2.1-1.3B ( 47 )0.6790.3450.3290.0980.6720.6080.6500.8560.4520.8580.3800.1970.715–
Wan2.2-14B ( 48 )0.6900.3370.4270.1810.7930.8030.8620.8360.4760.8360.3940.1590.657–
CogVideoX-5B ( 59 )0.7180.4290.3080.1430.6860.7760.8230.8900.4000.9000.3800.1770.774–
Cosmos3-Nano ( 1 )0.6930.4390.4250.2080.7010.8610.8970.9030.5160.8740.4100.2020.8160.044
Table 1: Quantitative evaluation of embodied video generation. Quantitative comparisons against representative foundation video generators. WALL-SS achieves the strongest interaction quality, instruction following, and trajectory accuracy while maintaining competitive visual and semantic quality. Bold and underlined values denote the best and second-best results.
Figure 6: Qualitative comparison of long-horizon video rollouts. Rows correspond to Wan2.1-1.3B, Wan2.2-14B, Infinity-8B, and our model, while columns show snapshots at 5, 30, and 60 seconds. The left task requires sorting tabletop objects into the corresponding baskets, and the right task requires pouring water into a cup. The comparison highlights each model’s ability to preserve scene geometry, object identity, and coherent task progression over extended horizons.
Figure 7: Counterfactual action-conditioned rollouts. Starting from the same initial observation, we vary only the commanded action and autoregressively generate the corresponding future. Blue and orange curves denote commanded and realized trajectories, respectively.
Figure 8: Long-horizon action following in a streaming rollout. For a single continuous rollout, shown as keyframes with the overlaid end-effector path, the frame-wise trajectory-following error remains below 0.5%0.5\% of the image diagonal throughout the contact-rich pouring phase, while the generated global path follows the commanded condition.
Figure 10: Perceptual quality over a 60-second autoregressive rollout. PSNR ( ↑\uparrow ), FID ( ↓\downarrow ), and FVD ( ↓\downarrow ) are measured in successive rollout windows. Across all windows, WALL-SS achieves higher PSNR and lower FID and FVD than others.
Figure 11: Cross-view consistency. WALL-SS achieves low 3D-probe and correspondence errors (both ↓\downarrow ), indicating 3D-aware and view-consistent representations.
Figure 12: Closed-loop calibration and within-task checkpoint ranking. (a) Task-level success-rate calibration between matched real-robot and generated rollouts. Each marker represents one task–checkpoint pair; the dashed diagonal denotes perfect agreement and the solid line denotes the least-squares fit. (b) Within-task ranking across five policy checkpoints. Bars compare real-robot and generated success rates, with error bars denoting per-cell binomial standard errors. The reported ρ\rho values are per-task Spearman rank correlations.
TaskTask typeSubtasks
T1Put glasses on shelfGrasp glasses; Place glasses upright on shelf
T2Put fruits in basketGrasp fruit in order; Place fruit in basket in order
T3Pour water into cupGrasp bottle; Align with cup; Pour water; Set bottle down
T4Put ring on rodGrasp ring; Align ring with rod; Slide ring onto rod
T5Put item in drawerOpen drawer; Grasp item; Place item in drawer; Close drawer
T6Put garment in hamperGrasp garment; Move over hamper; Place garment in hamper
Table 2: Tasks for closed-loop policy consistency evaluation. A variety of manipulation tasks are decomposed into ordered subtasks to assess execution consistency beyond final task success.
Figure 13: Closed-loop fidelity across outcomes and subtask transitions. (a) Confusion matrix of WALL-SS and real-robot outcomes for 600 paired rollouts; entries give counts and row-normalized rates. (b) Marginal generated-minus-real success-rate bias at each execution stage; error bars are binomial standard errors. (c) Conditional transition differences Twm−TrealT^{\mathrm{wm}}-T^{\mathrm{real}} identify where discrepancies first emerge; hatching indicates unavailable task–transition combinations. WALL-SS matches the real outcome in 527 episodes, with the remaining optimism concentrated at contact/insertion stages.
 Calibration (30 cells) Within-task ranking Episode agreement (600 pairs)
ModelMAE ↓\downarrowBias →0\rightarrow 0Slope →1\rightarrow 1rr ↑\uparrow Pair. Acc ↑\uparrowRegret ↓\downarrowρˉ\bar{\rho} ↑\uparrow Bal. Acc ↑\uparrowSucc. Rec. ↑\uparrowFail. Rec. ↑\uparrowFPR ↓\downarrow
WALL-SS0.062+0.028+0.0280.840.93 0.890.0250.88 0.880.900.860.14
Table 3: Closed-loop outcome alignment. Generated and real rollouts are compared at three levels: success-rate calibration across 30 task–checkpoint cells, checkpoint ranking over all untied within-task pairs, and outcome agreement over 600 matched episodes. Ranking metrics include pairwise accuracy, mean selection regret (Eq. ( 12 )), and mean per-task Spearman ρˉ\bar{\rho} .
Figure 14: Diverse real-robot manipulation. Task Progress on tabletop tasks covering direct physical manipulation, pick-and-place, insertion, instruction-conditioned sorting and classification, and sequential interaction. The final group reports the unweighted task average.
More in this topic

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 380 papers

Explore similar work

CardsList
Sep 13, 2026

World-Action Models for Robot Learning and Control: A Survey

Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: https://rcl-robotics.github.io/Awesome-World-Action-Models.

Zuxing Lu, Hongjia Zhai, Guanzhi Wang +13

MBZUAI · Caltech · Amazon FAR +4

World Action ModelsAction-Conditioned World ModelsWorld Models for RoboticsWorld Model LearningWorld Model-Based Planning
AskPDF
Oct 7, 2026

Long-WAM: Scaling the Context of World-Action Models

Real-time robot control demands enough visual history to infer motion and task progress, but processing that history can delay action. We present Long-WAM, a model-system framework for scaling the context of causal world-action models under real-time control constraints. Our central finding is that access to history is not the same as using it: longer histories pay off far more when the video foundation is pretrained autoregressively (AR). We first learn causal prediction from robot and egocentric videos without action labels, then preserve this history-to-future structure during world-action adaptation. On RoboCasa GR-1, increasing context from 0.0 to 19.2 seconds raises success from 63.3% to 78.7%, whereas a bidirectionally pretrained initialization shows no net gain; robot-domain AR pretraining further raises peak success on GR-1 and LIBERO-Long. Long-WAM also achieves the best results among compared methods on LIBERO-Long, RoboTwin 2.0, and DOMINO. Streaming observation encoding, asynchronous execution, and hardware-specific acceleration enable deployment on RTX 5090, DGX Spark, and Jetson AGX Thor without dropping future prediction; on RTX 5090, each action chunk, including future-video latent prediction, takes 107.4 ms. Real-time deployment on Unitree G1 and YAM supports dynamic and long-horizon manipulation, including 95% success on dynamic cup stacking, where Pi0.5 and Fast-WAM succeed in none of 20 trials. As a memory-informed executor, Long-WAM also complements higher-level planning in composite tasks.

Wei Huang, Bohan Zhang, Chenzhi Liu +13

NVIDIA · HKU · MIT +1

Long-Horizon Robotic ManipulationAction-Conditioned World ModelsEfficient Inference for World Action ModelsVideo Prediction
AskPDF
Jun 19, 2026

Beyond the Next Step: Variable-Length Latent World Models for Long-Horizon Planning

Recently, world models have emerged as a promising paradigm for building intelligent agents by learning predictive models that estimate future environment states conditioned on observations and actions. In particular, JEPA-style latent world models provide an efficient alternative to pixel space prediction by learning action-conditioned dynamics in compact representation spaces. However, existing latent world models typically rely on one-step prediction and must be recursively rolled out for long-horizon planning, which leads to compounding errors and a mismatch between training objectives and downstream planning tasks. To address this limitation, we propose Variable-length Latent World Models (VLWMs), a framework that learns to predict future latent states conditioned on action sequences of variable lengths. Instead of training only on one-step transitions, VLWMs directly model temporally extended dynamics, allowing the same predictor to evaluate action plans over different horizons. We further introduce a curriculum training strategy that progressively expands the action horizon, stabilizing optimization from short-range dynamics to long-range prediction. At test time, we design planning methods tailored to VLWMs to better exploit their variable-length predictive capabilities. Experiments on long-horizon control tasks show that VLWMs significantly improve latent space world models, achieving 13% average improvement over the state-of-the-art LeWM across different datasets, with especially large gains on tasks requiring extended planning. These results suggest that VLWM provides a simple yet effective paradigm for improving long-horizon prediction and planning in latent world models.

Tianqi Du, Qi Zhang, Yifei Wang +1

State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Amazon AGI SF Lab · Institute for Artificial Intelligence, Peking University

Latent Dynamics ModelingWorld ModelsWorld Model-Based PlanningAction-Conditioned World ModelsLong-Horizon PlanningModel-Based Planning+1
AskPDF