WALL-SS: Scaling Long-horizon World Models via Next-Scale Autoregression

Maeve Zhang; Rain Sun; Xiang Wang; Cyril Zhang; Shalfun Li; Meng Cao; Howard Lu; Ethan Chen

cs.ROAug 26, 2026

WALL-SS: Scaling Long-horizon World Models via Next-Scale Autoregression

Authors: Maeve Zhang, Rain Sun, Xiang Wang, Cyril Zhang, Shalfun Li, Meng Cao, Howard Lu, Ethan Chen,

Organizations: X Square Robot Team

Abstract

Generative world models provide robots with predictive models of how the world evolves under interaction, with growing potential for simulation, planning, policy evaluation, and robot learning. Beyond clip-level future prediction, a unified generative formulation should relate actions to consequences, support flexible horizons and continuous interaction, and enable reward-driven optimization. We introduce WALL-SS, a world model that generates visual futures through Scale-wise autoregressive Scaling, enabling action-controllable and long-horizon robotic simulation. WALL-SS represents embodied trajectories as causal sequences of temporally interleaved observations and actions, making action-dependent state transitions explicit while naturally supporting variable-length generation, streaming extension through reusable causal states, and direct optimization through sequence probabilities. To make this formulation effective over long horizons, we generate each future observation in a coarse-to-fine manner and develop three complementary components within the same hierarchy. Action-conditioned next-scale prediction injects scale-aligned action representations to improve action-future coupling and model both successful and failed behaviors. Scale-compressed long-horizon memory retains recent interactions at fine resolution while compressing distant observations and actions, with scale-wise dream forcing enhancing robustness to self-generated context. Finally, on-policy alignment optimizes autoregressive visual dynamics with action-following and long-term consistency rewards while preserving the pretrained visual distribution. Experiments show that WALL-SS improves action following and trajectory accuracy, supports coherent minute-long streaming rollout under bounded memory, and consistently benefits from on-policy alignment in reducing action drift and long-horizon inconsistency.

Figures & tables

Figure 1: Overview of WALL-SS . WALL-SS learns action-grounded visual dynamics from heterogeneous robot and UMI demonstrations. Given a task instruction, an initial multi-view observation, and prescribed robot actions, the model generates visual futures through coarse-to-fine next-scale autoregression and propagates state over streaming time using bounded time–scale memory. Generated observations can be recursively consumed by an external robot policy for closed-loop rollout. Bottom panels show representative long-horizon visual futures and quantitative evaluations of generation capability and real-robot calibration.

Figure 2: Comparison of generative world-model formulations. (a) Conventional video diffusion predicts a future clip from visual history through iterative denoising. (b) Action-conditioned diffusion additionally conditions generation on robot actions, but is typically organized as a clip-level prediction process. (c) In contrast, WALL-SS causally aligns observations, actions, and task context, generates visual futures through coarse-to-fine next-scale autoregression, and propagates bounded time–scale memory across clips for action-faithful streaming rollout.

Figure 3: Overall framework of WALL-SS . WALL-SS unifies action-conditioned next-scale visual prediction, streaming long-horizon state propagation, and on-policy alignment of autoregressive visual dynamics.

Figure 4: Overview of the training-data composition and curation pipeline. Public robot data, private robot and non-embodiment UMI recordings, and intervention/recovery trajectories are processed through video-integrity, temporal-alignment, calibration-and-kinematics, and outcome-aware quality checks, and are then standardized into semantic clips at 10 FPS. Samples with reliable action geometry are routed to the action-conditioned pool $\mathcal{D}_{\mathrm{ac}}$ , which supports action-conditioned future prediction, video generation, and value estimation. Samples with valid visual streams but unavailable or unreliable action geometry are retained in the video-only pool $\mathcal{D}_{\mathrm{vid}}$ for video generation.

Figure 5: On-policy visual alignment and closed-loop evaluation. (a) Under a fixed initial observation, instruction, and prescribed action sequence, a newly frozen behavior model samples fresh long-horizon rollouts. (b) At evaluation time, the same frozen external policy interacts independently with WALL-SS and the physical robot from matched tasks and initial states; their outcomes are compared without updating either system.

Model	Visual Quality		Motion Quality			Temporal & Semantic Consistency			Physical & Conditional Fidelity
Model	Image Quality $\uparrow$	Aesthetic Quality $\uparrow$	Dynamic Degree $\uparrow$	Flow Score $\uparrow$	Motion Smooth. $\uparrow$	Subject Consist. $\uparrow$	Background Consist. $\uparrow$	Semantic Align. $\uparrow$	Interaction Quality $\uparrow$	Perspectivity $\uparrow$	Instruction Following $\uparrow$	Trajectory Acc. $\uparrow$	Depth Acc. $\uparrow$	Action Following $\uparrow$
InfinityStar ( 34 )	0.602	0.321	0.253	0.044	0.601	0.803	0.879	0.860	0.484	0.768	0.406	0.251	0.749	–
Wan2.1-1.3B ( 47 )	0.679	0.345	0.329	0.098	0.672	0.608	0.650	0.856	0.452	0.858	0.380	0.197	0.715	–
Wan2.2-14B ( 48 )	0.690	0.337	0.427	0.181	0.793	0.803	0.862	0.836	0.476	0.836	0.394	0.159	0.657	–
CogVideoX-5B ( 59 )	0.718	0.429	0.308	0.143	0.686	0.776	0.823	0.890	0.400	0.900	0.380	0.177	0.774	–
Cosmos3-Nano ( 1 )	0.693	0.439	0.425	0.208	0.701	0.861	0.897	0.903	0.516	0.874	0.410	0.202	0.816	0.044

Table 1: Quantitative evaluation of embodied video generation. Quantitative comparisons against representative foundation video generators. WALL-SS achieves the strongest interaction quality, instruction following, and trajectory accuracy while maintaining competitive visual and semantic quality. Bold and underlined values denote the best and second-best results.

Figure 6: Qualitative comparison of long-horizon video rollouts. Rows correspond to Wan2.1-1.3B, Wan2.2-14B, Infinity-8B, and our model, while columns show snapshots at 5, 30, and 60 seconds. The left task requires sorting tabletop objects into the corresponding baskets, and the right task requires pouring water into a cup. The comparison highlights each model’s ability to preserve scene geometry, object identity, and coherent task progression over extended horizons.

Figure 7: Counterfactual action-conditioned rollouts. Starting from the same initial observation, we vary only the commanded action and autoregressively generate the corresponding future. Blue and orange curves denote commanded and realized trajectories, respectively.

Figure 8: Long-horizon action following in a streaming rollout. For a single continuous rollout, shown as keyframes with the overlaid end-effector path, the frame-wise trajectory-following error remains below $0.5\%$ of the image diagonal throughout the contact-rich pouring phase, while the generated global path follows the commanded condition.

Figure 10: Perceptual quality over a 60-second autoregressive rollout. PSNR (

\uparrow

), FID (

\downarrow

), and FVD (

\downarrow

) are measured in successive rollout windows. Across all windows, WALL-SS achieves higher PSNR and lower FID and FVD than others.

Figure 11: Cross-view consistency. WALL-SS achieves low 3D-probe and correspondence errors (both

\downarrow

), indicating 3D-aware and view-consistent representations.

Figure 12: Closed-loop calibration and within-task checkpoint ranking. (a) Task-level success-rate calibration between matched real-robot and generated rollouts. Each marker represents one task–checkpoint pair; the dashed diagonal denotes perfect agreement and the solid line denotes the least-squares fit. (b) Within-task ranking across five policy checkpoints. Bars compare real-robot and generated success rates, with error bars denoting per-cell binomial standard errors. The reported

\rho

values are per-task Spearman rank correlations.

Task	Task type	Subtasks
T1	Put glasses on shelf	Grasp glasses; Place glasses upright on shelf
T2	Put fruits in basket	Grasp fruit in order; Place fruit in basket in order
T3	Pour water into cup	Grasp bottle; Align with cup; Pour water; Set bottle down
T4	Put ring on rod	Grasp ring; Align ring with rod; Slide ring onto rod
T5	Put item in drawer	Open drawer; Grasp item; Place item in drawer; Close drawer
T6	Put garment in hamper	Grasp garment; Move over hamper; Place garment in hamper

Table 2: Tasks for closed-loop policy consistency evaluation. A variety of manipulation tasks are decomposed into ordered subtasks to assess execution consistency beyond final task success.

Figure 13: Closed-loop fidelity across outcomes and subtask transitions. (a) Confusion matrix of WALL-SS and real-robot outcomes for 600 paired rollouts; entries give counts and row-normalized rates. (b) Marginal generated-minus-real success-rate bias at each execution stage; error bars are binomial standard errors. (c) Conditional transition differences $T^{\mathrm{wm}}-T^{\mathrm{real}}$ identify where discrepancies first emerge; hatching indicates unavailable task–transition combinations. WALL-SS matches the real outcome in 527 episodes, with the remaining optimism concentrated at contact/insertion stages.

Calibration (30 cells)

Within-task ranking

Episode agreement (600 pairs)

Model

MAE

\downarrow

Bias

\rightarrow 0

Slope

\rightarrow 1

r

\uparrow

Pair. Acc

\uparrow

Regret

\downarrow

\bar{\rho}

\uparrow

Bal. Acc

\uparrow

Succ. Rec.

\uparrow

Fail. Rec.

\uparrow

FPR

\downarrow

WALL-SS

0.062

+0.028

0.84

0.93

0.89

0.025

0.88

0.90

0.86

0.14

Table 3: Closed-loop outcome alignment. Generated and real rollouts are compared at three levels: success-rate calibration across 30 task–checkpoint cells, checkpoint ranking over all untied within-task pairs, and outcome agreement over 600 matched episodes. Ranking metrics include pairwise accuracy, mean selection regret (Eq. ( 12 )), and mean per-task Spearman

\bar{\rho}

Figure 14: Diverse real-robot manipulation. Task Progress on tabletop tasks covering direct physical manipulation, pick-and-place, insertion, instruction-conditioned sorting and classification, and sequential interaction. The final group reports the unweighted task average.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 380 papers

World-Action Models for Robot Learning and Control: A Survey
Sep 13, 2026Zuxing Lu, Hongjia Zhai, Guanzhi Wang +13World Action Models Action-Conditioned World Models
MBZUAI · Caltech · Amazon FAR +4
Long-WAM: Scaling the Context of World-Action Models
Oct 7, 2026Wei Huang, Bohan Zhang, Chenzhi Liu +13Long-Horizon Robotic Manipulation Action-Conditioned World Models
NVIDIA · HKU · MIT +1
Beyond the Next Step: Variable-Length Latent World Models for Long-Horizon Planning
Jun 19, 2026Tianqi Du, Qi Zhang, Yifei Wang +1Latent Dynamics Modeling World Models
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Amazon AGI SF Lab · Institute for Artificial Intelligence, Peking University

Sep 13, 2026J/K move · Enter open · S save

World-Action Models for Robot Learning and Control: A Survey

Zuxing Lu, Hongjia Zhai, Guanzhi Wang, Huajian Zeng+12

MBZUAI · Caltech · Amazon FAR+4

World Action Models Action-Conditioned World Models World Models for Robotics World Model Learning World Model-Based Planning

Open paper AskPDF arXiv

Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: https://rcl-robotics.github.io/Awesome-World-Action-Models.

Abstract

World Models for Robotics

Explore similar work

World-Action Models for Robot Learning and Control: A Survey

Long-WAM: Scaling the Context of World-Action Models

Beyond the Next Step: Variable-Length Latent World Models for Long-Horizon Planning