Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Models

Yijie Zhu; Zitong Yu; Wei Li; Hui Ma; Wen Li; Rui Shao; Liqiang Nie

cs.CVSep 6, 2026

Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Models

Authors: Yijie Zhu, Zitong Yu, Wei Li, Hui Ma, Wen Li, Rui Shao, Liqiang Nie

Organizations: Harbin Institute of Technology (Shenzhen), 518055, China · Great Bay University, Dongguan 523000, China · University of Electronic Science and Technology of China, Chengdu 611731, China

Abstract

World Action Models (WAMs) extend Vision-Language-Action (VLA) models by incorporating future visual dynamics into action generation. However, existing WAMs often utilize imagined futures with limited adaptation to evolving execution progress, potentially introducing distracting or unreliable predictive cues. This limitation arises from two empirically identified forms of non-uniformity in future utility: (i) at the inter-progress level, the utility of imagined futures varies across execution stages as control demands change; and (ii) at the intra-progress level, individual future latents exhibit heterogeneous relevance within the same progress state. To address these limitations, we propose ProWAM, a Progress-Conditioned World Action Model that introduces execution progress as an explicit intermediate representation for adaptive imagination utilization. ProWAM comprises two tightly coupled components: (1) To obtain a reliable representation of execution progress, we propose the Self-Supervised Dual-Temporal Progress Encoder (SS-DTPE). SS-DTPE couples short-term action-observation interaction modeling with long-term recurrent progress aggregation to capture recent execution feedback and accumulated task history. (2) Conditioned on the progress representation from SS-DTPE, we propose the Hierarchical Progress-Conditioned Imagination Modulation (HPIM) to adapt imagination utilization to execution progress. HPIM operates at two complementary levels: an inter-progress global modulation mechanism adapts future utilization across execution stages, while an intra-progress relevance mechanism differentiates individual future latents within each progress state. Extensive experiments demonstrate consistent gains over strong VLA and WAM baselines.

Figures & tables

Fig. 1: (a) Motivation: Standard WAMs often adopt fixed imagination usage, overlooking progress-dependent utility and unequal relevance among future latents. ProWAM addresses these limitations through inter-progress global modulation and intra-progress future-latent relevance. (b) Architecture: Unlike Vanilla VLAs and Standard WAMs, ProWAM explicitly models execution progress to adapt imagination utilization. (c) Performance: ProWAM consistently outperforms representative baselines across simulation and real-world manipulation tasks.

Fig. 2: Empirical Motivation for Progress-Conditioned Imagination Usage. Left : Failure-case distribution across execution progress. The action-only variant tends to produce more failures in pre-contact stages such as search/transit and approach, suggesting that future visual latents can provide useful foresight before contact-rich interaction. Right: Sensitivity to perturbed future latent groups. Different progress stages show different sensitivity to near-, mid-, and long-horizon future latents, indicating that future information is not uniformly useful across the imagined sequence.

Fig. 3: Overview of ProWAM. ProWAM comprises a Self-Supervised Dual-Temporal Progress Encoder (SS-DTPE) and Hierarchical Progress-Conditioned Imagination Modulation (HPIM). SS-DTPE integrates short-term action–observation evidence with long-term execution history to encode the current progress state. Conditioned on this representation, HPIM jointly models an inter-progress global gate $d_{t}$ and intra-progress future-latent relevance $\omega_{t,f}^{l}$ , and combines them with the layer-wise coefficient $\eta_{l}$ to obtain the modulation weight $\beta_{t,f}^{l}$ . The resulting modulation is applied to the action-to-future attention pathway, enabling adaptive utilization of imagined future information.

Fig. 4: Self-Supervised Semantic Progress Alignment. Left (Property i): As execution progresses, the observation history aligns with increasingly complete instruction prefixes, which serve as latent semantic progress states. Right (Property ii): The expected semantic completion is constrained to evolve monotonically along successful trajectories.

Fig. 5: Experimental Setup. Top: Simulation evaluation on LIBERO, RoboTwin2.0, VLABench, RoboEval, and Mikasa-Robo, covering general manipulation, robustness, progress-aware execution, and temporal memory. Bottom: Real-world evaluation on the Galaxea R1 Lite and AgileX Cobot Magic platforms, covering progress-aware long-horizon manipulation and robustness to object and scene variations.

General VLA Models
Method	Spatial	Object	Goal	Long	Avg.
OpenVLA-OFT [ 1 ]	97.6	98.4	97.9	94.5	97.1
$\pi_{0}$ [ 2 ]	96.8	98.8	95.8	85.2	94.2
$\pi_{0.5}$ [ 51 ]	98.8	98.2	98.0	92.4	96.9
PD-VLA [ 7 ]	95.5	96.7	94.9	91.7	94.7
SmolVLA [ 52 ]	93.0	94.0	91.0	77.0	88.8

TABLE I: General Robotic Manipulation Performance on LIBERO. Comparison of task success rates across the four LIBERO suites. The average is computed over the four suites, and the best result is highlighted in bold.

General VLA Models
Method	Embodied PT.	Clean	Randomized	Avg.
$\pi_{0}$ [ 2 ]	$\checkmark$	65.9	58.4	62.2
$\pi_{0.5}$ [ 51 ]	$\checkmark$	82.7	76.8	79.8
X-VLA [ 72 ]	$\checkmark$	72.9	72.8	72.9
ABot-M0 [ 73 ]	$\checkmark$	86.1	85.1	85.6
World Action Models

TABLE II: Robustness and Generalization Performance on RoboTwin2.0. Following the standard protocol [ 10 , 21 , 11 ] , we report success rates under clean and randomized settings. Clean uses fixed initial configurations, while Randomized varies object poses and scene layouts. Embodied PT. indicates whether embodied pre-training is used.

Method	SR $\uparrow$	IS $\uparrow$	PS $\uparrow$
$\pi_{0}$ [ 2 ]	47.0	67.8	62.7
$\pi_{0}$ -FAST [ 78 ]	56.2	72.4	66.8
$\pi_{0.5}$ [ 51 ]	65.4	80.4	77.8
X-VLA [ 72 ]	–	–	67.8
ACoT-VLA [ 79 ]	–	79.8	66.1
Motus ∗ [ 21 ]	54.4	78.6	70.2

TABLE III: Progress-Aware Manipulation Performance on VLABench. We report Success Rate (SR), Intention Score (IS), and Progress Score (PS). ∗ denotes results reproduced by us under the same protocol.

Method	SR $\uparrow$	TP $\uparrow$
ACT [ 80 ]	0.278	0.454
$\pi_{0.5}$ (LoRA) [ 51 ]	0.105	0.336
Diffusion Policy [ 81 ]	0.030	0.200
X-VLA [ 72 ]	0.015	0.114
GR00T [ 82 ]	0.013	0.136
Fast-WAM ∗ [ 11 ]	0.313	0.404

TABLE IV: Progress-Aware Manipulation Performance on RoboEval. We report the mean Success Rate (SR) and Task Progression (TP) across eight tasks and 30 task variations.

Method	SGT	IM	RC3	RC5	RC9	Avg.
DP [ 81 ]	23	–	3	–	–	–
PTP [ 84 ]	26	–	4	–	–	–
MaIL [ 85 ]	27	–	11	–	–	–
Octo [ 86 ]	46	39	45	17	11	31.6
CronusVLA [ 87 ]	32	5	31	13	9	18.0
SpatialVLA [ 88 ]	23	27	27	17	11	21.0

TABLE V: Temporal and Memory-Dependent Manipulation Performance on Mikasa-Robo. Following the standard protocol [ 6 , 83 ] , we report success rates (%) over 100 evaluation episodes for each task. SGT, IM, and RC3/5/9 denote ShellGameTouch, InterceptMedium, and RememberColor with 3/5/9 candidate objects, respectively.

Method	Drawer M.			Fold C.			Plate L.			Object P.		Marker C.		Stack T.			Tray T.			Plate H.			Avg.
	Pull	+Place	+Push	Grasp	+Fold	+Release	Open	+Retrieve	+Cover	Open	+Pack	Insert	+Insert	Grasp	+Stack	+Stack	Pick	+Transfer	+Place	Pick	+Pass	+Place	(%)
Galaxea R1 Lite platform
OpenVLA †	60	52	40	40	34	20	56	46	38	48	46	60	54	54	52	48	40	34	20	56	54	50	40
OFT †	70	68	60	48	40	28	50	42	40	60	54	66	58	60	58	54	44	42	36	66	64	64	49
$\pi_{0.5}$ †	74	74	66	68	66	58	70	64	60	74	64	74	68	78	74	70	64	60	52	78	76	74	64
DreamVLA †	58	58	54	50	40	34	54	52	48	54	48	68	56	56	50	50	40	36	34	46	44	40	46

TABLE VI: Real-World Progress-Aware Long-Horizon Manipulation Results. † denotes results reproduced under the same training and deployment settings as ProWAM. Each entry reports the cumulative stage-completion success rate (%) over 50 evaluation trials. Avg. denotes the average full-task success rate across all eight tasks, computed from the completion rate of the final stage. The best result in each column is highlighted in bold.

Method	Sort B.			Clear O.		Fold (OOD)			Stack B.		Avg.
	Pick	+Sort	+Place	Pick	+Transfer	Grasp	+Fold	+Release	Pick	+Place	(%)
Galaxea R1 Lite platform
OFT †	64	60	56	58	56	24	16	14	66	62	47
$\pi_{0.5}^{\dagger}$	74	70	70	68	66	48	44	42	80	76	64
DreamVLA †	54	50	48	52	52	28	26	20	68	56	44
Fast-WAM †	68	66	62	66	64	42	36	32	74	72	58

TABLE VII: Real-World Generalization and Robustness Results. † denotes results reproduced under the same training and deployment settings as ProWAM. We report cumulative stage-completion rates (%) over 50 trials with lighting, scene, and object variations.

Method		LIBERO				Real-World
SS-DTPE	HPIM	Spatial	Object	Goal	Long	Long.	Gen.
$\times$	$\times$	96.6	99.6	97.4	94.8	56	58
$\checkmark$	$\times$	98.4	99.6	98.0	96.2	66	65
$\times$	$\checkmark$	98.2	99.4	98.2	96.6	62	64
$\checkmark$	$\checkmark$	99.6	100.0	98.8	97.8	72	69

TABLE VIII: Ablation on Model Components. We ablate the proposed SS-DTPE and HPIM to evaluate their contributions on LIBERO and real-world manipulation tasks.

Aggregator	LIBERO				Real-World
Aggregator	Spatial	Object	Goal	Long	Long.	Gen.
w/o Aggregator	96.2	99.2	97.0	95.2	58	59
LSTM [ 89 ]	97.2	99.6	98.2	96.6	60	63
Transformer [ 90 ]	97.6	99.8	97.6	96.4	62	61
Mamba (Ours)	99.6	100.0	98.8	97.8	72	69

TABLE IX: Ablation Studies on Long-Term Progress Aggregation in SS-DTPE. We compare different aggregation mechanisms while keeping other components unchanged.

Sem. Align.		Temp. Struct.	LIBERO				Real-World
Prop. i)	Prop. ii)	Temp. Struct.	Spatial	Object	Goal	Long	Long.	Gen.
$\times$	$\times$	$\times$	96.0	98.6	96.6	94.4	54	57
$\times$	$\times$	$\checkmark$	96.4	99.0	97.4	94.6	57	58
$\checkmark$	$\times$	$\times$	96.8	99.0	97.2	95.0	59	63
$\checkmark$	$\checkmark$	$\times$	97.8	99.4	98.2	97.0	67	66
$\checkmark$	$\checkmark$	$\checkmark$	99.6	100.0	98.8	97.8	72	69

TABLE X: Ablations on Self-Supervised Training Objectives in SS-DTPE. Sem. Align. denotes Semantic Progress Alignment, comprising Properties i) and ii) introduced in Sec. IV-B , while Temp. Struct. is Temporal Structure Learning.

Setting	LIBERO				Real-World
Setting	Spatial	Object	Goal	Long	Long.	Gen.
Hierarchical Modulation
w/o Inter.	96.8	99.4	97.8	95.4	58	61
w/o Intra.	97.4	99.6	97.4	95.8	62	64
Full HPIM	99.6	100.0	98.8	97.8	72	69
Layer-Wise Schedule

TABLE XI: Ablation on HPIM. The upper block evaluates the inter-progress gate and intra-progress relevance. w/o Inter. fixes

d_{t}=1

, applying local relevance at all execution stages; w/o Intra. replaces token-wise relevance with uniform future modulation. The lower block compares a uniform layer-wise schedule with our middle-emphasized schedule.

Setting	LIBERO				Real-World
Setting	Spatial	Object	Goal	Long	Long.	Gen.
w/o Progress Dynamics	98.0	98.4	97.0	96.8	59	61
Global Gate w/o Progress	97.2	98.0	96.6	96.0	57	58
Local Relevance w/o Progress	97.4	99.0	97.6	95.8	63	57
Full HPIM	99.6	100.0	98.8	97.8	72	69

TABLE XII: Ablations on Progress Conditioning in HPIM. w/o Progress Dynamics removes

\Delta Z_{t}^{\mathrm{prog}}

while retaining

Z_{t}^{\mathrm{prog}}

in the global gate. Global Gate w/o Progress removes both

Z_{t}^{\mathrm{prog}}

and

\Delta Z_{t}^{\mathrm{prog}}

from the global gate. Local Relevance w/o Progress removes progress conditioning from the local relevance branch. Other components are unchanged.

Modulation Path	LIBERO				Real-World
Modulation Path	Spatial	Object	Goal	Long	Long.	Gen.
Action-to-All-Visual	98.8	99.0	97.8	96.4	65	63
All-to-Future	98.4	98.8	97.4	96.6	64	67
Action-to-Future	99.6	100.0	98.8	97.8	72	69

TABLE XIII: Ablation Studies on Attention Modulation Paths. Action-to-All-Visual modulates action attention to both current and future visual tokens; All-to-Future modulates attention from all query tokens to future visual tokens; Action-to-Future only modulates attention from action tokens to future visual tokens, as adopted in our ProWAM.

Progress Source	LIBERO				Real-World
Progress Source	Spatial	Object	Goal	Long	Long.	Gen.
Time-Only	96.8	97.8	97.0	95.2	56	59
SS-DTPE (Ours)	99.6	100.0	98.8	97.8	72	69

TABLE XIV: Analysis on Progress Source. Time-Only uses normalized execution time as the progress signal, while SS-DTPE estimates progress from action-observation feedback and recurrent execution history.

Setting	LIBERO				Real-World
Setting	Spatial	Object	Goal	Long	Long.	Gen.
Effect of Gate Supervision
w/o Gate Supervision	97.2	98.0	96.4	95.6	59	61
w/ Gate Supervision	99.6	100.0	98.8	97.8	72	69
Robustness to Label Perturbation
Perturb. (5%)	99.6	99.8	98.8	97.4	70	69

TABLE XV: Analysis on Global-Gate Supervision. The upper block evaluates gate supervision, while the lower examines robustness to perturbations of the weak labels.

Fig. 6: Qualitative Visualization of Progress-Conditioned Attention Modulation. Compared with unmodulated attention, HPIM produces more concentrated responses around instruction-relevant objects and task-critical interaction regions across both main and wrist views.

Fig. 7: Qualitative Visualization of Progress Representations Learned by SS-DTPE. (a) Semantic progress estimates on held-out trajectories exhibit consistent and approximately monotonic evolution while preserving trajectory-specific variations. (b) Progress-token similarity shows strong local temporal coherence and lower similarity between temporally distant states.

Fig. 8: Qualitative Visualization of Inter-Progress Global Modulation. Representative stages from Clear Objects into Bin and Transfer Objects from Tray to Tray show that the global gate $d_{t}$ varies with execution progress, enabling stage-dependent modulation rather than fixed imagination utilization. Dashed arrows indicate temporal progression.

Fig. 9: Qualitative Comparison on Real-World Manipulation Tasks. Fast-WAM exhibits execution errors at critical interaction and transition stages, whereas ProWAM maintains more consistent task progression and successfully completes the subsequent stages. Red and green annotations highlight failure and successful execution events, respectively.

Fig. 10: Qualitative Visualization of Intra-Progress Future-Latent Relevance. $F_{1}\!:\!F_{2}$ denotes the relevance-mass split between the two future frames. The curves show cumulative relevance over ranked future latents, with the gray diagonal indicating uniform allocation; marked values report the mass captured by the top $20\%$ latents.

Fig. 11: Real-world Robotic Platforms Used in Our Experiments. Left: Galaxea R1 Lite. Right: AgileX Cobot Magic. The two platforms provide complementary mobile-manipulation embodiments for evaluating progress-aware long-horizon manipulation and generalization.

Fig. 12: Qualitative Simulation Results Across Five Benchmarks. Representative execution sequences of ProWAM on LIBERO, RoboTwin 2.0, Mikasa-Robo, VLABench, and RoboEval are shown in temporal order from left to right. The examples cover long-horizon object manipulation, bimanual coordination and handover, contact-rich interaction, memory-dependent control, fine-grained semantic grounding, and robustness to task variations.

Fig. 13: Qualitative Results on Real-World Manipulation Tasks. Representative execution sequences of ProWAM across eight progress-aware long-horizon tasks and four generalization and robustness tasks are shown in temporal order from left to right. The examples cover sequential object manipulation, bimanual coordination and handover, deformable-object interaction, multi-object transfer, fine-grained insertion, and out-of-distribution appearance and scene variations.

World Action Models

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 206 papers

Fast-WAM: Do World Action Models Need Test-time Future Imagination?
Mar 17, 2026Tianyuan Yuan, Zibin Dong, Yicheng Liu +1World Action Models Efficient Inference for World Action Models
IIIS, Tsinghua University · Galaxea AI
Latent Action as Intention Enables Efficient Future Imagination for World Action Models
Aug 25, 2026Xiang Li, Yupeng Zheng, Songen Gu +11Latent Action Learning World Models for Robotics
1CollegeAI, THU · 2AIR, THU · 4TARS Robotics +6
World Action Modeling with Progressive Visual Planning
Oct 1, 2026Fei Zhang, Zhaochong An, Duncan Frost +5World Model-Based Planning Robot Task Planning
Shanghai Jiao Tong University · SII · Meta +1

Mar 17, 2026J/K move · Enter open · S save

Fast-WAM: Do World Action Models Need Test-time Future Imagination?

Tianyuan Yuan, Zibin Dong, Yicheng Liu, Hang Zhao

IIIS, Tsinghua University · Galaxea AI

World Action Models Efficient Inference for World Action Models World Model Learning VLMs for Robotics

Open paper AskPDF arXiv

World Action Models (WAMs) have emerged as a promising alternative to Vision-Language-Action (VLA) models for embodied control because they explicitly model how visual observations may evolve under action. Most existing WAMs follow an imagine-then-execute paradigm, incurring substantial test-time latency from iterative video denoising, yet it remains unclear whether explicit future imagination is actually necessary for strong action performance. In this paper, we ask whether WAMs need explicit future imagination at test time, or whether their benefit comes primarily from video modeling during training. We disentangle the role of video modeling during training from explicit future generation during inference by proposing Fast-WAM, a WAM architecture that retains video co-training during training but skips future prediction at test time. We further instantiate several Fast-WAM variants to enable a controlled comparison of these two factors. Across these variants, we find that Fast-WAM remains competitive with imagine-then-execute variants, while removing video co-training causes a much larger performance drop. Empirically, Fast-WAM achieves competitive results with state-of-the-art methods both on simulation benchmarks (LIBERO and RoboTwin) and real-world tasks, without embodied pretraining. It runs in real time with 190ms latency, over 4

\times

faster than existing imagine-then-execute WAMs. These results suggest that the main value of video prediction in WAMs may lie in improving world representations during training rather than generating future observations at test time. Project page: https://yuantianyuan01.github.io/FastWAM/

Abstract

World Action Models

Explore similar work

Fast-WAM: Do World Action Models Need Test-time Future Imagination?

Latent Action as Intention Enables Efficient Future Imagination for World Action Models

World Action Modeling with Progressive Visual Planning