UniAction: Unified Action Conditioning for World Models

Bowen Xue; Zizhang Li; Hong-Xing Yu; Jiajun Wu

cs.CVOct 23, 2026

UniAction: Unified Action Conditioning for World Models

Authors: Bowen Xue, Zizhang Li, Hong-Xing Yu, Jiajun Wu

Organizations: Stanford University

Abstract

Actions that condition world models are highly heterogeneous, and thus existing action-conditioned world models are typically built for individual action types, fragmenting already limited action-annotated data across separate models. We introduce UniAction, a unified action-conditioning framework that represents any action as a sparse set of action elements, each with a discrete semantic identity and a continuous spatiotemporal grounding coordinate. By mapping these coordinates directly into the pretrained video model's native positional space, UniAction allows heterogeneous actions to condition a shared world model within the same spatiotemporal structure. We further introduce cardinality-calibrated attention to address the bias from differing action representation granularities. We evaluate UniAction on four representative action forms: force, camera control, hand pose, and robot pose. UniAction outperforms conditioning alternatives in action following, and a single jointly trained model surpasses the action-specific experts, showing that heterogeneous action data benefit from unified model and joint training.

Figures & tables

Figure 1: One world model for heterogeneous actions. Instead of training a separate model per action type (left), UniAction conditions a single world model (middle) on heterogeneous actions and outperforms the action-specific experts across all four action forms (right).

Figure 2: Overview of UniAction. Diverse action forms are represented as sparse action elements: semantic identities index tokens in a shared codebook (Sec. 3.1 ), spatiotemporal positions use the backbone’s native positional encoding (Sec. 3.2 ), and action and video tokens interact via cardinality-calibrated attention (Sec. 3.3 ).

Action Form	Method	Automatic Evaluation		Human Evaluation
Action Form	Method	$\vbox{\hbox{\shortstack[c]{Action\\ Following}}}\,\uparrow$	$\vbox{\hbox{\shortstack[c]{LPIPS}}}\,\downarrow$	$\vbox{\hbox{\shortstack[c]{Action\\ Following}}}\,\uparrow$	$\vbox{\hbox{\shortstack[c]{Visual\\ Quality}}}\,\uparrow$
	VACE	0.6046	0.1178	2.47	3.40
	Cross Attention	0.1618	0.1341	1.96	3.13
Force	UniAction (Ours)	0.7630	0.1163	4.37	4.00
	VACE	0.4997	0.5100	3.30	3.55
	Cross Attention	0.5306	0.5124	3.13	3.35

Table 1: Comparison with visual-based (VACE ( Jiang et al., 2025 ) ) and token-based (Cross-Attn ( NVIDIA, 2026b ) ) conditioning mechanisms across four action forms. Best and second-best results are bold and underlined .

Figure 3: Qualitative comparison across four action forms. UniAction follows the input actions more faithfully than other conditioning mechanisms. Overlays indicate the input action.

Figure 4: Qualitative comparison with action-specific experts. Compared with separate training, the jointly trained UniAction follows the input actions more faithfully and produces more plausible dynamics and visual appearance. Overlays indicate the input action.

Action	Action Following $\uparrow$		LPIPS $\downarrow$
Action	Expert	Ours	Expert	Ours
Force	0.5436	0.7630	0.1295	0.1163
Camera	0.5182	0.6371	0.4780	0.4863
Hand	0.3701	0.4119	0.0934	0.0910
Robot	0.6310	0.6521	0.1286	0.1164

Table 2: Jointly trained UniAction vs. action-specific experts. Best in bold .

Variant	Action Following $\uparrow$
Variant	Force	Camera	Hand	Robot
Self-attn $\to$ Cross-attn	0.2975	0.5050	0.0533	0.0733
RoPE $\to$ Sinusoidal APE	0.2084	0.5278	0.0603	0.0755
RoPE $\to$ Learned APE	0.1485	0.5206	0.0656	0.0693
w/o Cardinality Calib.	0.5186	0.4966	0.4303	0.6995
UniAction (Ours)	0.7190	0.5842	0.4266	0.7004

Table 3: Ablations of spatiotemporal grounding and cardinality calibration. Each variant changes one component (

A\!\to\!B

A

replaced by

B

). Best in bold , second-best underlined .

Figure 5: Generalization. Both models are trained only on force data, with all forces applied at $t=0$ ; columns show test-time forces at different $t$ . Trajectories are colored by frame.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Action Form	Construction	#Conditions
Force	8 ball scenes $\times$ 5; 16 domino scenes $\times$ 2; 16 flower scenes $\times$ 2	104
Camera Control	13 scenes $\times$ 8 directions ( W/A/S/D/I/J/K/L )	104
Hand Pose	20 samples $\times$ {original, stationary, horizontal flip, vertical flip, flip along both axes}	100
Robot Pose	34 samples $\times$ {original, horizontally flipped center trajectory, reversed gripper trajectory}	102
Total		410

Appendix

Table 4: Construction of the test set.

Method	Params	$\Delta$ TFLOPs				$\Delta$ Latency (%)
Method	Params	Force	Camera	Hand	Robot	Force	Camera	Hand	Robot
VACE	3.1B	351.2	351.2	351.2	351.2	21.3	21.0	21.2	21.1
Cross Attention	640M	33.5	35.9	36.0	36.0	8.8	8.6	9.0	9.0
UniAction	404M	0.1	5.9	72.9	37.2	3.6	3.0	6.6	4.9

Appendix

Table 5: Additional cost of action conditioning per denoising forward pass, relative to the backbone without action input.

Panel of Figure 6 — Figure 6: Additional qualitative comparisons. The camera input in (b) is W (forward camera motion). All overlays indicate input conditions.

Panel of Figure 7 — Figure 7: Additional qualitative comparisons. Overlays indicate input conditions.

Panel of Figure 8 — Figure 8: Additional qualitative comparisons. Overlays indicate input conditions.

Figure 9: Temporal generalization on dominoes.

Figure 10: Temporal generalization on balls.

Figure 11: Scene and appearance generalization across four action forms. All six variants in (c) share the same hand-pose input. Arrows and highlighted keys indicate the input actions.

Figure 12: Generalization to unseen action compositions. Each block (a)–(c) shows one video in three rows of six selected frames, read from left to right and then from top to bottom. Arrows, pose overlays, and highlighted keys indicate the input actions.

Action-Conditioned World Models

Action-conditioned world models predict how environments change under actions to support planning, control, and simulation.

See all 337 papers

Can World and Action Modeling Benefit Each Other under Unified Causal Training?
Oct 15, 2026Yue Su, Bokai Lin, Xinkai Wang +9Action-Conditioned World Models World Model Learning
IAAI Team · SII · THU +1
AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps
Apr 13, 2026Liaoyuan Fan, Zetian Xu, Chen Cao +3World Action Models Robotic Manipulation
INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China · The University of Hong Kong, Hong Kong SAR, China · Shanghai Jiao Tong University, Shanghai, China
CAER: Causal Action Effect Reweighting for World Model Training
Aug 31, 2026Jianjie Fang, Xvyuan Liu, Ziyou Wang +9Action-Conditioned Video Generation World Model Learning
1Tsinghua University · University of Science and Technology of China · 2Manifold AI

Oct 15, 2026J/K move · Enter open · S save

Can World and Action Modeling Benefit Each Other under Unified Causal Training?

Yue Su, Bokai Lin, Xinkai Wang, Chubin Zhang+8

IAAI Team · SII · THU+1

Action-Conditioned World Models World Model Learning Visuomotor Policy Learning Robotic Manipulation

Open paper AskPDF arXiv

World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear. We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture. Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling. We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments. Learning the visual consequences of specified actions improves policy performance and generalization across changing scenes. In turn, action learning improves multi-view world-prediction fidelity under the same architecture. Together, these results answer both questions positively, supporting reciprocal benefits between world modeling and action generation under unified causal training.

Abstract

Action-Conditioned World Models

Explore similar work

Can World and Action Modeling Benefit Each Other under Unified Causal Training?

AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps

CAER: Causal Action Effect Reweighting for World Model Training