What Matters for World Modeling in Vision-Language-Action Models?

Qisen Ma; Yixiang Chen; Yuan Xu; Peiyan Li; Jiabing Yang; Kai Wang; Jianhua Yang; Jianlou Si

cs.ROOct 14, 2026

What Matters for World Modeling in Vision-Language-Action Models?

Authors: Qisen Ma, Yixiang Chen, Yuan Xu, Peiyan Li, Jiabing Yang, Kai Wang, Jianhua Yang, Jianlou Si,

Organizations: New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Alibaba Group · FiveAges

Abstract

Integrating world models into Vision-Language-Action models (VLAs) has emerged as a promising direction, enabling policies to anticipate future dynamics beyond reactive observation-to-action mapping. However, existing approaches vary simultaneously in their world representations, integration paradigms, and policy backbones, making it difficult to determine what truly matters for effective world model integration. To disentangle these factors, we categorize existing efforts into four representative paradigms, namely Implicit Future Alignment, Explicit Future Query, MoT Future Expert, and External World Predictor. We instantiate each within a unified framework and systematically evaluate these paradigms with diverse world representations and policy backbones across in-domain, out-of-domain, long-horizon, dynamic, few-shot, and real-world settings. Our study yields four main findings: 1) world representation effectiveness is paradigm-dependent, whereas 2D semantic representations offer robust, cross-paradigm gains; 2) no single paradigm is universally optimal, yet the decoupled MoT Future Expert offers competitive overall performance, particularly in OOD robustness; 3) world modeling consistently benefits policies built on standard VLMs, while decoupled designs exhibit greater compatibility with pretrained VLAs; and 4) world modeling improves performance over action-only baselines across multiple settings, with notable gains on long-horizon and dynamic tasks. Project page and video demonstrations are available at https://dylanmqs.github.io/WM-in-VLAs/.

Figures & tables

Figure 1: Systematic empirical investigation of world modeling in vision-language-action policies across four core dimensions, spanning world representation, integration paradigms, backbone generalization, and comprehensive performance across diverse settings.

Figure 2: Overview of the four representative world modeling integration paradigms for vision-language-action models: (a) Implicit Future Alignment, (b) Explicit Future Query, (c) MoT Future Expert, and (d) External World Predictor, categorized by how world dynamics are conditioned on and decoupled from the policy.

Figure 3: World-representation performance across integration paradigms. Success rate gains over the baseline on LIBERO (ID, top) and LIBERO-Plus (OOD, bottom). In (a, c), horizontal bars and diamonds indicate the min–max range and mean across the four paradigms, respectively. In (b, d), heatmaps display individual representation–paradigm gains.

Method	LIBERO	LIBERO-Plus
Method	LIBERO	Camera	Robot	Language	Light	Background	Noise	Layout	Avg.
OpenVLA-OFT	97.1	56.4	31.9	79.5	88.7	93.3	75.8	74.2	69.6
Fast-WAM	97.6	18.8	45.7	70.1	83.2	45.7	29.8	62.7	49.1
4D-WAM	98.6	45.2	64.3	90.6	94.3	57.7	69.1	79.2	71.0
Baseline	96.7	48.2	51.4	91.4	96.0	92.9	65.3	71.3	71.8
WM-Align	98.6 ( $\uparrow$ 1.9)	58.0 ( $\uparrow$ 9.8)	49.7 ( $\downarrow$ 1.7)	89.5 ( $\downarrow$ 1.9)	94.6 ( $\downarrow$ 1.4)	93.5 ( $\uparrow$ 0.6)	71.8 ( $\uparrow$ 6.5)	73.9 ( $\uparrow$ 2.6)	74.1 ( $\uparrow$ 2.3)

Table 1: LIBERO and LIBERO-Plus performance across world modeling paradigms. Scores report success rates (%), with parenthesized arrows indicating gains over the action-only baseline. Bold and underline denote best and second-best results, respectively.

Method	Clean2Clean				Clean2Random
Method	Short	Medium	Long	Avg.	Short	Medium	Long	Avg.
starVLA	49.7	51.9	28.8	46.5	4.0	3.2	0.6	3.2
AHA-WAM	60.3	76.7	59.1	64.3	2.6	5.8	1.2	3.2
GalaxeaVLA	63.0	71.6	48.9	62.7	11.5	19.6	6.7	12.7
Baseline	53.4	65.0	30.7	52.3	13.4	15.4	6.6	12.7
WM-Align	62.0 ( $\uparrow$ 8.6)	72.4 ( $\uparrow$ 7.4)	48.8 ( $\uparrow$ 18.1)	62.3 ( $\uparrow$ 10.0)	15.3 ( $\uparrow$ 1.9)	17.9 ( $\uparrow$ 2.5)	8.9 ( $\uparrow$ 2.3)	14.8 ( $\uparrow$ 2.1)

Table 2: RoboTwin 2.0 performance under Clean2Clean and Clean2Random evaluation. Scores report success rates (%), with parenthesized arrows indicating gains over the action-only baseline. Horizons are categorized by average demonstration length: short (

\leq 200

), medium (

200

–

300

), and long (

>300

steps). Bold and underline denote best and second-best results, respectively.

Figure 4: Data scaling on LIBERO and LIBERO-Plus. Absolute success rates using 25%, 50%, and 100% of the training demonstrations.

Method	Clean2Clean				Clean2Random
Method	Short	Medium	Long	Avg.	Short	Medium	Long	Avg.
$\pi_{0.5}$	67.9	72.5	68.2	69.1	45.6	53.3	39.0	46.4
$\pi_{0.5}$ + WM-Align	64.1 ( $\downarrow$ 3.8)	66.8 ( $\downarrow$ 5.7)	57.6 ( $\downarrow$ 10.6)	63.6 ( $\downarrow$ 5.5)	39.9 ( $\downarrow$ 5.7)	47.2 ( $\downarrow$ 6.1)	35.9 ( $\downarrow$ 3.1)	41.0 ( $\downarrow$ 5.4)
$\pi_{0.5}$ + WM-Query	67.8 ( $\downarrow$ 0.1)	72.4 ( $\downarrow$ 0.1)	66.0 ( $\downarrow$ 2.2)	68.7 ( $\downarrow$ 0.4)	45.4 ( $\downarrow$ 0.2)	50.7 ( $\downarrow$ 2.6)	42.3 ( $\uparrow$ 3.3)	46.2 ( $\downarrow$ 0.2)
$\pi_{0.5}$ + WM-MoT	69.6 ( $\uparrow$ 1.7)	76.8 ( $\uparrow$ 4.3)	68.6 ( $\uparrow$ 0.4)	71.3 ( $\uparrow$ 2.2)	49.6 ( $\uparrow$ 4.0)	51.5 ( $\downarrow$ 1.8)	47.2 ( $\uparrow$ 8.2)	49.7 ( $\uparrow$ 3.3)
$\pi_{0.5}$ + WM-Ext	71.5 ( $\uparrow$ 3.6)	74.9 ( $\uparrow$ 2.4)	67.9 ( $\downarrow$ 0.3)	71.7 ( $\uparrow$ 2.6)	48.1 ( $\uparrow$ 2.5)	52.4 ( $\downarrow$ 0.9)	44.1 ( $\uparrow$ 5.1)	48.5 ( $\uparrow$ 2.1)

Table 4:

\pi_{0.5}

performance on RoboTwin 2.0 under Clean2Clean and Clean2Random evaluation. Scores report success rates (%), with parenthesized arrows indicating gains over the

\pi_{0.5}

baseline. Bold and underline denote best and second-best results, respectively.

Figure 5: Real-world experimental setup and manipulation tasks.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Training & Optimization
Hyperparameter / Component	Specification
Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.95$ , $\epsilon=10^{-8}$ )
Learning rate schedule	Cosine annealing with 5,000-step linear warmup
Peak learning rate	$1\times 10^{-5}$ (VLM), $1\times 10^{-4}$ (action/future modules)
Batch size	16 per GPU; 128 global (8 GPUs, no accumulation)
Precision	Mixed BF16

Appendix

Table A.1: Hyperparameters and architectural configurations for the Qwen-based LIBERO main comparison and compute-cost measurements.

Method	Train Params.	50k Train Time	Deploy Params.	Policy	Encode	E2E
	(B)	(h [GPU-h])	(B)	(ms)	(ms)	(ms)
Baseline	2.608	8.71 (69.7)	2.608	103.3	–	103.3
WM-Align	2.629	9.13 (73.0)	2.608	103.0	–	103.0
WM-Query	2.613	9.04 (72.3)	2.608	105.4	–	105.4
WM-MoT	2.866	10.53 (84.2)	2.952	129.8	6.3	136.1
WM-Ext	2.908	9.59 (76.7)	2.994	112.9	6.3	119.3

Appendix

Table A.2: Computational and deployment efficiency across world modeling paradigms (

8\times

NVIDIA H100). Training time is measured over 50k steps. Deployment parameters include the frozen DINOv3 encoder (85.7M) when required. Latencies are per-step batch-1 medians: “Policy” measures policy forward pass, “Encode” isolates online visual feature extraction, and “E2E” denotes end-to-end latency.

1. Out-of-Domain Superiority of WM-MoT
Finding / Evaluation Setting	Benchmark	Seed 1 (Default)	Seed 2	Seed 3	Mean $\pm$ Std
Action-Only Baseline	LIBERO-Plus (OOD)	71.8%	71.4%	71.1%	71.4 $\pm$ 0.4%
WM-MoT (DINOv3)	LIBERO-Plus (OOD)	76.2%	77.4%	75.5%	76.4 $\pm$ 1.0%
2. Representation Ranking Stability (DINOv3 vs. SigLIP)
WM-MoT + DINOv3	LIBERO-Plus (OOD)	76.2%	77.4%	75.5%	76.4 $\pm$ 1.0%
WM-MoT + SigLIP	LIBERO-Plus (OOD)	74.3%	73.8%	74.6%	74.2 $\pm$ 0.4%

Appendix

Table B.1: Multi-seed training verification across central positive and negative findings. All models are trained across three independent seeds. Results report individual run outcomes alongside the Mean

\pm

Std aggregated over the full benchmark suites.

Action Baselines and Capacity Ablations ( $\lambda_{\mathrm{world}}=0$ )
Method / Paradigm	Current DINOv3	LIBERO-Plus
Method / Paradigm	Feature Stream	Camera	Robot	Language	Light	Background	Noise	Layout	Average
Action-Only Baseline	No	48.2	51.4	91.4	96.0	92.9	65.3	71.3	71.8
Action-Only Baseline (+ DINOv3 stream)	Yes	49.9	51.7	91.3	96.1	93.3	67.0	71.7	72.5
WM-Ext ( $\lambda_{\mathrm{world}}=0$ , w/o world loss)	Yes	47.5	46.0	90.2	95.5	93.8	68.5	70.8	71.1
WM-MoT ( $\lambda_{\mathrm{world}}=0$ , w/o world loss)	Yes	56.5	52.7	89.7	95.2	93.6	69.1	71.2	73.6

Appendix

Table B.2: Controlled evaluation of auxiliary DINOv3 current-frame input on LIBERO-Plus (OOD). All paradigms use DINOv3 ViT-B/16. Policies are evaluated with versus without the frozen DINOv3 current-frame feature stream fused into action generation. Success rates (%) are reported across the seven perturbation dimensions and on average.

Paradigm	Input $E_{\text{in}}$	Target $E_{\text{target}}$	ID (%)	OOD (%)	$\Delta_{\text{OOD}}$
WM-MoT	DINOv3	DINOv3 (Same)	97.9%	76.4 $\pm$ 1.0%	–
	DINOv3	V-JEPA 2.1	97.8%	75.8 $\pm$ 0.7%	-0.6%
	V-JEPA 2.1	DINOv3	97.6%	74.7 $\pm$ 1.3%	-2.9%
	V-JEPA 2.1	V-JEPA 2.1 (Same)	97.9%	77.6 $\pm$ 0.6%	–
WM-Ext	DINOv3	DINOv3 (Same)	98.2%	74.6 $\pm$ 0.4%	–
	DINOv3	V-JEPA 2.1	97.8%	71.1 $\pm$ 0.6%	-3.5%

Appendix

Table B.3: Factorial isolation of current-input versus future-supervisory encoders. Evaluated on LIBERO (ID, 2,000 episodes) and LIBERO-Plus (OOD, 10,030 episodes). Success Rate (%). Results reported as 3-seed Mean

\pm

Std for OOD. “Same” denotes that the supervisory target uses the same encoder as the input (

E_{\text{target}}=E_{\text{in}}

\Delta_{\text{OOD}}

is computed relative to the respective baseline with identical input and target encoders.

Configuration	Supervision Layer	$\lambda_{\mathrm{world}}$	C2C (%)	$\Delta_{\text{C2C}}$ vs. Base	C2R (%)	$\Delta_{\text{C2R}}$ vs. Base
$\pi_{0.5}$ Action Baseline	–	–	69.1%	–	46.4%	–
Varying Loss Weight $\lambda$	Layer 12 (Default)	0.01	67.8%	-1.3%	45.2%	-1.2%
	Layer 12 (Default)	0.05	65.4%	-3.7%	43.1%	-3.3%
	Layer 12 (Default)	0.10	63.6%	-5.5%	41.0%	-5.4%
	Layer 12 (Default)	0.20	60.2%	-8.9%	37.8%	-8.6%
Varying Supervision Depth	Layer 6 (Intermediate)	0.10	60.8%	-8.3%	38.5%	-7.9%

Appendix

Table B.4: Hyperparameter sensitivity of

\pi_{0.5}

+ WM-Align on RoboTwin 2.0 (Success Rate %).

Configuration	Query Count $N_{q}$	$\lambda_{\mathrm{world}}$	Suite Avg. (%)	Goal Avg. (%)	$\Delta$ vs. 25% Base
25% Action Baseline	–	–	84.9%	77.6%	–
Varying Query Tokens $N_{q}$	$N_{q}=4$	0.10	78.4%	56.2%	-6.5% (Suite) / -21.4% (Goal)
	$N_{q}=8$	0.10	74.2%	41.5%	-10.7% (Suite) / -36.1% (Goal)
	$N_{q}=16$ (Default)	0.10	70.1%	28.8%	-14.8% (Suite) / -48.8% (Goal)
	$N_{q}=32$	0.10	62.5%	18.2%	-22.4% (Suite) / -59.4% (Goal)
Varying Loss Weight $\lambda$	$N_{q}=16$	0.05	73.8%	38.0%	-11.1% (Suite) / -39.6% (Goal)

Appendix

Table B.5: Hyperparameter sensitivity of WM-Query under 25% data on LIBERO (Success Rate %).

Methods	Spatial	Object	Goal	Long	Average
Baseline	97.6	98.8	97.4	92.8	96.7
WM-Align + DINOv3	99.2	98.8	98.0	98.2	98.6
WM-Align + SigLIP	98.6	99.0	97.4	95.0	97.5
WM-Align + VGGT	99.6	99.4	98.2	95.2	98.1
WM-Align + ImageVAE	99.0	98.8	98.4	95.6	98.0
WM-Align + WanVAE	99.0	98.2	98.6	95.4	97.8

Appendix

Table C.1: Detailed in-domain LIBERO success rates (%).

Methods	Camera	Robot	Language	Light	Background	Noise	Layout	Average
Baseline	48.2	51.4	91.4	96.0	92.9	65.3	71.3	71.8
WM-Align + DINOv3	58.0	49.7	89.5	94.6	93.5	71.8	73.9	74.1
WM-Align + SigLIP	55.4	52.3	90.1	95.4	93.2	69.6	72.1	73.7
WM-Align + VGGT	57.3	48.5	88.5	96.2	94.1	70.6	71.5	73.4
WM-Align + ImageVAE	56.0	51.4	86.8	93.7	89.9	75.4	74.4	73.8
WM-Align + WanVAE	46.1	48.1	91.0	96.1	95.1	71.3	73.4	72.4

Appendix

Table C.2: Detailed LIBERO-Plus success rates (%) under seven perturbations.

Data	Model	Spatial	Object	Goal	Long	Average
25%	Baseline	89.4	94.2	77.6	78.2	84.9
	WM-Align	87.6	92.4	70.4	78.4	82.2
	WM-Query	85.0	94.6	28.8	72.0	70.1
	WM-MoT	89.0	93.2	85.2	77.0	86.1
	WM-Ext	93.2	93.4	86.8	77.4	87.7
50%	Baseline	97.4	99.4	87.2	92.0	94.0

Appendix

Table C.3: Detailed LIBERO data-scaling success rates (%). Within each training-data fraction, best and second-best results are bolded and underlined, respectively.

Data	Model	Camera	Robot	Language	Light	Background	Noise	Layout	Average
25%	Baseline	37.7	32.4	75.2	82.8	79.1	52.1	57.6	57.5
	WM-Align	35.6	29.9	78.1	86.6	77.3	60.1	56.5	58.6
	WM-Query	26.4	27.0	69.6	75.8	63.0	41.5	46.3	48.1
	WM-MoT	41.7	37.5	80.9	87.6	84.3	59.8	59.0	62.4
	WM-Ext	37.4	28.9	76.3	81.8	73.8	56.2	55.5	56.8
50%	Baseline	46.2	41.9	84.0	87.9	85.9	53.6	67.0	64.7

Appendix

Table C.4: Detailed LIBERO-Plus data-scaling success rates (%) under seven perturbations. Within each training-data fraction, best and second-best results are bolded and underlined, respectively.

Task	Clean2Clean					Clean2Random
Task	Baseline	WM-Align	WM-Query	WM-MoT	WM-Ext	Baseline	WM-Align	WM-Query	WM-MoT	WM-Ext
Short-horizon tasks
adjust_bottle	100	100	100	100	99	8	13	3	4	3
beat_block_hammer	58	87	84	67	69	1	6	9	2	2
click_alarmclock	9	3	2	2	2	71	63	47	61	55
click_bell	3	19	8	33	11	64	61	41	54	38

Appendix

Table C.5: Per-task success rates (%) for the action-only baseline and world-modeling variants on RoboTwin 2.0. Group and overall averages correspond to the respective rows in Table 2 .

Task	Clean2Clean					Clean2Random
Task	$\pi_{0.5}$	WM-Align	WM-Query	WM-MoT	WM-Ext	$\pi_{0.5}$	WM-Align	WM-Query	WM-MoT	WM-Ext
Short-horizon tasks
adjust_bottle	100	100	100	100	100	90	90	96	93	97
beat_block_hammer	83	89	92	92	90	29	21	25	37	27
click_alarmclock	63	50	49	56	58	57	50	56	52	60
click_bell	59	24	44	46	57	44	25	34	46	48

Appendix

Table C.6: Per-task success rates (%) for the

\pi_{0.5}

experiments on RoboTwin 2.0 under Clean2Clean and Clean2Random evaluation. Group and overall averages correspond to Table 4 .

Task	Baseline		WM-Align		WM-Query		WM-MoT		WM-Ext
Task	SR	MS	SR	MS	SR	MS	SR	MS	SR	MS
adjust_bottle	63	77.60	59	74.60	51	71.40	69	80.80	63	76.50
beat_block_hammer	9	21.70	7	18.00	4	16.60	6	18.40	8	21.70
click_alarmclock	12	18.00	11	17.00	11	16.80	14	18.60	16	21.30
click_bell	2	11.90	0	11.20	1	10.70	0	9.00	0	11.00
dump_bin_bigbin	19	35.00	28	38.80	12	33.00	29	43.00	34	46.60

Appendix

Table C.7: Per-task performance on DOMINO Level 1 under the clean dynamic setting, reported as success rate (SR) and manipulation score (MS).

Method	Static Tasks			Dynamic Tasks			Overall
Method	Box Cover Block	Place Corn on Rack	Avg.	Pick Tomato	Place Tomato on Plate	Avg.	Overall
Baseline	62.5	60.0	61.3	37.5	20.0	28.8	45.0
WM-Align	77.5	85.0	81.3	47.5	25.0	36.3	58.8
WM-Query	67.5	65.0	66.3	57.5	35.0	46.3	56.3
WM-MoT	87.5	77.5	82.5	75.0	55.0	65.0	73.8
WM-Ext	90.0	67.5	78.8	70.0	42.5	56.3	67.5

Appendix

Table C.8: Real-world manipulation success rates (%) on the physical Franka Research 3. Each task is evaluated over 40 trials. Bold and underline denote the best and second-best results, respectively.

Figure 6: Qualitative real-robot rollouts for Box Cover Block . Each row shows one selected rollout, with its method and outcome indicated on the left.

Figure 7: Qualitative real-robot rollouts for Place Corn on Rack . Each row shows one selected rollout, with its method and outcome indicated on the left.

Figure 8: Qualitative real-robot rollouts for Pick Tomato . Each row shows one selected rollout, with its method and outcome indicated on the left.

Figure 9: Qualitative real-robot rollouts for Place Tomato on Plate . Each row shows one selected rollout, with its method and outcome indicated on the left.

World Models for VLA Models

See all 28 papers

Towards VLA-Dreamer: Refining VLA Behavior Using World Models
Sep 25, 2026Parsa Mastouri Kashani, Jan-Gerrit Habekost, Stefan WermterWorld Models for VLA Models Vision-Language-Action Models
Knowledge Technology, Dept. of Informatics, University of Hamburg, Germany
Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies
Sep 21, 2026Trung Dao, Sankalp Yamsani, Jaden Park +2World Models Robotic Manipulation
University of Wisconsin–Madison, USA. · University of Illinois Urbana-Champaign, USA.
AWM-VLA: AlignedWorld Modeling for Efficient and Explainable Vision-Language-Action Policies
Aug 15, 2026An Lanji, Dawei Liu, Jin Li +3World Models for VLA Models Vision-Language-Action Models
University of Electronic Science and Technology of China, Chengdu, China.

Sep 25, 2026J/K move · Enter open · S save

Towards VLA-Dreamer: Refining VLA Behavior Using World Models

Parsa Mastouri Kashani, Jan-Gerrit Habekost, Stefan Wermter

Knowledge Technology, Dept. of Informatics, University of Hamburg, Germany

World Models for VLA Models Vision-Language-Action Models Efficient VLA Models World Model Learning Embodied World Models Model-Based Planning

Open paper AskPDF arXiv

Vision-Language-Action models (VLAs), while showing strong potential for robot control, require massive amounts of high-quality imitation learning data. Moreover, the absence of an explicit world model casts further doubt on their control capabilities. In this concept paper, we propose a novel architecture that addresses sample efficiency in VLAs by training a predictive world model on the embedding space of the VLA's vision encoder. We hypothesize that these embeddings are action-relevant and usable for future prediction. To this end, we propose using the suggested architecture to investigate how well these embeddings predict the future based on actions, as the inability to do so would mark a key limitation of VLA architectures: the lack of a non-lossy implicit world model to simulate real-world dynamics. The proposed architecture differs from the standard world model dynamics as the loss comes from the embedding space rather than the pixel space, similar to joint embedding predictive architectures. Furthermore, the trained world model can be utilized for short-term planning tasks by sampling VLA actions given goal images. We intend to examine the richness of vision embeddings in VLAs and reduce their high data requirements through a world model that can also generate plans during inference.

Abstract

World Models for VLA Models

Explore similar work

Towards VLA-Dreamer: Refining VLA Behavior Using World Models

Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies

AWM-VLA: AlignedWorld Modeling for Efficient and Explainable Vision-Language-Action Policies