VIDEAS: Distilling Explicit Action Semantics from Demonstration Videos for World Models via Prior-Guided Simulation

Jianan Wang; Haoquan Zhai; Siyang Zhang; Bin Li; Juan Chen; Jingtao Qi; Zhuo Zhang; Enze Wang

cs.ROSep 27, 2026

VIDEAS: Distilling Explicit Action Semantics from Demonstration Videos for World Models via Prior-Guided Simulation

Authors: Jianan Wang, Haoquan Zhai, Siyang Zhang, Bin Li, Juan Chen, Jingtao Qi, Zhuo Zhang, Enze Wang, +2 more

Organizations: College of Computer Science and Technology, National University of Defense Technology · School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences · Intelligent Game and Decision Lab (IGDL) · School of Artificial Intelligence, Shanghai Jiao Tong University

Abstract

World models learn internal representations of environment dynamics to predict future states, enabling agents to optimize action plans without physical interactions. However, developing world models that genuinely internalize underlying causal physical laws to explicitly reason about action preconditions and subsequent state transitions remains an open challenge. In this paper, we propose VIDEAS, a data distillation framework that transforms continuous physical dynamics from operational videos into explicit action semantics for foundation models. Specifically, it deconstructs visual demonstrations into discrete action trajectories and utilizes advanced vision-language models (VLMs) to extract structured knowledge encapsulating action preconditions and effects. To ensure physical consistency, we introduce a prior-guided trajectory simulation mechanism grounded within a text-based environment to rigorously validate the extracted knowledge. Notably, we incorporate negative trajectories to enrich knowledge completeness and enhance data diversity to mitigate cognitive bias. Furthermore, we present VIDEAS-WM, an 8B/9B-parameter suite of language-based world models trained on 34K high-quality samples derived from AgiBot-World dataset. Extensive experiments demonstrate that VIDEAS-WM establishes state-of-the-art performance in high-level embodied action semantic reasoning, exhibiting profound physical understanding and robust generalization across unseen scenarios.

Figures & tables

Figure 1: Conventional world models vs. VIDEAS-WM. For an infeasible action, the former ignores action preconditions and hallucinates successful state transitions, while VIDEAS-WM verifies preconditions before predicting environment evolution, correctly judging the action infeasible.

Figure 2: Overview of VIDEAS. Based on in-the-wild demonstration videos, VIDEAS first curates them into discrete action trajectories and grounds them into text-based structured environments, where it leverages VLMs to iteratively simulate each action via Consider–Decide–Transfer phases under execution priors. Moreover, it constructs negative trajectories via dependency-aware mutation to enrich precondition knowledge. Finally, the distilled 34K-sample corpus serves to train VIDEAS-WM, an 8B/9B-parameter suite of language-based world models.

Figure 3: Training corpus statistics. In (b), Succ. and Neg. denote samples derived from successful and negative trajectories respectively. (c) shows representative action types.

Model	BTSIMBENCH					LogicEnvEval
Model	Delivery	Good	CFactuals	Unreach	Avg	Delivery	Good	CFactuals	Unreach	LBranch	Avg
Closed-Source Models
Gemini-3.1-Pro	100.00	92.00	64.00	96.00	84.00	100.00	92.00	72.00	92.00	96.00	88.00
Gemini-3-Flash	100.00	88.00	64.00	88.00	80.00	100.00	76.00	76.00	88.00	96.00	84.00
Open-Source Models
DeepSeek-V4-Pro	100.00	44.00	88.00	76.00	69.33	100.00	52.00	84.00	88.00	96.00	80.00

Table 1: Performance (%) on BTSIMBENCH and LogicEnvEval benchmarks. Each policy category contains 25 samples. Gray and green rows represent closed-source baselines and VIDEAS-WM, respectively. The best performance among open-source models is in bold .

	Model	WorldPrediction-WM
	Model	COIN	CrossTask	IKEA-ASM	EPIC-KITCHENS-100	Overall
VLMs	InternVL3.5-38B	47.69	55.56	40.88	73.44	54.87
	InternVL3.5-14B	47.18	45.56	35.85	51.56	45.44
	InternVL3.5-8B	45.13	44.44	32.08	40.63	40.41
	Qwen3.5-27B	51.28	55.56	47.80	67.19	55.82
	Qwen3-VL-32B	53.85	54.44	45.28	69.27	56.45

Table 2: Performance (%) on WorldPrediction-WM benchmark. Green rows represent our VIDEAS-WM. The best performance within each category (VLMs and Socratic LLMs) is in bold .

Model	Method	BTSIMBENCH
Model	Method	Good	CFactuals	Unreach	Avg
Llama3.1-8B	Base	0.00	76.00	8.00	28.00
	Naive Distillation	72.00	32.00	88.00	64.00
	W/O NTI	88.00	28.00	100.00	72.00
	VIDEAS-WM	92.00	96.00	96.00	94.67
Qwen3-8B	Base	12.00	80.00	32.00	41.33

Table 3: Results of ablation study on the prior-guided trajectory simulation mechanism (Naive Distillation) and the negative trajectory integration (W/O NTI).

Figure 4: Ablation study results on training data scales for the BTSIMBENCH benchmark.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Training epochs	3
Learning rate	$1\times 10^{-4}$
Warmup ratio	0.03
Batch size	4
Gradient accumulation steps	1
LoRA rank	32

Appendix

Table 4: Training hyperparameters.

Dataset	Number of Samples
COIN	195
CrossTask	90
IKEA-ASM	159
EPIC-KITCHENS-100	193
Total	637

Appendix

Table 5: Distribution of accessible task samples in WorldPrediction-WM.

Figure 5: Pareto frontier of accuracy versus average simulation time per robot policy. VIDEAS-WM establishes the optimal balance between reasoning capability and computational efficiency.

Figure 6: An illustrative example of the text-based environment. The complete JSON structure encompasses comprehensive fine-grained properties, which are abbreviated here for brevity.

Panel of Figure 7 — Figure 7: The prompt structure and expected output for the consider stage.

Panel of Figure 8 — Figure 8: The prompt structure and expected output for the decide stage.

Panel of Figure 9 — Figure 9: The prompt structure and expected output for the capture stage.

Panel of Figure 10 — Figure 10: The prompt structure and expected output for the transfer stage.

Figure 11: An illustrative example of negative trajectories. By removing the initial Open action, the subsequent Pick action is deliberately designed to fail due to unmet physical preconditions.

Action Sequence	Predicted Preconditions & Effects (JSON Structure)
(1) Open_cabinet : The robot grips the cabinet door handle, pulls it open smoothly, ensuring a safe and controlled motion.	Preconditions: ⬇ { " is_cabinet_closed ?": true , " is_cabinet_inside_robot_gripper_contact ?": true } Effects: ⬇ { " states_transferred ": { " insert ": { " spatial_relation ": [], " non_spatial_relation ": [] }, " delete ": [], " update ": [ " objects - cabinet - open_close_state " ] } }
(2) Take_out_kettle : The robot extends its arm, grasps the kettle, and takes it out from the cabinet.	Preconditions: ⬇ { " is_cabinet_open ?": true , " is_kettle_in_cabinet ?": true , " is_kettle_inside_robot_gripper_contact ?": true } Effects: ⬇ { " states_transferred ": { " insert ": { " spatial_relation ": [], " non_spatial_relation ": [ [" kitchen_robot ", " hold ", " kettle "] ] }, " delete ": [ " spatial_relation - kettle_cabinet_1 " ], " update ": [ " objects - kettle - position " ] } }
(3) Move_to_stove : The robot moves to the location of the stove.	Preconditions: ⬇ {} Effects: ⬇ { " states_transferred ": { " insert ": { " spatial_relation ": [], " non_spatial_relation ": [] }, " delete ": [], " update ": [ " agents - kitchen_robot - position ", " objects - kettle - position " ] } }

Appendix

Figure 12: Case study illustrating the simulation of a robot policy from the BTSIMBENCH benchmark using VIDEAS-WM-9B. For each action within the sequence, the predicted preconditions and effects are formalized in JSON format.

Action Sequence	Predicted Preconditions & Effects (JSON Structure)
(4) Place_kettle_on_stove : The robot places the kettle on the stove smoothly.	Preconditions: ⬇ { " is_stove_inside_robot_gripper_contact ?": true , " whether_robot_hold_kettle ?": true } Effects: ⬇ { " states_transferred ": { " insert ": { " spatial_relation ": [ [" kettle ", " on_top_of ", " stove "] ], " non_spatial_relation ": [] }, " delete ": [ " non_spatial_relation - kitchen_robot_kettle_c7af0617 " ], " update ": [ " objects - kettle - position " ] } }
(5) Turn_on_stove : The robot turns on the stove switch to start heating.	Preconditions: ⬇ { " is_stove_inside_robot_gripper_contact ?": true , " is_stove_off ?": true } Effects: ⬇ { " states_transferred ": { " insert ": { " spatial_relation ": [], " non_spatial_relation ": [] }, " delete ": [], " update ": [ " objects - stove - power_state " ] } }
(6) Wait_water_boiled : The robot waits for the stove to heat the kettle until the water in the kettle reaches boiling point.	Preconditions: ⬇ { " is_kettle_on_stove ?": true , " kettle_has_water ?": true , " kettle_is_closed ?": true , " kettle_is_not_boiled ?": true , " stove_is_on ?": true } Effects: ⬇ { " states_transferred ": { " insert ": { " spatial_relation ": [], " non_spatial_relation ": [] }, " delete ": [], " update ": [ " objects - kettle - temperature ", " objects - kettle - boiling_state " ] } }

Appendix

Figure 13: Case study illustrating the simulation of a robot policy from the BTSIMBENCH benchmark using VIDEAS-WM-9B (Continued).

Explore similar work

CausalWM: Causal Chain-of-Thought Reasoning for Embodied World Model
Sep 19, 2026Ziming Xu, Shuang Liang, Ruobing Han +10Efficient World-Action Model Future Video Prediction
Aether AI · University of California, San Diego · Vanderbilt University
World Action Models: The Next Frontier in Embodied AI
May 12, 2026Siyin Wang, Junhao Shi, Zhaoyang Fu +11Recent World-Action Models World Models
1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore
From World Models to World Action Models: A Concise Tutorial for Robotics
Jul 1, 2026Xiaoxiong Zhang, Xiong Zeng, Wei ZhangWorld Models Action Prediction
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China

Sep 19, 2026, cs.CVJ/K move · Enter open · S save

CausalWM: Causal Chain-of-Thought Reasoning for Embodied World Model

Ziming Xu, Shuang Liang, Ruobing Han, Ziqiao Xi+9

Aether AI · University of California, San Diego · Vanderbilt University

Efficient World-Action Model Future Video Prediction Causal Chain-of-Thought Reasoning Reinforcement Learning Post-Training

Open paper AskPDF arXiv

Embodied world models learn to predict future physical dynamics from visual observations and control signals, where physical knowledge is implicitly entangled within latent representations. We introduce CausalWM, a 16B embodied world model that performs explicit causal chain-of-thought reasoning before future video prediction. CausalWM organizes useful variables into a reasoning trajectory, allowing the model to progressively capture causal dependencies underlying physical evolution. To train CausalWM, we collect 31K hours embodied data and develop a three-stage paradigm consisting of large-scale video pre-training, causal CoT mid-training, and multi-objective RL post-training. Despite using only a limited set of supervised CoT variables, CausalWM exhibits emergent in-context learning capabilities, enabling contextual visual feature guidance and efficient few-step generation. CausalWM achieves state-of-the-art performance across language-conditioned, action-conditioned, single-view and multi-view benchmarks, including Top-1 performance on TriWorldBench leaderboard.

Abstract

Explore similar work

CausalWM: Causal Chain-of-Thought Reasoning for Embodied World Model

World Action Models: The Next Frontier in Embodied AI

From World Models to World Action Models: A Concise Tutorial for Robotics