Organizations: East China Normal University · Beijing Academy of Artificial Intelligence (BAAI) · Northwestern Polytechnical University · School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences
Multimodal large language models (MLLMs) show promising reasoning capabilities, but reliable physical interaction requires planning that accounts for when actions are executable and how they change the world. These preconditions and effects are environment-dependent and often not captured by MLLMs, making adaptation challenging. Yet existing approaches either require costly environment-specific training or rely on unstructured interaction experience that is difficult to reuse. Therefore, we introduce CausGround, a non-parametric causal interaction grounding framework that distills interaction experience into reusable action-centered causal knowledge. CausGround identifies action preconditions and effects and organizes them in a factorized world-state representation capturing object attributes and inter-object relations. At planning time, task-relevant causal dependencies are grounded in current observations to support backward reasoning over preconditions and forward reasoning over effects for action selection and multi-step planning. We evaluate CausGround on simulated high-level planning, interaction reasoning, and real-world robot manipulation. On adapted VirtualHome and RoboCasa benchmarks, CausGround improves vision-based high-level planning across MLLM backbones, increasing average task success from 24.9% to 53.3% on VirtualHome and closed-loop planning success from 43.0% to 57.0% on RoboCasa. On SWITCH, average interaction reasoning accuracy increases from 25.8% to 31.9%, indicating improved action selection, recovery, and verification. Moreover, real-world experiments show that CausGround reasons backward from MLLM-interpreted goals to derive executable high-level action sequences for sequential manipulation. These results demonstrate that reusable action preconditions and effects support long-horizon embodied planning across diverse environments without parameter updates.
Figures & tables
Figure 1: Overview of CausGround. CausGround distills open-world interaction experience into reusable action-centered causal knowledge by identifying action preconditions and effects. At planning time, task-relevant causal dependencies are grounded in the current observation to form a local causal graph, supporting backward reasoning over preconditions and forward reasoning over effects for executable long-horizon planning without parameter updates.
Figure 2: Overview of CausGround. Left: CausGround distills interaction trajectories into reusable action-centered causal knowledge by extracting action preconditions and effects, which are organized into generic and instance-specific knowledge. Right: Given the task description and current world state, task-relevant causal knowledge is grounded to construct a causal graph that provides executable guidance for a frozen planner.
Figure 3: Evaluation environments and representative tasks. Left: SWITCH examples covering action selection, error recovery, state verification, and verification action. Right: High-level planning trajectories in RoboCasa and VirtualHome, showing post-action observations. RoboCasa steps correspond to annotated skill stages; VirtualHome uses repaired image observations.
Backbone
Method
VirtualHome
RoboCasa
Short
Medium
Long
Average
One-shot
Replan
SR
P
SR
P
SR
P
SR
P
Qwen3.5-9B
Vanilla
28.0
34.6
4.0
18.1
0.0
18.9
10.7
23.8
31.0
36.7
SelfReVision
48.0
52.7
12.0
22.5
8.0
27.2
22.7
34.1
26.8
26.5
LookPlanGraph
44.0
56.8
12.0
35.5
4.0
26.9
20.0
39.7
19.7
28.6
CausGround
80.0
83.6
32.0
51.3
20.0
49.8
44.0
61.5
45.1
46.9
Table 1: High-level planning performance. VirtualHome reports success rate (SR) and final progress (P) across short, medium, and long horizon tasks, with Average denoting the macro-average over the three difficulty levels. RoboCasa reports task success rate under one-shot planning and closed-loop replanning. Best results are shown in bold and second-best results are underlined .
Backbone
Method
Action
Recovery
Verification Action
Verification State
Average
Qwen3.5-9B
Vanilla
24.2
36.8
30.2
19.6
27.7
CausGround
28.1
44.7
36.5
22.2
32.9
Qwen3.5-27B
Vanilla
31.9
28.9
36.5
14.2
27.9
CausGround
33.7
39.5
52.4
15.1
35.2
Qwen3.5-35B-A3B
Vanilla
27.1
23.7
23.8
12.0
21.7
CausGround
27.4
28.9
41.3
13.3
27.7
Table 2: Interaction reasoning on SWITCH. Accuracy (%) is reported across four interaction reasoning categories and three different backbones. Best results are shown in bold.
Figure 4: Case Study. Vanilla identifies the correct chair but repeatedly executes SIT without satisfying the NEAR precondition, while CausGround uses causal knowledge to complete the task.
Method
SR
Progress
Vanilla
33.3
46.9
+ Factorized State
36.0
55.1
+ Causal Knowledge
53.3
70.8
CausGround
62.7
79.8
Table 3: Component ablations. Contributions of the local graph and causal interaction knowledge.
Figure 5: Generalization across model families and scales. CausGround improves SR and final progress across backbones.
Figure 6: Real-world Deployment. Given initial observation and task, CausGround retrieves task-relevant causal knowledge from the interpreted goal state and grounds it to executable actions.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Short
Medium
Long
Total
# Tasks
25
25
25
75
Reference actions per task
3–6
7–10
11–22
3–22
Mean reference actions per task
4.28
8.12
13.32
8.57
Max. action attempts
20
40
60
–
Appendix
Table 4: Statistics of the adapted VirtualHome benchmark. Tasks are divided into Short, Medium, and Long splits according to the number of reference actions per task. The maximum action attempts specify the independent interaction budget available to the agent during closed-loop evaluation.
Component
Definition
Task input
A fixed natural-language task instruction.
Visual input
Three simultaneous RGB observations rendered from the current simulator state.
Action history
The planner’s own previously attempted actions together with their SUCCESS/FAILURE outcomes.
Action space
A fixed set of 45 atomic high-level action templates supported by the VirtualHome semantic executor, e.g., WALK() , GRAB() , OPEN() , PUTIN() , and SWITCHON() .
Object vocabulary
A public menu of executable object-class names used to bind predicted action arguments to simulator entities; private simulator locations and hidden state are excluded.
Planner output
One atomic action together with its object-class argument(s).
Appendix
Table 5: Shared planning interface in VirtualHome. All methods receive the same public task information, visual observations, action templates, object vocabulary, and execution feedback. Private simulator states and reference programs are not exposed to the planner.
Figure 7: Representative observation repairs in VirtualHome. Each row shows a transition example with the original pre and post observations (left two columns) and the repaired pre/post observations (right two columns). The repair corrects visually inconsistent state transitions, such as object transfer, container interaction, and posture change, while preserving unrelated scene content.
Figure 8: Representative high-level planning trajectories in RoboCasa. We show five representative tasks covering object manipulation, navigation, container interaction, fixture operation, and multi-step composition. Each trajectory is represented by intermediate RGB observations and the corresponding high-level actions from our unified action interface.
Subtask
# Examples
Action
442
Recovery
38
Verification Action
63
Verification State
225
Total
768
Appendix
Table 6: Statistics of the selected SWITCH subtasks. We evaluate four interaction-reasoning categories from the official SWITCH benchmark.
Figure 9: Representative cases from the selected SWITCH subtasks. We show examples of Action Selection, Recovery, Verification Action, and Verification State. These subtasks respectively evaluate next-action prediction, error recovery, active verification, and state verification under embodied interactions.
Figure 10: Illustration of the factorized world state. Visual observations are converted into a structured representation that explicitly separates object attributes from inter-object relations. The same representation is used to describe the current state as well as the preconditions and effects associated with environment interactions.
Figure 11: Illustration of action-centered causal factor identification. CausGround compares executions of the same grounded action under variations of pre-action state factors. Factors associated with changes in action executability are retained as relevant preconditions, whereas incidental factors that do not affect the execution outcome are discarded.
Figure 12: Task-grounded planning in CausGround. The task description is used to retrieve task-relevant causal knowledge from the CausGround library. At each planning step, the current observation is converted into a factorized world state and used to ground the retrieved causal graph in the current environment state before action prediction. The predicted action is executed in the environment, and the resulting observation is returned for the next decision step.
Method
Qwen3.5-9B
Qwen3.5-27B
Qwen3.5-35B-A3B
Success Rate
Progress
Success Rate
Progress
Success Rate
Progress
Vanilla
10.7
23.8
33.3
46.9
30.7
44.4
+ Factorized State
17.3
32.3
36.0
55.1
40.0
56.3
+ Causal Knowledge
41.3
57.6
53.3
70.8
50.7
69.1
CausGround (Ours)
44.0
61.5
62.7
79.8
53.3
72.6
Appendix
Table 7: Component ablations across planner backbones. Contributions of the factorized state and causal interaction knowledge across different Qwen3.5 backbones.
Retrieval Strategy
Success Rate
Progress
Vanilla
10.7
23.8
+ Semantic Retrieval
18.7
37.6
+ Causal Retrieval
42.7
60.4
CausGround (Ours)
44.0
61.5
Appendix
Table 8: Ablation of retrieval strategies. We compare semantic retrieval, causal backward retrieval, and their combination using Qwen3.5-9B on the VirtualHome benchmark.
Knowledge Variant
Success Rate
Progress
Vanilla
10.7
23.8
+ Preconditions Only
36.0
52.1
+ Effects Only
38.7
55.3
CausGround (Ours)
44.0
61.5
Appendix
Table 9: Ablation of action preconditions and effects. We separately expose only action preconditions or only action effects to evaluate their individual contributions on the VirtualHome benchmark.
Figure 13: Real-world execution of the microwave task. Given the instruction “Move the bread from the microwave to the plate,” CausGround produces the high-level sequence OPEN(Microwave) → PICK(Bread) → PLACE(Bread, Plate) , which is executed by the robot in the real environment.
Figure 14: Real-world execution of the multi-object task. Given the instruction “Put the banana and bread on the plate,” CausGround sequentially generates PICK(Banana) → PLACE(Banana, Plate) → PICK(Bread) → PLACE(Bread, Plate) , demonstrating multi-object sequential manipulation.
Current benchmarks for embodied vision-language planning inadvertently favor linguistic next-token prediction over physically grounded next-state reasoning. This rewards models that mimic statistical language priors rather than track true causal dependencies, reducing complex physical planning to shallow sequence modeling. Hence, achieving genuine physical autonomy requires a fundamental shift from linguistically grounded token prediction toward physically grounded causal reasoning. To this end, we introduce Causal-Plan-Bench, a high-fidelity diagnostic suite spanning four causal dimensions, curated via multi-stage verification. To endow models with this capability, a four-stage annotation pipeline extracts structured interaction records from egocentric videos to construct Causal-Plan-1M, a dense million-scale corpus of explicit causal reasoning traces. Extensive evaluation reveals a striking gap: leading models struggle to demonstrate genuine physical agency -- even GPT-6-astra scores only 43.04. In contrast, our tailored training recipe enables Causal Planner to internalize the complex physical logic required for accurate next-state estimation. Built upon Qwen3-VL-8B, Causal Planner raises its backbone's score from 33.23 to 45.28, a 36.3% relative gain, and improves on three external benchmarks without benchmark-specific adaptation. We further observe an empirical Causal-Supervision Scaling Trend. Paired no-vision controls also reveal substantial visual dependence, while cross-judge comparisons and human scoring assess the reliability of automated evaluation. More importantly, we initiate the first effort to turn agents from superficial token predictors into physically grounded causal reasoners, bridging language modeling and world modeling.
Zheng Lu, Mingqi Gao, Qinlei Xie +7
Tsinghua University · Peking University · MBZUAI +1
Embodied world models learn to predict future physical dynamics from visual observations and control signals, where physical knowledge is implicitly entangled within latent representations. We introduce CausalWM, a 16B embodied world model that performs explicit causal chain-of-thought reasoning before future video prediction. CausalWM organizes useful variables into a reasoning trajectory, allowing the model to progressively capture causal dependencies underlying physical evolution. To train CausalWM, we collect 31K hours embodied data and develop a three-stage paradigm consisting of large-scale video pre-training, causal CoT mid-training, and multi-objective RL post-training. Despite using only a limited set of supervised CoT variables, CausalWM exhibits emergent in-context learning capabilities, enabling contextual visual feature guidance and efficient few-step generation. CausalWM achieves state-of-the-art performance across language-conditioned, action-conditioned, single-view and multi-view benchmarks, including Top-1 performance on TriWorldBench leaderboard.
Ziming Xu, Shuang Liang, Ruobing Han +10
Aether AI · University of California, San Diego · Vanderbilt University
Manipulation requires not only reasoning about the external environment, but also about the robot's physical condition. A strategy may remain geometrically feasible while becoming physically unsuitable due to increased joint load or limited mobility, yet internal physical state is typically used only for low-level control. We propose body-grounded high-level replanning, which uses internal physical state to adapt manipulation strategies during execution. Body-state events trigger strategy replanning, and an LLM interprets the underlying joint-level state, recent execution statistics, and execution history to select a context-dependent alternative, while leaving the task objective and low-level controller unchanged. We evaluate the framework on a reaching task under controlled load and asymmetric mobility constraints in simulation and on a real robot. Our experiments show that body-grounded replanning maintains high task success while reducing physical effort and enabling more efficient strategy adaptation. Additional contact-rich manipulation experiments demonstrate the applicability of the same replanning interface beyond reaching. These results show that internal physical state can inform not only low-level control, but also high-level decisions about how a manipulation task should be performed.
Namiko Saito, Hiroshi Kera
Microsoft Research Asia - Tokyo · Waseda University · Chiba University +1