Organizations: East China Normal University · Beijing Academy of Artificial Intelligence (BAAI) · Northwestern Polytechnical University · School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences
Multimodal large language models (MLLMs) show promising reasoning capabilities, but reliable physical interaction requires planning that accounts for when actions are executable and how they change the world. These preconditions and effects are environment-dependent and often not captured by MLLMs, making adaptation challenging. Yet existing approaches either require costly environment-specific training or rely on unstructured interaction experience that is difficult to reuse. Therefore, we introduce CausGround, a non-parametric causal interaction grounding framework that distills interaction experience into reusable action-centered causal knowledge. CausGround identifies action preconditions and effects and organizes them in a factorized world-state representation capturing object attributes and inter-object relations. At planning time, task-relevant causal dependencies are grounded in current observations to support backward reasoning over preconditions and forward reasoning over effects for action selection and multi-step planning. We evaluate CausGround on simulated high-level planning, interaction reasoning, and real-world robot manipulation. On adapted VirtualHome and RoboCasa benchmarks, CausGround improves vision-based high-level planning across MLLM backbones, increasing average task success from 24.9% to 53.3% on VirtualHome and closed-loop planning success from 43.0% to 57.0% on RoboCasa. On SWITCH, average interaction reasoning accuracy increases from 25.8% to 31.9%, indicating improved action selection, recovery, and verification. Moreover, real-world experiments show that CausGround reasons backward from MLLM-interpreted goals to derive executable high-level action sequences for sequential manipulation. These results demonstrate that reusable action preconditions and effects support long-horizon embodied planning across diverse environments without parameter updates.
Figures & tables
Figure 1: Overview of CausGround. CausGround distills open-world interaction experience into reusable action-centered causal knowledge by identifying action preconditions and effects. At planning time, task-relevant causal dependencies are grounded in the current observation to form a local causal graph, supporting backward reasoning over preconditions and forward reasoning over effects for executable long-horizon planning without parameter updates.
Figure 2: Overview of CausGround. Left: CausGround distills interaction trajectories into reusable action-centered causal knowledge by extracting action preconditions and effects, which are organized into generic and instance-specific knowledge. Right: Given the task description and current world state, task-relevant causal knowledge is grounded to construct a causal graph that provides executable guidance for a frozen planner.
Figure 3: Evaluation environments and representative tasks. Left: SWITCH examples covering action selection, error recovery, state verification, and verification action. Right: High-level planning trajectories in RoboCasa and VirtualHome, showing post-action observations. RoboCasa steps correspond to annotated skill stages; VirtualHome uses repaired image observations.
Backbone
Method
VirtualHome
RoboCasa
Short
Medium
Long
Average
One-shot
Replan
SR
P
SR
P
SR
P
SR
P
Qwen3.5-9B
Vanilla
28.0
34.6
4.0
18.1
0.0
18.9
10.7
23.8
31.0
36.7
SelfReVision
48.0
52.7
12.0
22.5
8.0
27.2
22.7
34.1
26.8
26.5
LookPlanGraph
44.0
56.8
12.0
35.5
4.0
26.9
20.0
39.7
19.7
28.6
CausGround
80.0
83.6
32.0
51.3
20.0
49.8
44.0
61.5
45.1
46.9
Table 1: High-level planning performance. VirtualHome reports success rate (SR) and final progress (P) across short, medium, and long horizon tasks, with Average denoting the macro-average over the three difficulty levels. RoboCasa reports task success rate under one-shot planning and closed-loop replanning. Best results are shown in bold and second-best results are underlined .
Backbone
Method
Action
Recovery
Verification Action
Verification State
Average
Qwen3.5-9B
Vanilla
24.2
36.8
30.2
19.6
27.7
CausGround
28.1
44.7
36.5
22.2
32.9
Qwen3.5-27B
Vanilla
31.9
28.9
36.5
14.2
27.9
CausGround
33.7
39.5
52.4
15.1
35.2
Qwen3.5-35B-A3B
Vanilla
27.1
23.7
23.8
12.0
21.7
CausGround
27.4
28.9
41.3
13.3
27.7
Table 2: Interaction reasoning on SWITCH. Accuracy (%) is reported across four interaction reasoning categories and three different backbones. Best results are shown in bold.
Figure 4: Case Study. Vanilla identifies the correct chair but repeatedly executes SIT without satisfying the NEAR precondition, while CausGround uses causal knowledge to complete the task.
Method
SR
Progress
Vanilla
33.3
46.9
+ Factorized State
36.0
55.1
+ Causal Knowledge
53.3
70.8
CausGround
62.7
79.8
Table 3: Component ablations. Contributions of the local graph and causal interaction knowledge.
Figure 5: Generalization across model families and scales. CausGround improves SR and final progress across backbones.
Figure 6: Real-world Deployment. Given initial observation and task, CausGround retrieves task-relevant causal knowledge from the interpreted goal state and grounds it to executable actions.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Short
Medium
Long
Total
# Tasks
25
25
25
75
Reference actions per task
3–6
7–10
11–22
3–22
Mean reference actions per task
4.28
8.12
13.32
8.57
Max. action attempts
20
40
60
–
Appendix
Table 4: Statistics of the adapted VirtualHome benchmark. Tasks are divided into Short, Medium, and Long splits according to the number of reference actions per task. The maximum action attempts specify the independent interaction budget available to the agent during closed-loop evaluation.
Component
Definition
Task input
A fixed natural-language task instruction.
Visual input
Three simultaneous RGB observations rendered from the current simulator state.
Action history
The planner’s own previously attempted actions together with their SUCCESS/FAILURE outcomes.
Action space
A fixed set of 45 atomic high-level action templates supported by the VirtualHome semantic executor, e.g., WALK() , GRAB() , OPEN() , PUTIN() , and SWITCHON() .
Object vocabulary
A public menu of executable object-class names used to bind predicted action arguments to simulator entities; private simulator locations and hidden state are excluded.
Planner output
One atomic action together with its object-class argument(s).
Appendix
Table 5: Shared planning interface in VirtualHome. All methods receive the same public task information, visual observations, action templates, object vocabulary, and execution feedback. Private simulator states and reference programs are not exposed to the planner.
Figure 7: Representative observation repairs in VirtualHome. Each row shows a transition example with the original pre and post observations (left two columns) and the repaired pre/post observations (right two columns). The repair corrects visually inconsistent state transitions, such as object transfer, container interaction, and posture change, while preserving unrelated scene content.
Figure 8: Representative high-level planning trajectories in RoboCasa. We show five representative tasks covering object manipulation, navigation, container interaction, fixture operation, and multi-step composition. Each trajectory is represented by intermediate RGB observations and the corresponding high-level actions from our unified action interface.
Subtask
# Examples
Action
442
Recovery
38
Verification Action
63
Verification State
225
Total
768
Appendix
Table 6: Statistics of the selected SWITCH subtasks. We evaluate four interaction-reasoning categories from the official SWITCH benchmark.
Figure 9: Representative cases from the selected SWITCH subtasks. We show examples of Action Selection, Recovery, Verification Action, and Verification State. These subtasks respectively evaluate next-action prediction, error recovery, active verification, and state verification under embodied interactions.
Figure 10: Illustration of the factorized world state. Visual observations are converted into a structured representation that explicitly separates object attributes from inter-object relations. The same representation is used to describe the current state as well as the preconditions and effects associated with environment interactions.
Figure 11: Illustration of action-centered causal factor identification. CausGround compares executions of the same grounded action under variations of pre-action state factors. Factors associated with changes in action executability are retained as relevant preconditions, whereas incidental factors that do not affect the execution outcome are discarded.
Figure 12: Task-grounded planning in CausGround. The task description is used to retrieve task-relevant causal knowledge from the CausGround library. At each planning step, the current observation is converted into a factorized world state and used to ground the retrieved causal graph in the current environment state before action prediction. The predicted action is executed in the environment, and the resulting observation is returned for the next decision step.
Method
Qwen3.5-9B
Qwen3.5-27B
Qwen3.5-35B-A3B
Success Rate
Progress
Success Rate
Progress
Success Rate
Progress
Vanilla
10.7
23.8
33.3
46.9
30.7
44.4
+ Factorized State
17.3
32.3
36.0
55.1
40.0
56.3
+ Causal Knowledge
41.3
57.6
53.3
70.8
50.7
69.1
CausGround (Ours)
44.0
61.5
62.7
79.8
53.3
72.6
Appendix
Table 7: Component ablations across planner backbones. Contributions of the factorized state and causal interaction knowledge across different Qwen3.5 backbones.
Retrieval Strategy
Success Rate
Progress
Vanilla
10.7
23.8
+ Semantic Retrieval
18.7
37.6
+ Causal Retrieval
42.7
60.4
CausGround (Ours)
44.0
61.5
Appendix
Table 8: Ablation of retrieval strategies. We compare semantic retrieval, causal backward retrieval, and their combination using Qwen3.5-9B on the VirtualHome benchmark.
Knowledge Variant
Success Rate
Progress
Vanilla
10.7
23.8
+ Preconditions Only
36.0
52.1
+ Effects Only
38.7
55.3
CausGround (Ours)
44.0
61.5
Appendix
Table 9: Ablation of action preconditions and effects. We separately expose only action preconditions or only action effects to evaluate their individual contributions on the VirtualHome benchmark.
Figure 13: Real-world execution of the microwave task. Given the instruction “Move the bread from the microwave to the plate,” CausGround produces the high-level sequence OPEN(Microwave) → PICK(Bread) → PLACE(Bread, Plate) , which is executed by the robot in the real environment.
Figure 14: Real-world execution of the multi-object task. Given the instruction “Put the banana and bread on the plate,” CausGround sequentially generates PICK(Banana) → PLACE(Banana, Plate) → PICK(Bread) → PLACE(Bread, Plate) , demonstrating multi-object sequential manipulation.