Organizations: FNii-Shenzhen, CUHKSZ · Shenzhen University · SAI, CUHKSZ · University of Sydney · SSE, CUHKSZ · Harbin Engineering University · Ising AI
Embodied task planning demands vision-language models to generate action sequences that are both visually grounded and causally coherent over time. However, existing training paradigms face a critical trade-off: joint end-to-end training often leads to premature temporal binding, while standard reinforcement learning methods suffer from optimization instability. To bridge this gap, we present Staged Vision-Language Learning (SVLL), a unified three-stage framework for robust, physically-grounded embodied planning. In the first two stages, SVLL decouples spatial grounding from temporal reasoning, establishing robust visual dependency before introducing sequential action history. In the final stage, we identify a key limitation of standard Direct Preference Optimization (DPO), its purely relative nature -- optimizing only the preference gap between winning and losing trajectories while neglecting absolute likelihood constraints on optimal path, often yields unsafe or hallucinated behaviors. To address this, we further introduce Bias-DPO, a novel alignment objective that injects an inductive bias toward expert trajectories by explicitly maximizing likelihood on ground-truth actions while penalizing overconfident hallucinations. By anchoring the policy to the expert manifold and mitigating causal misalignment, SVLL, powered by Bias-DPO, ensures strict adherence to environmental affordances and effectively suppresses physically impossible shortcuts. Finally, extensive experiments on the interactive AI2-THOR benchmark and real-world robotic deployments demonstrate that SVLL outperforms both state-of-the-art open-source (e.g., Qwen2.5-VL-7B) and closed-source models (e.g., GPT-4o, Gemini-2.0-flash) in task success rate, while significantly reducing physical constraint violations.
Figures & tables
Figure 1: A comparison between existing task planning methods and our proposed SVLL framework. (a) Existing Methods: Existing methods suffer from premature temporal binding due to joint end-to-end training and likelihood displacement caused by standard DPO, which pushes the policy away from the expert distribution. (b) Ours (SVLL): SVLL addresses these issues by integrating staged learning and Bias-DPO, where the former decouples spatial grounding from temporal reasoning, while the latter introduces strong inductive biases to explicitly anchor the policy to the feasible expert manifold and suppresses confident errors.
Figure 2: Execution of long-horizon embodied tasks via SVLL. (a) Real-World Deployment: A physical robotic arm executes a sequence of visually grounded actions: locating a banana, picking it up, placing it into a basket, and moving the basket to a designated target. (b) Simulated Environment: An agent in an interactive 3D simulator successfully performs a multi-step task involving finding a pen, picking it up, opening a drawer, and physically placing the pen inside. Both deployments demonstrate strict adherence to physical constraints and robust causal reasoning.
Figure 3: Formulation of embodied task planning. The agent engages in a closed-loop interaction with the environment based on natural language instructions. The objective is to maximize the task success rate ( Rsuccess ), subject to a critical admissibility constraint—every intermediate action at must be physically executable and grounded within the valid visual affordance space V(It) of the current observation.
Figure 4: Overview of the SVLL framework. To mitigate the premature temporal binding in standard end-to-end training, SVLL decouples spatial and temporal learning across three stages: Stage 1 blocks the action history to force reliance on current visual affordances; Stage 2 unlocks the history context, inheriting robust visual features from Stage 1 to learn sequential dependencies; Stage 3 refines the initial policy by anchoring it to the expert manifold via an auxiliary LSFT and a threshold-triggered penalty LUL , effectively overcoming the likelihood displacement inherent in standard DPO.
Category
Model
SR ↑
SPL ↑
VAR ↑
LR ↓
CVR ↓
Closed-Source VLMs
GPT-4o [ 22 ]
48.45
46.83
59.68
1.03
40.32
Gemini-2.0-flash
58.76
49.32
68.45
3.09
31.55
Open-Source General VLMs
Qwen2.5-VL-7B [ 2 ]
18.56
17.85
45.13
5.15
54.87
Embodied-Specific (Ours & Baselines)
RoboBrain2.0-32B [ 12 ]
27.84
26.98
47.39
6.19
52.61
Ours (Direct Stage 2)
68.04
55.24
78.37
10.31
30.65
Ours (SVLL-Stage 1)
64.95
51.18
76.42
12.37
33.42
Table 1: Main results on the AI2-THOR embodied planning benchmark. We compare our SVLL framework (with 7B parameters) against closed-source VLMs, open-source general VLMs, and embodied-specific models. SR and SPL measure overall task performance (higher is better). VAR, LR, and CVR diagnose specific behavioral modes (VAR: higher is better; LR and CVR: lower is better). All metrics are reported in percentages (%). Our full framework (SVLL-Stage 3) achieves state-of-the-art performance while drastically reducing logical hallucinations.
Model
Parameters
SR ↑
CVR ↓
RoboBrain2.0 [ 12 ]
32B
22.22
10.94
Ours (SVLL-Stage 3)
7B
55.56
4.35
Table 2: Real-world zero-shot evaluation. Comparison between the 32B-parameter state-of-the-art embodied model and our SVLL framework (with 7B parameters) across 9 real-world tasks. All metrics are reported in percentages (%). Our SVLL-Stage 3 framework achieves superior SR while maintaining a significantly lower CVR, demonstrating robust causal reasoning and physical grounding capabilities.
Figure 5: Comparison of the RoboBrain2.0-32B model and our 7B-parameter SVLL-Stage 3 model in executing real-world tasks . Both models are tasked with locating an object and placing it inside the microwave. (a) Physical Constraint Violation: The baseline model successfully navigates to and picks up the apple but commits a critical physical constraint violation by attempting to Place it without first opening the microwave door. (b) Strict Causal Adherence: Our model successfully locates the pepper, explicitly executes the prerequisite Open action, and then safely places the object inside, demonstrating strict adherence to causal physical constraints throughout the task.
Embodied planning increasingly relies on vision-language models (VLMs) to translate instructions and visual observations into executable action sequences. However, fluent plans are not always executable. A VLM may refer to objects that are not visually observed, select actions whose required affordances are unavailable, or violate syntax and action constraints. We introduce CLAMP, a multimodal constraint-grounding framework that turns scene evidence into decoding-time constraints for a frozen VLM planner. CLAMP uses the initial observation to restrict object references to those supported by the scene, while a provided symbolic action model specifies state transitions and goals. During decoding, hard masks eliminate invalid next-token candidates, while a Hidden Markov Model (HMM)-based world-state lookahead module reweights the probabilities of the remaining feasible candidates based on action preconditions and goal reachability. This allows the planner to retain the VLM's language prior while preventing visually unsupported, unsafe, or infeasible candidates from entering the plan. For unseen tasks and environments, CLAMP adapts the HMM at test time using label-free continuations sampled from the frozen VLM. Experiments on VLABench, SafeAgentBench, and TaPA show that scene-grounded constraints improve object grounding and safety, while most remaining failures stem from perception errors or misaligned constraint specifications.
This paper focuses on embodied task planning, where an agent acquires visual observations from the environment and executes atomic actions to accomplish a given task. Although recent Vision-Language Models (VLMs) have achieved impressive results in multimodal understanding and reasoning, their performance remains limited when applied to embodied planning that involves multi-turn interaction, long-horizon reasoning, and extended context analysis. To bridge this gap, we propose RoboAgent, a capability-driven planning pipeline in which the model actively invokes different sub-capabilities. Each capability maintains its own context, and produces intermediate reasoning results or interacts with the environment according to the query given by a scheduler. This framework decomposes complex planning into a sequence of basic vision-language problems that VLMs can better address, enabling a more transparent and controllable reasoning process. The scheduler and all capabilities are implemented with a single VLM, without relying on external tools. To train this VLM, we adopt a multi-stage paradigm that consists of: (1) behavior cloning with expert plans, (2) DAgger training using trajectories collected by the model, and (3) reinforcement learning guided by an expert policy. Across these stages, we exploit the internal information of the environment simulator to construct high-quality supervision for each capability, and we further introduce augmented and synthetic data to enhance the model's performance in more diverse scenarios. Extensive experiments on widely used embodied task planning benchmarks validate the effectiveness of the proposed approach. Our codes will be available at https://github.com/woyut/RoboAgent_CVPR26.
Peiran Xu, Jiaqi Zheng, Yadong Mu
Peking University, Beijing, China · XYZ Embodied AI, Beijing, China
Current benchmarks for embodied vision-language planning inadvertently favor linguistic next-token prediction over physically grounded next-state reasoning. This rewards models that mimic statistical language priors rather than track true causal dependencies, reducing complex physical planning to shallow sequence modeling. Hence, achieving genuine physical autonomy requires a fundamental shift from linguistically grounded token prediction toward physically grounded causal reasoning. To this end, we introduce Causal-Plan-Bench, a high-fidelity diagnostic suite spanning four causal dimensions, curated via multi-stage verification. To endow models with this capability, a four-stage annotation pipeline extracts structured interaction records from egocentric videos to construct Causal-Plan-1M, a dense million-scale corpus of explicit causal reasoning traces. Extensive evaluation reveals a striking gap: leading models struggle to demonstrate genuine physical agency -- even GPT-6-astra scores only 43.04. In contrast, our tailored training recipe enables Causal Planner to internalize the complex physical logic required for accurate next-state estimation. Built upon Qwen3-VL-8B, Causal Planner raises its backbone's score from 33.23 to 45.28, a 36.3% relative gain, and improves on three external benchmarks without benchmark-specific adaptation. We further observe an empirical Causal-Supervision Scaling Trend. Paired no-vision controls also reveal substantial visual dependence, while cross-judge comparisons and human scoring assess the reliability of automated evaluation. More importantly, we initiate the first effort to turn agents from superficial token predictors into physically grounded causal reasoners, bridging language modeling and world modeling.
Zirui Song, Zheng Lu, Mingqi Gao +8
MBZUAI · Tsinghua University · Microsoft Research Asia +2