This paper focuses on embodied task planning, where an agent acquires visual observations from the environment and executes atomic actions to accomplish a given task. Although recent Vision-Language Models (VLMs) have achieved impressive results in multimodal understanding and reasoning, their performance remains limited when applied to embodied planning that involves multi-turn interaction, long-horizon reasoning, and extended context analysis. To bridge this gap, we propose RoboAgent, a capability-driven planning pipeline in which the model actively invokes different sub-capabilities. Each capability maintains its own context, and produces intermediate reasoning results or interacts with the environment according to the query given by a scheduler. This framework decomposes complex planning into a sequence of basic vision-language problems that VLMs can better address, enabling a more transparent and controllable reasoning process. The scheduler and all capabilities are implemented with a single VLM, without relying on external tools. To train this VLM, we adopt a multi-stage paradigm that consists of: (1) behavior cloning with expert plans, (2) DAgger training using trajectories collected by the model, and (3) reinforcement learning guided by an expert policy. Across these stages, we exploit the internal information of the environment simulator to construct high-quality supervision for each capability, and we further introduce augmented and synthetic data to enhance the model's performance in more diverse scenarios. Extensive experiments on widely used embodied task planning benchmarks validate the effectiveness of the proposed approach. Our codes will be available at https://github.com/woyut/RoboAgent_CVPR26.
Figures & tables
Figure 1: (a) The pipeline of a CoT-enhanced embodied task planner. (b) The pipeline of the proposed RoboAgent framework. By explicitly invoking specific vision–language capabilities, our method achieves a more reliable reasoning process while fully leveraging the perception and understanding proficiency of the VLM. The scheduler and the capabilities are all implemented with a single model.
Figure 2: An illustration of the scheduler and 5 capabilities involved in our RoboAgent.
Figure 3: Upper: illustration of the data types and modules involved in different training stages. Lower: demonstration of the training objective in the RFT stage. Traditional RL algorithms optimize the improvement of return over the original policy, whereas the proposed EIPO optimizes the discrepancy between the policy return and the expert return.
Base Model
Avg
Base
Common
Complex
Visual
Spatial
Long
Zero-Shot [ 125 ]
GPT-4o
56.3
64
54
68
46
52
54
Claude-3.7-Sonnet
67.7
68
68
70
68
62
70
Gemini-1.5-Pro
62.3
70
64
72
58
52
58
Qwen-VL-Max
41.3
44
48
44
42
38
32
Qwen2.5-VL-72B
39.7
50
42
42
36
34
34
RoboBrain2.0 [ 104 ]
Qwen2.5-VL-7B
14.0
-
-
-
-
-
-
Table 1: Performance comparison on EB-ALFRED [ 125 ] .
Table 5Table 6
Figure 4: The curve of SR on ALFWorld’s val_seen split during the training process of EIPO and GRPO.
Figure 5: The input and output format of the EG capability.
Figure 6: The input and output format of the OG capability.
Figure 7: The input and output format of the SD capability.
Figure 8: The input and output format of the AD capability for exploration sub-plans (using ALFWorld’s action space).
Figure 9: The input and output format of the AD capability for manipulation sub-plans (using ALFWorld’s action space).
Figure 10: The input and output format of the ES capability.
Figure 11: The input and output format of the scheduler.
Task Types
Action Space
Nav. Target
Instruction
Test Scene
ALFWorld
6
go to, pick, put, turn on, open, close,
only receptacles
template-based
seen&unseen
heat, cool, clean
EB-ALFRED
7
go to, pick, put, turn on, open, close,
all objects
human-written
seen
slice
Table 7: A comparison between ALFWorld [ 94 ] and EB-ALFRED [ 125 ] . We slightly modify the names of the atomic actions ( e.g. , replacing “find” with “go to”) to enable a clearer comparison of the differences.
SFT-expert
SFT-DAgger
EIPO
SR
✓
✗
✗
42.5
✓
✓
✗
59.7
✓
✓
w.o. syn. task
67.2
✓
✓
w. syn. task
68.6
Table 8: The results of solely using ALFWorld for training.
Cap.
SFT
EIPO
SR
✗
✓
✗
40.3
✗
✓
✓
48.5
✓
✓
✗
59.7
✓
✓
✓
68.6
Table 9: The results of an VLM planer with/without explicit capability invocation.
objective
SR
rollout return (episode-level group)
72.8±3.6
rollout return (episode&step-level group)
86.7±1.7
expert advantage (episode-level group)
94.8 ±3.9
expert advantage (episode&step-level group)
92.7±3.9
Table 10: A comparison between GRPO and EIPO for training LLM-based (Qwen2.5-1.5B) planner on ALFWorld. The reported results are averaged across 3 independent runs.
objective
ALFWorld
EB-ALFRED
none
73.1
64.3
reward
69.4
65.0
expert advantage
77.6
67.0
Table 11: A comparison between GRPO and EIPO for the training stage 3 of our pipeline.
time
# call
# token in
# token out
action-based
35.6s
16.3
11.3k
1.1k
capability-based
36.3s
35.1
10.1k
1.1k
Table 12: Statics of VLM inference for capability-based and action-based planner.
Figure 12: A demonstration of real-world deployment. We present some of the input images and output actions. A human operator carried out the actions and captured the images with camera.
Figure 13: Error distribution on EB-ALFRED.
Figure 14: Error distribution on ALFWorld.
Figure 15: A visualization of two typical failure cases of RoboAgent. In the task above, the scheduler ignores the key information “illumination” from the instruction, and does not provide a complete and correct query to the invoked capabilities. In the task below, OG fails to understand the open-vocabulary query, resulting in the detection of an incorrect target object (soap bottle, instead of towel).
Figure 16: A visualization of the generated plan on EB-ALFRED (long horizon split), part 1.
Figure 17: A visualization of the generated plan on EB-ALFRED (long horizon split), part 2.
Figure 18: A visualization of the generated plan on EB-ALFRED (long horizon split), part 3.
Figure 19: A visualization of the generated plan on EB-ALFRED (visual appearance split).
Figure 20: A visualization of the generated plan on ALFWorld’s visual environment, part 1.
Figure 21: A visualization of the generated plan on ALFWorld’s visual environment, part 2.
Figure 22: A visualization of the generated plan on ALFWorld’s textual environment.