Bridging Task Planning and Execution for Language-Conditioned Bimanual Manipulation
Organizations: The authors are with the University of Genoa, DIBRIS, Genoa, Italy.
Abstract
Robots are increasingly expected to execute natural-language manipulation tasks in real-world environments. This is particularly relevant in household assistance and logistics, where clutter requires reasoning about both task objectives and execution feasibility. This challenge becomes even more pronounced for bimanual humanoid platforms, where coordinating two manipulators introduces additional interaction and kinematic constraints. We present a zero-shot vision-language framework for execution-aware bimanual manipulation that jointly reasons about task objectives and physical feasibility. Planning is grounded in a structured scene representation that combines semantic object relations with execution constraints, enabling the selection of obstacle relocations and execution choices while preserving task semantics. Compatible actions are organised into parallel bimanual stages with explicit pre- and post-conditions. A hierarchical validator evaluates these conditions under perceptual uncertainty and uses stage-level execution history to support local recovery or targeted replanning. Experiments on the CENTAURO hybrid wheeled-legged humanoid robot across nine language-conditioned manipulation scenarios show higher task success than a non-execution-aware baseline. In dedicated failure scenarios, hierarchical validation recovers from execution failures with fewer model queries than always replanning. These results demonstrate the benefits of combining execution reasoning directly into task planning. Videos and additional material are available on the Project Webpage: https://advrhumanoids.github.io/bridging-planning-execution/.
Figures & tables
| Strategy | Scene | Action | Manip. | Scheduler |
| Perception | Revision | Planning | ||
| repeat | ||||
| modify | ||||
| replan |
| Scenario performance | Module performance (Ours) | |||||||||||||
| Non-execution-aware & Binary validation | Execution-aware & Binary validation | Execution-aware & Hierarchical validation (Ours) | Scene Perception | Manipulation Planning | Execution Scheduling | Hierarchical Validation | ||||||||
| Scenario | SR | Queries | SR | Queries | SR | Queries | SR | Latency | SR | Latency | SR | Latency | SR | Latency |
| Execution-Aware Planning | ||||||||||||||
| Stacked Boxes | 100% | 8.2 | 100% | 8.2 | 100% | 8.2 | 100% | 5.10 | 100% | 9.69 | 100% | 8.42 | 100% | 5.85 |
| Boxes Aside | 0% | – | 75% | 12.3 | 85% | 11.8 | 96% | 7.33 | 92% | 12.82 | 100% | 8.32 | 98% | 4.45 |
| Bottles on Box | 70% | 9.6 | 90% | 9.4 | 95% | 9.4 | 98% | 7.78 | 100% | 10.93 | 85% | 10.01 | 97% | 4.91 |
| Backbone | SR | Latency |
| Gemini Robotics-ER 1.6 | 73.3% | 41.9 s |
| GPT-5.2 | 5.0% | 34.6 s |
| GPT-o3 | 80.0% | 91.4 s |
Language-Conditioned Robot Manipulation
Language-conditioned robot manipulation connects natural-language instructions to robot perception, planning, and physical actions.
See all 253 papers