TAMPAgent: Agentic Belief-Space Planning for Mobile Manipulation
Organizations: NVIDIA Research · Georgia Institute of Technology · University of Utah
Abstract
Open-world robotic mobile manipulation requires acting in physical environments with only partial information. While recent Large Vision-Language Model (LVLM) agents show strong open-world reasoning capabilities, existing architectures that rely on direct tool calling struggle with geometric uncertainty and thus are limited to fully-observable tabletop environments. In contrast, Task and Motion Planning (TAMP) systems can solve long-horizon tasks under partial observability but incur high computational costs and require closed worlds. We propose TAMPAgent, an agentic planning framework that unifies the reasoning of LVLMs with the structured constraint satisfaction of TAMP systems. We endow an LVLM agent with a belief-space planning model operating over an extensible API of perception, planning, and control tools. The agent formulates a partial task plan composed of parameterized observation and execution actions, which is subsequently completed using classical search algorithms. After extracting the plan's geometric constraints, the agent leverages its tool API to bind valid parameter values, satisfying feasibility requirements and producing an executable motion plan. Upon execution, the agent continuously updates its belief history and repeats this closed-loop process by interleaving observation and manipulation actions. Evaluated across complex household mobile manipulation tasks on Rainbow Y1 and Unitree G1 platforms, TAMPAgent significantly outperforms direct tool-calling agentic baselines, demonstrating that integrating TAMP constraint satisfaction into agentic planning loops is essential for unstructured, partially observable environments.
Figures & tables
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Step | First action | Outcome | Gap | Agent’s response |
|---|---|---|---|---|
| 0 | detect [apple] | OCCLUDED | epistemic | hypothesis: behind the penguin |
| 1 | pick [penguin] | accepted | ||
| 2 | place [penguin, table] | accepted | apple observed | |
| 3 | pick [apple] | accepted | detect [apple] dropped | |
| 4 | detect [bowl] | accepted | ||
| 5 | place [apple, bowl] | EMPTY | kinematic | extend plan: move closer |
| Task | Apple | What the agent must handle | Category |
|---|---|---|---|
| T1 | dining table | Nothing. The apple lies next to the plate. | MM |
| T2 | counter | Carry the apple from the counter to the dining table. | MM |
| T3 | refrigerator shelf | Open the closed refrigerator doors before the apple is visible. | MM + occlusion |
| T4 | counter | A cup stands directly in front of the apple, 10.2 cm between their centers. | MM + non-prehensile |
| T5 | counter | An open drawer below the counter blocks the approach to the apple. | MM + non-prehensile |
| T6 | counter | Both the cup of T4 and the open drawer of T5. | MM + non-prehensile |
| Task | Goal prompt |
|---|---|
| T1 to T6, T8, T9 | “place apple on plate on dining table” |
| T7, T10 | “place the apple from counter to plate on dining table” |
| T11, T13 | “place bowl from living room to kitchen table” |
| T12 | “throw away the used sponge” |
| Tool | Model or method | Output |
|---|---|---|
| Segmentation | SAM 3 [ 19 ] | masks for a text prompt and the observed geometry of the object |
| Grasp generation | GraspGenX [ 21 ] | grasp candidates from the observed geometry |
| Grasp planning | cuRobo [ 22 ] | signed, collision-checked trajectories to a grasp |
| Place planning | cuRobo [ 22 ] | placement candidates and signed trajectories for the held object |
| Base planning | RGB-D route planner | signed base routes and bounded motion windows |
| Stereo depth (real) | Fast-FoundationStereo [ 20 ] | depth from the rectified stereo pair of the Aria glasses |
| Simulation | Real robot | |
|---|---|---|
| Robot | MuJoCo RBY1 runtime with the task scenes | RBY1 with an XHAND hand and Aria glasses |
| Perception | SAM 3, simulated depth | SAM 3 and Fast-FoundationStereo depth |
| Holding after a pick | observed by the runtime | asserted by the agent from its observations |
| Doors and drawers | fixture skills available | not available |
| Base motion | planned routes and straight legs up to 0.5 m | straight legs up to 1.5 m, no route planner |
| Torso height | about 1.00 to 1.29 m, with empty hands only | about 1.02 to 1.31 m |
Robot Task Planning
Robot task planning turns goals and observations into executable actions while coordinating skills, motion, uncertainty, and safety.
See all 254 papers