Open-world robotic mobile manipulation requires acting in physical environments with only partial information. While recent Large Vision-Language Model (LVLM) agents show strong open-world reasoning capabilities, existing architectures that rely on direct tool calling struggle with geometric uncertainty and thus are limited to fully-observable tabletop environments. In contrast, Task and Motion Planning (TAMP) systems can solve long-horizon tasks under partial observability but incur high computational costs and require closed worlds. We propose TAMPAgent, an agentic planning framework that unifies the reasoning of LVLMs with the structured constraint satisfaction of TAMP systems. We endow an LVLM agent with a belief-space planning model operating over an extensible API of perception, planning, and control tools. The agent formulates a partial task plan composed of parameterized observation and execution actions, which is subsequently completed using classical search algorithms. After extracting the plan's geometric constraints, the agent leverages its tool API to bind valid parameter values, satisfying feasibility requirements and producing an executable motion plan. Upon execution, the agent continuously updates its belief history and repeats this closed-loop process by interleaving observation and manipulation actions. Evaluated across complex household mobile manipulation tasks on Rainbow Y1 and Unitree G1 platforms, TAMPAgent significantly outperforms direct tool-calling agentic baselines, demonstrating that integrating TAMP constraint satisfaction into agentic planning loops is essential for unstructured, partially observable environments.
Figures & tables
Figure 1: Integrating agents with planning. We compare four systems that maps a history of observations to whole body control for long-horizon mobile-manipulation tasks: (a) TAMP [ 14 , 7 ] solves hand-authored problems with a task planner that orchestrates tools. (b) VLM-TAMP [ 9 , 10 ] uses the agent to write the problem from open-world understanding but keeps the planning pipeline fixed. (c) CaP [ 3 , 15 , 5 , 6 ] lets the agent write programs that orchestrate tools directly to find solutions. (d) TAMPAgent (ours) lets the agent orchestrate both the task planner and the tools to find certified solutions that satisfy the planner derived constraints.
Figure 2: Plan skeleton as a factor graph. The skeleton a=⟨detect[A],pick[A],place[A,B]⟩ unrolled over states s0,…,s3 (columns) becomes factor graph with: (i) nodes V as the planning variables (rows), such as the configurations q1,q2,q3 , the apple’s pose, the grasp g2 , and the placement x3 and (ii) factors F as the constraints between them, such as RGBD , Grasp , Kin , Motion , and Stable visualized in boxes. The apple’s pose starts as a belief bA and becomes the observed pose x1 through detect action.
Figure 3: Hybrid beliefs over perception and manipulation variables. Left: The belief over the apple’s location moves from Unif , a uniform prior over the room, to a Lang hypothesis written by the agent, “… on the visible counter / dining table / inside the refrigerator …”. The agent then chooses a viewpoint and calls SAM 3 to represent apple’s location as a Mask , and then as a RGBD estimate when its close enough. Right: The grasp (top) and placement (bottom) p′ requires cup’s RGBD estimate as input. The agent injects language context ℓ , such as “I must place the cup far from the apple” and calls tools to certify valid grasps and placement locations. Finally, the agent chooses a valid option based on implicitly represented planning objectives.
Figure 4: TAMPAgent planning loop. We show how the agent orchestrates the planner and tools to solve the task “put the apple on the black table (hidden behind a toy penguin) in a bowl”. (1-3) The agent observes, hypothesizes where the apple is, and writes the problem and a plan, (4) the planner compiles the plan into a CSP, and (5) the agent writes a computation graph that calls tools to find an assignment βt satisfying the CSP. (6) The tools certify the first action’s constraints and the planner accepts βt , allowing (7) the robot to execute the first action. (8) The loop repeats until (9-10) the placement is planned and the goal is achieved.
Figure 5: Evaluation tasks. We ask the robot to place the apple on the plate on the dining table (prompts in Table A3 ), starting with the robot facing away from the apple. Each task shows the whole scene with the robot at its start pose (top, walls and ceiling hidden) and a close-up of the area around the apple (bottom). The tasks fall into four categories. In the mobile manipulation (MM) tasks T1 and T2, the apple is in plain view, next to the plate or on the countertop. The MM + non-prehensile tasks add an object the robot must move first. In T4 to T6, a cup, an open drawer, or both block the direct pick, in T10, a mustard bottle stands in front of the apple, and in T8, a cracker box stands in front of the plate. The MM + occlusion tasks hide the apple, inside the fridge in T3 and behind a cracker box in T7, so the agent must hypothesize where to search. T9 is MM + non-prehensile + occlusion: the apple is inside the fridge behind a cup, which the robot must move before it can pick the apple. In the close-ups of T3 and T9, we draw the refrigerator doors translucent so that the shelf is visible. In the simulation the doors are closed.
Figure 6: Three-room tasks. T11 to T13 take place in a three-room home with a living room, a kitchen, and a washroom. In T11, the robot must move a bowl from the living-room table to the kitchen table, and in T12, it must throw a used sponge from the laundry machine in the washroom into the kitchen trash can. T13 has the same goal as T11, but a chair blocks the living-room doorway. We show the layout of the home at the start of T11 (ceiling hidden), followed by close-ups of the bowl on the living-room table (T11), the sponge on the laundry machine (T12), and the chair in the living-room doorway, seen from the kitchen (T13).
Figure 7: Success on the RBY1 , over 20 episodes per task and method, with the tasks grouped by category. (a) Absolute success. (b) Partial success, where each episode earns the fraction of its subtasks it completes ( Figure A1 , Appendix C ). The last group averages each over the tasks.
Figure 8: Failure attribution for RBY1 simulation experiments. Failed episodes out of 20 per task, in which an object fell or the scene was destroyed (solid), or which timed out (hatched).
Figure 9: Success on the three-room tasks ( Figure 6 ) on the RBY1, over 5 episodes per task and method.
Figure 10: Learning from attempts on T9. (a) Partial success of each attempt, the fraction of the four stages (approach the refrigerator and find the apple, relocate the cup, pick the apple, place the apple) it completes, until the first success. (b) Uncached input and (c) output tokens, summed over all attempts. The tokens of GaP include its helper agents.
Figure 11: Clearing a blocker on a real G1. The goal is to hold a yellow can. (1) The agent finds a toy dog that blocks the grasp of the can and plans to move it. (2) From the first stance, the planner does not certify the grasp and kinematics of pick[dog] , so (3) the agent adds an approach to the plan. (4) From the new stance, the tools certify the grasp, kinematics, and motion, and (5) the robot picks and lifts the dog. Insets show the robot’s camera, the SAM 3 masks, and the grasp candidates. The run ends before the can is picked, when placing the dog is not certified.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Step
First action
Outcome
Gap
Agent’s response
0
detect [apple]
OCCLUDED
epistemic
hypothesis: behind the penguin
1
pick [penguin]
accepted
2
place [penguin, table]
accepted
apple observed
3
pick [apple]
accepted
detect [apple] dropped
4
detect [bowl]
accepted
5
place [apple, bowl]
EMPTY
kinematic
extend plan: move closer
Appendix
Table A1: The run of Figure 4 , step by step.
Task
Apple
What the agent must handle
Category
T1
dining table
Nothing. The apple lies next to the plate.
MM
T2
counter
Carry the apple from the counter to the dining table.
MM
T3
refrigerator shelf
Open the closed refrigerator doors before the apple is visible.
MM + occlusion
T4
counter
A cup stands directly in front of the apple, 10.2 cm between their centers.
MM + non-prehensile
T5
counter
An open drawer below the counter blocks the approach to the apple.
MM + non-prehensile
T6
counter
Both the cup of T4 and the open drawer of T5.
MM + non-prehensile
Appendix
Table A2: Kitchen tasks. We list where the apple is, what the agent must handle before it can pick the apple or place it, and the category of the task (MM: mobile manipulation).
Task
Goal prompt
T1 to T6, T8, T9
“place apple on plate on dining table”
T7, T10
“place the apple from counter to plate on dining table”
T11, T13
“place bowl from living room to kitchen table”
T12
“throw away the used sponge”
Appendix
Table A3: Goal prompt per task. CaP, GaP, and TAMPAgent receive the same prompt, and nothing else, with no description of the scene.
Figure A1: Subtask completion on the RBY1 , out of 20 episodes per task and method, with the tasks in category order. All three methods find the apple equally often when it is in the open. TAMPAgent separates from CaP and GaP at finding the apple in the fridge (T3) and behind the box (T7), at clearing each blocker (T4 to T6, T10), and at placing next to the box on the dining table (T8), and the gap persists through the remaining subtasks.
Figure A2: Success on the G1 , over five episodes per task and method. Partial success is the fraction of applicable subtasks completed, shown per subtask in Figure A3 .
Figure A3: Subtask completion on the G1 , out of five episodes per task and method. A hairline marks a subtask that applies but was never completed.
Figure A4: Robots and sensors. Left: the Rainbow Robotics RB-Y1 with Project Aria glasses for the ego view, an XHAND dexterous hand, and its battery pack. Right: the Unitree G1 with an Intel RealSense ego camera, an OAK-D look-ahead camera, and Dex3 hands.
Figure A5: System architecture. We run the agent in a sandbox that reaches the system only through a gateway. The TAMPAgent API is the robot-facing API. It holds the belief session, forwards perception and planning calls to the tools, executes skills on the robot only under grants from the TAMP service, and records the episode. The visualization reads the TAMP session without changing it.
Tool
Model or method
Output
Segmentation
SAM 3 [ 19 ]
masks for a text prompt and the observed geometry of the object
Grasp generation
GraspGenX [ 21 ]
grasp candidates from the observed geometry
Grasp planning
cuRobo [ 22 ]
signed, collision-checked trajectories to a grasp
Place planning
cuRobo [ 22 ]
placement candidates and signed trajectories for the held object
Base planning
RGB-D route planner
signed base routes and bounded motion windows
Stereo depth (real)
Fast-FoundationStereo [ 20 ]
depth from the rectified stereo pair of the Aria glasses
Appendix
Table A4: Tools. We list each tool service, the model or method behind it, and what its output certifies. The planners sign their outputs, and the TAMPAgent API executes only trajectories with a valid signature.
Simulation
Real robot
Robot
MuJoCo RBY1 runtime with the task scenes
RBY1 with an XHAND hand and Aria glasses
Perception
SAM 3, simulated depth
SAM 3 and Fast-FoundationStereo depth
Holding after a pick
observed by the runtime
asserted by the agent from its observations
Doors and drawers
fixture skills available
not available
Base motion
planned routes and straight legs up to 0.5 m
straight legs up to 1.5 m, no route planner
Torso height
about 1.00 to 1.29 m, with empty hands only
about 1.02 to 1.31 m
Appendix
Table A5: Simulation and real-robot profiles. We list where the two profiles differ. They share the planner, the gateway, the sandbox, the recorder, and the agent’s documentation apart from the pages named here.