Search, Ground, Plan: Functional Sufficiency for Task and Motion Planning under Incomplete Scene Knowledge

Narendhiran Vijayakumar; Nav Singhal; Girish Varma; Antony Thomas

cs.ROSep 19, 2026

Search, Ground, Plan: Functional Sufficiency for Task and Motion Planning under Incomplete Scene Knowledge

Authors: Narendhiran Vijayakumar, Nav Singhal, Girish Varma, Antony Thomas

Organizations: Robotics Research Center, IIIT Hyderabad · Center for Security, Theory and Algorithmic Research, IIIT Hyderabad

Abstract

Foundation models (FMs) have expanded task and motion planning (TAMP) to manipulation problems specified through language and visual observations. However, incomplete scene knowledge leaves a critical gap between understanding what the task requires and knowing whether the physical scene can actually realize it. We introduce GRAB-TAMP, an FM-based TAMP framework that searches for scene entities required for task completion, grounds functional roles to valid physical objects, and plans only after a complete joint assignment establishes functional sufficiency. We represent the task through functional roles, relations, and assignment constraints, and incrementally inspect the scene while requirements remain unresolved, verifying candidate objects through semantic, geometric, and relational checks. We evaluate GRAB-TAMP across 32 scene variants spanning Kitchen, Living Room, and Workshop domains. Across 200 feasible trials, our approach achieves 54.0% end-to-end success with 67.3% plan goal coverage. Compared with three FM-based TAMP frameworks under the same execution setting, GRAB-TAMP improves end-to-end success by 25.7 percentage points over the mean baseline. Implementation and evaluation code: https://github.com/Narendhiranv04/GRAB-TAMP

Figures & tables

Fig. 2: GRAB-TAMP uses an FM to convert the goal and initial RGB observation into functional requirements $G_{F}$ , a task-relevant perception vocabulary, and an inspection ranking. Scene observations incrementally update $G_{O}^{k}$ , while semantic and geometric verification support joint object–role grounding to obtain $\phi^{*}$ . Once the scene is functionally sufficient, the grounded assignment and observed state instantiate the symbolic planner, which produces the executable action sequence. The complete framework is described in Section IV .

Component	Role in the framework
$\mathcal{V}_{g}$	Specifies what perception should look for.
$\pi_{R}$	Defines where the scene should be inspected next.
$G_{F}$	Represents the functional requirements of the task.
$G_{O}^{k}$	Stores the scene information observed up to step $k$ .
$\phi$	Maps functional roles to observed physical objects.
$\Phi_{k}$	Contains the assignments that satisfy the functional requirements.

TABLE I: Components in task specification and functional grounding.

Fig. 3: Functional assignments need not be one-to-one. Objects may be reused, regions may be shared, and multiple candidates may satisfy the same task role, as formulated by $\mathcal{A}_{F}$ in ( 7 ).

Domain	Goal instruction	Functional relations	Hidden objects	Candidate regions	Variants F / I
Kitchen	Prepare and serve one coffee and one soup for each of two people. Make each coffee using coffee and water and stir it before serving. Serve each soup bowl with its own suitable eating utensil.	OPEN_CAVITY , ELONGATED_OBJECT , INSERTABLE_IN , and REACHES_BOTTOM .	Cups, bowls, stirring and eating utensils; required objects may be distributed across storage regions.	Countertop, drawers D1--D2 , cabinets C1--C2 , and bin B1 .	6 / 6
Living Room	Prepare the living room for two people to enjoy refreshments while watching television. Provide each person with their own refreshment setting nearby, and place the entertainment control where it is accessible to both people.	PLANAR_SUPPORT , FITS_ON , FITS_SET_ON , and NEAR_SEAT .	None; task objects and placement regions are initially visible.	Left and right personal side tables and the shared coffee table.	6 / 4
Workshop	Identify the compatible components required to complete the fastening at the marked workbench location, complete the fastening, and leave any reusable equipment used for the task safely on the workbench.	GRASPABLE , FITS_TOOL_HEAD , FITS_IN , and NEAR_WORKPIECE .	Manual and power drivers, compatible Phillips screw, and wooden-hammer distractor.	Left drawer, right drawer, tool cabinet, and marked workbench target.	8 / 2

TABLE II: Task and scene definitions used for evaluating functional grounding.

	Feasible			All trials	Infeasible
Method	$N_{F}$	PGC	E2E succ.	Avg. replans	$N_{I}$	NC
Kitchen hidden across five storage regions
VLM-TAMP [ 9 ]	60	67.5	0.0	12.6	60	0.0
OWL-TAMP [ 12 ]	60	45.5	0.0	0.0	60	13.3
ROBUST-TAMP [ 10 ]	60	49.2	0.0	11.9	60	25.0
GRAB-TAMP	60	82.8	65.0	–	60	50.0

TABLE III: End-to-end comparison of 3 task domains using Google Robot. Feasible and infeasible variants are reported separately.

Verification Ablation
Scene	Verification	Trials	Matching $\mathbf{G_{O}}$ (%)	Mismatching $\mathbf{G_{O}}$ (%)	Missing $\mathbf{G_{O}}$ (%)
Kitchen	Semantic only	60	96 (12.3)	19 (2.4)	665 (85.3)
	+ Unary	60	103 (13.2)	21 (2.7)	656 (84.1)
	+ Binary	60	514 (65.9)	63 (8.1)	203 (26.0)
Living Room	Semantic only	60	141 (18.1)	1 (0.1)	638 (81.8)
	+ Unary	60	96 (12.3)	3 (0.4)	681 (87.3)

TABLE IV: Verification and inspection ablations.

Fig. 4: Failure breakdown of GRAB-TAMP for feasible variants. Categories give the earliest stage that failed, not the point of termination: only Task Specification precludes a plan (PGC = 0), while others retain partial plan and contribute nonzero goal coverage.

Motion Planning

Motion planning finds safe, feasible paths and trajectories for robots and vehicles amid geometric, dynamic, and task constraints.

See all 392 papers

Deciding If, When, and Where to Replan for Robust Manipulation
Aug 28, 2026Narendhiran Vijayakumar, Puru Ojha, Nav Singhal +2Large Language Model-Based Robot Planning Robotic Manipulation
Robotics Research Center, IIIT Hyderabad · Center for Security, Theory and Algorithmic Research, IIIT Hyderabad
Imagine-TAMP: Imagination-Guided Task and Motion Planning in Partial Observability
Sep 17, 2026Antareep Singha, Shivaram Kumar, Yoonwoo Kim +1Belief-Space Planning Active Perception
Nanyang Technological University · The University of Texas at Austin
Evidence-Gated Task and Motion Planning with Vision-Language Models
Aug 20, 2026Tsunehiko Tanaka, Matthew Stephenson, Alistair Macvicar +1Long-Horizon Robotic Manipulation Motion Planning
Waseda University · Flinders University

Aug 28, 2026J/K move · Enter open · S save

Deciding If, When, and Where to Replan for Robust Manipulation

Narendhiran Vijayakumar, Puru Ojha, Nav Singhal, Girish Varma+1

Robotics Research Center, IIIT Hyderabad · Center for Security, Theory and Algorithmic Research, IIIT Hyderabad

Large Language Model-Based Robot Planning Robotic Manipulation Motion Planning

Open paper AskPDF arXiv

Foundation-model (FM) based task and motion planning (TAMP) methods typically assume that the full scene is known before planning. In long-horizon manipulation, however, relevant objects may initially be hidden and only become observable as the robot interacts with the environment. We introduce Robust-TAMP, a task and motion planning framework that first decides whether a discovery needs a replan. If replanning is needed, the robot executes unaffected actions while the FM synthesizes corrective actions. These actions are inserted into the existing plan based on urgency, either executed immediately or deferred while preserving the remaining plan. We evaluate Robust-TAMP on 14 MuJoCo environments with hidden objects, blocked placements, articulated containers, and multi-step tasks, and on five real-world settings. Our proposed method completes 92.1% of simulated tasks, outperforming four baseline comparison architectures by 47.1 percentage points under the same settings. Ablations show that each component either raises success or lowers planner calls, planning time, and robot idle time. An extensive study across model sizes, text and vision inputs, reasoning modes, and prompting further shows that in-context examples help the planner follow multi-step procedures that zero-shot prompting struggles with. Our implementation code and project details are available at https://narendhiranv04.github.io/ROBUST-TAMP/.

Abstract

Motion Planning

Explore similar work

Deciding If, When, and Where to Replan for Robust Manipulation

Imagine-TAMP: Imagination-Guided Task and Motion Planning in Partial Observability

Evidence-Gated Task and Motion Planning with Vision-Language Models