Foundation-model (FM) based task and motion planning (TAMP) methods typically assume that the full scene is known before planning. In long-horizon manipulation, however, relevant objects may initially be hidden and only become observable as the robot interacts with the environment. We introduce Robust-TAMP, a task and motion planning framework that first decides whether a discovery needs a replan. If replanning is needed, the robot executes unaffected actions while the FM synthesizes corrective actions. These actions are inserted into the existing plan based on urgency, either executed immediately or deferred while preserving the remaining plan. We evaluate Robust-TAMP on 14 MuJoCo environments with hidden objects, blocked placements, articulated containers, and multi-step tasks, and on five real-world settings. Our proposed method completes 92.1% of simulated tasks, outperforming four baseline comparison architectures by 47.1 percentage points under the same settings. Ablations show that each component either raises success or lowers planner calls, planning time, and robot idle time. An extensive study across model sizes, text and vision inputs, reasoning modes, and prompting further shows that in-context examples help the planner follow multi-step procedures that zero-shot prompting struggles with. Our implementation code and project details are available at https://narendhiranv04.github.io/ROBUST-TAMP/.
Figures & tables
Figure 1. Opening a container reveals a hidden object, affecting the robot’s ongoing plan and subsequent actions.
Figure 2. Overview of ROBUST-TAMP. Scene observations, memory, and execution history inform IF : whether an observed object requires correction; WHEN : which independent actions can continue during planning; and WHERE : how corrective actions enter remaining plan.
Symbol
Meaning
g
Natural-language task goal
stvis
Current observed scene description
mt
Memory of previous observations
ht
Completed-action history
πt
Current remaining action sequence
ct
Planning context supplied to the FM
Table 1. Core notation used by ROBUST-TAMP .
Figure 3. Urgent and deferred corrections during execution. Independent actions continue while FM plans a correction. Urgent blocks enter before the updated remaining plan; deferred blocks follow selected pending action or are appended to the plan.
(a) Scale, modality, and prompting
(b) 8B-class models
SR
PGC
SR
PGC
Scale
Mod.
Prompt
K
G
All
K
G
All
Plan.
Mod.
Reas.
Model
K
G
All
K
G
All
Plan.
4B
LLM
ZS
14.4
0.0
9.3
18.0
41.5
26.4
242
VLM
–
Qwen3-VL-8B-Instruct
1.1
2.0
1.4
2.9
24.5
10.6
168
ICL
14.4
14.0
14.3
18.0
54.5
31.1
206
Ministral-3-8B-Instruct
16.7
2.0
11.4
54.2
52.7
53.7
47
VLM
ZS
53.3
6.0
36.4
55.5
43.8
51.3
736
InternVL3.5-8B
18.9
0.0
12.1
28.1
20.1
25.2
111
ICL
53.3
70.0
59.3
55.5
90.1
67.9
560
✓
Qwen3-VL-8B-Thinking
94.4
22.0
68.6
97.7
66.3
86.5
540
Table 2. Planner comparison across (a) model size, modality, and prompting, and (b) reasoning and non-reasoning 8B models.
ID
Hidden
Rel.
Ovl.
Exp.
Ceil.
SR
PGC
Calls
Plan.
Idle
Time
Urg.
A. Basic
K0
None
–
–
–
100.0
100.0
100.0
2.2
229
–
298
–
G0
None
–
–
–
100.0
100.0
100.0
1.1
117
–
197
–
B. Core
K1
Phone
×
✓
U
100.0
90.0
93.3
5.7
481
189
561
10.0
K2
Phone
×
×
–
100.0
90.0
95.0
4.0
328
–
397
–
Table 3. Task variants and ROBUST-TAMP performance across basic, core, and cardinality settings.
Kitchen
Grill
Condition
SR
C
I
P
T
SR
C
I
P
T
Full
95.0
4.2
118
377
457
86.7
2.7
135
281
373
− Mem.
85.0
4.0
131
342
420
60.0
2.6
175
343
435
− IF
95.0
5.5
199
441
511
76.7
2.8
171
317
408
− WHERE
75.0
3.9
76
348
434
56.7
4.5
221
546
648
Front
90.0
4.0
116
394
473
30.0
6.1
185
821
916
Table 4. Ablation study of our proposed ROBUST-TAMP .
Success (%) ↑
Cost ↓
Method
SR K
SR G
SR
PGC
Calls
Plan.
Lat.
Time
VLM-TAMP ( Yang et al., 2025 )
23.3
6.0
17.1
71.4
2.1
77
31.8
176
+ ICL
10.0
18.6
71.9
2.1
85
35.9
183
OWL-TAMP ( Kumar et al., 2026 )
17.8
2.0
12.1
49.1
6.3
730
115.3
790
+ ICL
8.0
14.3
53.3
6.4
740
114.6
803
Inner Monologue ( Huang et al., 2023 )
64.4
6.0
43.6
76.5
14.4
956
62.3
1114
Table 5. Baseline Comparison across 14 MuJoCo variants
Figure 4. (a) In-context example improvement across the Qwen3 (Table 2 a) family. (b) Outcomes of the selected planner over 140 trials (Table 3 ). (c) ROBUST-TAMP failure analysis.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Rejected if
Format
a block is malformed or has missing/invalid fields
Coverage
a trigger object is omitted, or an unrelated object is included
References
an action/object/region is unsupported or unknown
Manipulation
the block contains an invalid or incomplete manipulation sequence
Insertion
an urgent block is not at front , or its anchor is invalid
Merged plan
the merged plan fails re-parsing from the current executor state
Appendix
Table S1. Checks applied to a corrective response before merging.
Figure S3. Recorded corrective requests of the ICL run. Top: an accepted correction and its logged merge. Bottom: a rejected answer, the feedback line added to the re-query, and the accepted revision. Text panels with a recorded corrective request, the model's answers, a feedback line and the merged plan.
Model
Qwen3-VL-8B-Thinking (BF16), reasoning on
Sampling
temperature 1.0, top- p 0.95, top- k 20, presence penalty 0
Output limit
24,576 tokens per call
Visual input
one composite image per request
Engine
vLLM 0.30.0, one server per run, context 32,768, GPU memory 0.90, Qwen3 reasoning parser, ≤2 images
Timeout
1,800 s per call
Hardware
NVIDIA RTX 5090, 32 GB
Appendix
Table S2. Planner inference settings (logged with every trial and in the run records).
Robots operating in cluttered environments must often manipulate objects whose locations are only partially observable. A central challenge is deciding whether to acquire another observation or to first manipulate objects that may occlude the target. Conventional task and motion planning (TAMP) approaches typically make this decision using symbolic action costs or expensive geometric planning, neither of which adequately captures how likely an observation is to reveal an occluded target. We introduce Imagine-TAMP, an interleaved planning and execution framework that uses semantic and geometric imagination to compare alternative task-level strategies under partial observability before committing to expensive motion planning. A vision-language model shapes a particle belief over target locations using commonsense relationships between the target and visible objects, while a generative scene model estimates plausible geometry in unobserved regions. Given a target hypothesis and imagined scene, Imagine-TAMP generates multiple symbolic plan skeletons and assigns non-unit costs that approximate both manipulation effort and target visibility from sensing actions, distinguishing a short but poorly informative observation strategy from a longer strategy that first manipulates an occluder to better expose the target. The selected skeleton is then refined into a feasible continuous plan and executed, with new observations updating the belief and triggering replanning when necessary. Experiments show that imagination-guided evaluation improves observation-versus-manipulation decisions: in viewpoint-constrained shelf scenes, non-unit geometric evaluation increases success from 46.0% to 84.0%, while semantic belief shaping further reduces manipulation and replanning. On a real robot, the complete system reduces planning time by 32% relative to a geometry-only ablation.
Antareep Singha, Shivaram Kumar, Yoonwoo Kim +1
Nanyang Technological University · The University of Texas at Austin
Foundation models (FMs) have expanded task and motion planning (TAMP) to manipulation problems specified through language and visual observations. However, incomplete scene knowledge leaves a critical gap between understanding what the task requires and knowing whether the physical scene can actually realize it. We introduce GRAB-TAMP, an FM-based TAMP framework that searches for scene entities required for task completion, grounds functional roles to valid physical objects, and plans only after a complete joint assignment establishes functional sufficiency. We represent the task through functional roles, relations, and assignment constraints, and incrementally inspect the scene while requirements remain unresolved, verifying candidate objects through semantic, geometric, and relational checks. We evaluate GRAB-TAMP across 32 scene variants spanning Kitchen, Living Room, and Workshop domains. Across 200 feasible trials, our approach achieves 54.0% end-to-end success with 67.3% plan goal coverage. Compared with three FM-based TAMP frameworks under the same execution setting, GRAB-TAMP improves end-to-end success by 25.7 percentage points over the mean baseline. Implementation and evaluation code: https://github.com/Narendhiranv04/GRAB-TAMP
Narendhiran Vijayakumar, Nav Singhal, Girish Varma +1
Robotics Research Center, IIIT Hyderabad · Center for Security, Theory and Algorithmic Research, IIIT Hyderabad
Robot planning in partially observable environments, where not all objects are known or visible, is a challenging problem, as it requires reasoning under uncertainty through partially observable Markov decision processes. During the execution of a computed plan, a robot may unexpectedly observe task-irrelevant objects, which are typically ignored by naive planners. In this work, we propose incorporating two types of common-sense knowledge: (1) certain objects are more likely to be found in specific locations; and (2) similar objects are likely to be co-located, while dissimilar objects are less likely to be found together. Manually engineering such knowledge is complex, so we explore leveraging the powerful common-sense reasoning capabilities of large language models (LLMs). Our planning and execution framework, CoCo-TAMP, introduces a hierarchical state estimation that uses LLM-guided information to shape the belief over task-relevant objects, enabling efficient solutions to long-horizon task and motion planning problems. In experiments, CoCo-TAMP achieves an average reduction of 62.7% in planning and execution time in simulation, and 72.6% in real-world demonstrations, compared to a baseline that does not incorporate either type of common-sense knowledge.
Yoonwoo Kim, Raghav Arora, Roberto Martín-Martín +3
The University of Texas at Austin · Nanyang Technological University