Foundation-model (FM) based task and motion planning (TAMP) methods typically assume that the full scene is known before planning. In long-horizon manipulation, however, relevant objects may initially be hidden and only become observable as the robot interacts with the environment. We introduce Robust-TAMP, a task and motion planning framework that first decides whether a discovery needs a replan. If replanning is needed, the robot executes unaffected actions while the FM synthesizes corrective actions. These actions are inserted into the existing plan based on urgency, either executed immediately or deferred while preserving the remaining plan. We evaluate Robust-TAMP on 14 MuJoCo environments with hidden objects, blocked placements, articulated containers, and multi-step tasks, and on five real-world settings. Our proposed method completes 92.1% of simulated tasks, outperforming four baseline comparison architectures by 47.1 percentage points under the same settings. Ablations show that each component either raises success or lowers planner calls, planning time, and robot idle time. An extensive study across model sizes, text and vision inputs, reasoning modes, and prompting further shows that in-context examples help the planner follow multi-step procedures that zero-shot prompting struggles with. Our implementation code and project details are available at https://narendhiranv04.github.io/ROBUST-TAMP/.
Figures & tables
Figure 1. Opening a container reveals a hidden object, affecting the robot’s ongoing plan and subsequent actions.
Figure 2. Overview of ROBUST-TAMP. Scene observations, memory, and execution history inform IF : whether an observed object requires correction; WHEN : which independent actions can continue during planning; and WHERE : how corrective actions enter remaining plan.
Symbol
Meaning
g
Natural-language task goal
stvis
Current observed scene description
mt
Memory of previous observations
ht
Completed-action history
πt
Current remaining action sequence
ct
Planning context supplied to the FM
Table 1. Core notation used by ROBUST-TAMP .
Figure 3. Urgent and deferred corrections during execution. Independent actions continue while FM plans a correction. Urgent blocks enter before the updated remaining plan; deferred blocks follow selected pending action or are appended to the plan.
(a) Scale, modality, and prompting
(b) 8B-class models
SR
PGC
SR
PGC
Scale
Mod.
Prompt
K
G
All
K
G
All
Plan.
Mod.
Reas.
Model
K
G
All
K
G
All
Plan.
4B
LLM
ZS
14.4
0.0
9.3
18.0
41.5
26.4
242
VLM
–
Qwen3-VL-8B-Instruct
1.1
2.0
1.4
2.9
24.5
10.6
168
ICL
14.4
14.0
14.3
18.0
54.5
31.1
206
Ministral-3-8B-Instruct
16.7
2.0
11.4
54.2
52.7
53.7
47
VLM
ZS
53.3
6.0
36.4
55.5
43.8
51.3
736
InternVL3.5-8B
18.9
0.0
12.1
28.1
20.1
25.2
111
ICL
53.3
70.0
59.3
55.5
90.1
67.9
560
✓
Qwen3-VL-8B-Thinking
94.4
22.0
68.6
97.7
66.3
86.5
540
Table 2. Planner comparison across (a) model size, modality, and prompting, and (b) reasoning and non-reasoning 8B models.
ID
Hidden
Rel.
Ovl.
Exp.
Ceil.
SR
PGC
Calls
Plan.
Idle
Time
Urg.
A. Basic
K0
None
–
–
–
100.0
100.0
100.0
2.2
229
–
298
–
G0
None
–
–
–
100.0
100.0
100.0
1.1
117
–
197
–
B. Core
K1
Phone
×
✓
U
100.0
90.0
93.3
5.7
481
189
561
10.0
K2
Phone
×
×
–
100.0
90.0
95.0
4.0
328
–
397
–
Table 3. Task variants and ROBUST-TAMP performance across basic, core, and cardinality settings.
Kitchen
Grill
Condition
SR
C
I
P
T
SR
C
I
P
T
Full
95.0
4.2
118
377
457
86.7
2.7
135
281
373
− Mem.
85.0
4.0
131
342
420
60.0
2.6
175
343
435
− IF
95.0
5.5
199
441
511
76.7
2.8
171
317
408
− WHERE
75.0
3.9
76
348
434
56.7
4.5
221
546
648
Front
90.0
4.0
116
394
473
30.0
6.1
185
821
916
Table 4. Ablation study of our proposed ROBUST-TAMP .
Success (%) ↑
Cost ↓
Method
SR K
SR G
SR
PGC
Calls
Plan.
Lat.
Time
VLM-TAMP ( Yang et al., 2025 )
23.3
6.0
17.1
71.4
2.1
77
31.8
176
+ ICL
10.0
18.6
71.9
2.1
85
35.9
183
OWL-TAMP ( Kumar et al., 2026 )
17.8
2.0
12.1
49.1
6.3
730
115.3
790
+ ICL
8.0
14.3
53.3
6.4
740
114.6
803
Inner Monologue ( Huang et al., 2023 )
64.4
6.0
43.6
76.5
14.4
956
62.3
1114
Table 5. Baseline Comparison across 14 MuJoCo variants
Figure 4. (a) In-context example improvement across the Qwen3 (Table 2 a) family. (b) Outcomes of the selected planner over 140 trials (Table 3 ). (c) ROBUST-TAMP failure analysis.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Rejected if
Format
a block is malformed or has missing/invalid fields
Coverage
a trigger object is omitted, or an unrelated object is included
References
an action/object/region is unsupported or unknown
Manipulation
the block contains an invalid or incomplete manipulation sequence
Insertion
an urgent block is not at front , or its anchor is invalid
Merged plan
the merged plan fails re-parsing from the current executor state
Appendix
Table S1. Checks applied to a corrective response before merging.
Figure S3. Recorded corrective requests of the ICL run. Top: an accepted correction and its logged merge. Bottom: a rejected answer, the feedback line added to the re-query, and the accepted revision. Text panels with a recorded corrective request, the model's answers, a feedback line and the merged plan.
Model
Qwen3-VL-8B-Thinking (BF16), reasoning on
Sampling
temperature 1.0, top- p 0.95, top- k 20, presence penalty 0
Output limit
24,576 tokens per call
Visual input
one composite image per request
Engine
vLLM 0.30.0, one server per run, context 32,768, GPU memory 0.90, Qwen3 reasoning parser, ≤2 images
Timeout
1,800 s per call
Hardware
NVIDIA RTX 5090, 32 GB
Appendix
Table S2. Planner inference settings (logged with every trial and in the run records).