I Won't Know Until I Try: Interactive Information Gathering for Long-Horizon Manipulation

Linfei Yuan; Rachel Holladay

cs.ROOct 25, 2026

I Won't Know Until I Try: Interactive Information Gathering for Long-Horizon Manipulation

Authors: Linfei Yuan, Rachel Holladay

Organizations: GRASP Laboratory, University of Pennsylvania, Philadelphia, PA 19104, USA

Abstract

Long-horizon manipulation often depends on object properties that cannot be reliably inferred from appearance alone. Physical interaction can reveal these properties, but it can also change the world in ways that invalidate subsequent actions. We present a task and motion planning (TAMP) framework that integrates task-directed interactive information-gathering with interleaved planning and execution. Building on PDDLStream, our framework plans optimistically with these interactive actions, verifies their outcomes during execution, and adapts the remaining plan by retaining still-applicable actions and replanning only the connections needed to continue execution. We evaluate the framework in simulation and on a real robot across tasks that require determining whether an object is movable, a pen has ink, a drawer can be opened, or a container is empty. These experiments demonstrate how interaction-driven state updates and recovery replanning support continued execution under uncertain outcomes and external disturbances.

Figures & tables

Fig. 1: Testing pen functionality through physical interaction. The robot attempts to draw on paper and checks for ink marks to determine whether the pen is functional.

Fig. 2: Example execution trace with online monitoring and recovery. Blue blocks show the initial plan and successive updated plans, containing retained actions and newly introduced bridging actions. The initial plan consists of actions $a_{1},\ldots,a_{12}$ , while $a^{k}_{i,j}$ denotes the $k$ th action of a bridge plan reconnecting execution after $a_{i}$ to retained action $a_{j}$ . After each interactive information-gathering action, Aftercheck updates the symbolic state from the observed outcome (green blocks), recording that the object has been checked and whether the tested property holds. Validator removes actions that are no longer valid: orange crosses denote collision-infeasible actions, while red crosses denote actions whose symbolic preconditions are infeasible. Before each retained action, Precheck verifies executability (red blocks). Detected precondition mismatches trigger RecoveryReplan (yellow blocks), which searches for a bridge to the next retained action or, if unsuccessful, to a later retained action. Each successful repair produces an updated plan combining the recovered bridge with the retained suffix. The system executes the bridge and resumes the retained plan, repeating state updates, validation, and on-demand repair until the goal is reached. Robot snapshots show the corresponding physical execution from the initial scene to the goal state.

Fig. 3: Recovery replanning through bridge construction. After executing $a_{1}$ and $a_{2}$ (gray), the current planning state $s_{2}^{\prime}$ differs from the predicted state $s_{2}$ , and $a_{3}$ fails its Precheck (red). RecoveryReplan searches from $s_{2}^{\prime}$ for a bridge that establishes $\operatorname{Pre}(a_{3})$ ; if unsuccessful, it tries later retained actions in order. The first bridge found is prepended to the retained suffix beginning at its target action. If no bridge is found to any retained action, global replanning searches from $s_{2}^{\prime}$ toward the active goal set $G$ (green), producing a new action sequence $[a_{3}^{\prime},a_{4}^{\prime},\ldots]$ . If this search also fails, the algorithm returns failure.

Panel of Figure 4 — Fig. 4: Representative simulation experiments in two domains. Top: Container-Emptiness, where a negative interaction result (non-empty mug) leads from picking to pouring. Bottom: Drawer-Openability, where a positive interaction result (openable drawer) leads from pulling to placing the mug inside.

Domain	Level	Ours	Global	Joint $n$
Object-Movability	Easy	18/20	19/20	17
	Hard	20/20	18/20	18
Pen-Functionality	Easy	20/20	20/20	20
	Hard	19/20	20/20	19
Drawer-Openability	Easy	19/20	20/20	19
	Hard	17/20	16/20	13

TABLE I: Task completion rates over 20 trials

Domain	Level	Object condition	Method	$n$	$T_{\mathrm{off}}$	$T_{\mathrm{on}}$	$T_{\mathrm{exec}}$	$T_{\mathrm{total}}$	$L_{0}$	$L_{\mathrm{exec}}$
Object-Movability	Easy	2 movable 1 immovable	Ours	17	$135.91\pm 213.31$	$42.51\pm 49.32$	$88.12\pm 11.14$	$\textbf{271.63}\pm 210.60$	$19.47\pm 1.94$	$21.65\pm 3.64$
	Easy	2 movable 1 immovable	Global	17	$90.50\pm 102.41$	$138.63\pm 162.85$	$85.50\pm 8.91$	$319.25\pm 176.95$	$19.06\pm 1.48$	$17.00\pm 1.90$
	Hard	3 movable 2 immovable	Ours	18	$274.78\pm 199.55$	$53.68\pm 46.93$	$165.02\pm 13.95$	$\textbf{507.94}\pm 205.49$	$33.50\pm 3.88$	$33.83\pm 4.05$
	Hard	3 movable 2 immovable	Global	18	$208.01\pm 179.15$	$527.23\pm 376.16$	$150.89\pm 18.97$	$902.73\pm 357.15$	$33.89\pm 3.58$	$29.56\pm 3.47$
Pen-Functionality	Easy	2 with ink 1 without ink	Ours	20	$28.11\pm 6.99$	$3.12\pm 1.10$	$17.84\pm 1.96$	$\textbf{58.29}\pm 8.66$	$18.15\pm 0.67$	$19.10\pm 0.45$
	Easy	2 with ink 1 without ink	Global	20	$26.67\pm 4.93$	$8.62\pm 6.10$	$16.75\pm 1.46$	$61.53\pm 9.30$	$18.00\pm 0.00$	$19.00\pm 0.00$

TABLE II: SIMULATION EFFICIENCY RESULTS ACROSS 20-TRIAL

Level	Object- Movability	Pen- Functionality	Drawer- Openability	Container- Emptiness
Easy	5/18	0/20	1/19	0/20
Hard	4/20	0/19	4/17	0/19

TABLE III: Total recovery-to-global fallback events in Ours

Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.

See all 435 papers

Deciding If, When, and Where to Replan for Robust Manipulation
Aug 28, 2026Narendhiran Vijayakumar, Puru Ojha, Nav Singhal +2Large Language Model-Based Robot Planning Robotic Manipulation
Robotics Research Center, IIIT Hyderabad · Center for Security, Theory and Algorithmic Research, IIIT Hyderabad
Imagine-TAMP: Imagination-Guided Task and Motion Planning in Partial Observability
Sep 17, 2026Antareep Singha, Shivaram Kumar, Yoonwoo Kim +1Belief-Space Planning Active Perception
Nanyang Technological University · The University of Texas at Austin
ADM-Planner: LLM-Guided Long-Horizon Planning for Mobile Manipulators with Attention-Enhanced Dynamic Memory
Sep 24, 2026Jiaping Xiao, Pingyuan Ji, Mir FeroskhanMemory-Augmented Language Model Agents Mobile Manipulation
School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore

Aug 28, 2026J/K move · Enter open · S save

Deciding If, When, and Where to Replan for Robust Manipulation

Narendhiran Vijayakumar, Puru Ojha, Nav Singhal, Girish Varma+1

Robotics Research Center, IIIT Hyderabad · Center for Security, Theory and Algorithmic Research, IIIT Hyderabad

Large Language Model-Based Robot Planning Robotic Manipulation Motion Planning

Open paper AskPDF arXiv

Foundation-model (FM) based task and motion planning (TAMP) methods typically assume that the full scene is known before planning. In long-horizon manipulation, however, relevant objects may initially be hidden and only become observable as the robot interacts with the environment. We introduce Robust-TAMP, a task and motion planning framework that first decides whether a discovery needs a replan. If replanning is needed, the robot executes unaffected actions while the FM synthesizes corrective actions. These actions are inserted into the existing plan based on urgency, either executed immediately or deferred while preserving the remaining plan. We evaluate Robust-TAMP on 14 MuJoCo environments with hidden objects, blocked placements, articulated containers, and multi-step tasks, and on five real-world settings. Our proposed method completes 92.1% of simulated tasks, outperforming four baseline comparison architectures by 47.1 percentage points under the same settings. Ablations show that each component either raises success or lowers planner calls, planning time, and robot idle time. An extensive study across model sizes, text and vision inputs, reasoning modes, and prompting further shows that in-context examples help the planner follow multi-step procedures that zero-shot prompting struggles with. Our implementation code and project details are available at https://narendhiranv04.github.io/ROBUST-TAMP/.

Abstract

Long-Horizon Robotic Manipulation

Explore similar work

Deciding If, When, and Where to Replan for Robust Manipulation

Imagine-TAMP: Imagination-Guided Task and Motion Planning in Partial Observability

ADM-Planner: LLM-Guided Long-Horizon Planning for Mobile Manipulators with Attention-Enhanced Dynamic Memory