Zero-shot Interactive Perception

Venkatesh Sripada; Frank Guerin; Amir Ghalamzan

cs.ROFeb 20, 2026

Zero-shot Interactive Perception

Authors: Venkatesh Sripada, Frank Guerin, Amir Ghalamzan

Organizations: University of Surrey · University of Sheffield

Abstract

Interactive perception (IP) enables robots to extract hidden information in their workspace and execute manipulation plans by physically interacting with objects and altering the state of the environment -- crucial for resolving occlusions and ambiguity in complex, partially observable scenarios. We present Zero-Shot IP (ZS-IP), a novel framework that couples multi-strategy manipulation (pushing and grasping) with a memory-driven Vision Language Model (VLM) to guide robotic interactions and resolve semantic queries. ZS-IP integrates three key components: (1) an Enhanced Observation (EO) module that augments the VLM's visual perception with both conventional keypoints and our proposed pushlines -- a novel 2D visual augmentation tailored to pushing actions, (2) a memory-guided action module that reinforces semantic reasoning through context lookup, and (3) a robotic controller that executes pushing, pulling, or grasping based on VLM output. Unlike grid-based augmentations optimized for pick-and-place, pushlines capture affordances for contact-rich actions, substantially improving pushing performance. We evaluate ZS-IP on a 7-DOF Franka Panda arm across diverse scenes with varying occlusions and task complexities. Our experiments demonstrate that ZS-IP outperforms passive and viewpoint-based perception techniques such as Mark-Based Visual Prompting (MOKA), particularly in pushing tasks, while preserving the integrity of non-target elements.

Figures & tables

Figure 1: Zero-shot Interactive Perception Framework: The Perception Analyser (i.e. $z_{t}=\mathcal{F}_{VLM}({o}_{t},q)$ ) evaluates the scene observation $o_{t}$ at robot configuration $\textbf{x}_{t}$ and attempts to resolve the query, $q$ . If unsuccessful, EO annotates the image, $\tilde{o}_{t}=EO(o_{t})$ , after segmenting target objects with Grounded SAM, e.g. push lines ( $EO_{\text{P}}$ ), keypoints ( $EO_{\text{G}}$ ), and 2D grid ( $EO_{\text{AP}}$ ). A history of interactions is kept in memory $\mathcal{M}_{t}$ to avoid redundant actions and enhance task efficiency. Action $a_{t}=\pi(\mathcal{M}_{t},\textbf{x}_{t},z_{t},\tilde{o}_{t})$ is generated to make a proper physical interaction to answer the query. $\mathcal{M}_{t}$ states, $S_{t}$ contains both images $o_{t}$ and a summary of scene description, $s_{t}$ , relevant to query. The robot performs the interaction and saves the corresponding image and robot’s state in $S_{t+1}$ which then goes to the Perception Analyser.

Panel of Figure 2 — Figure 2: Four 2x2 images to the left show Push lines: (Top left and right) A virtual 2D grid overlaid on the table, projected onto the 2D image, along with push lines generated for Task I (Query: What is under the eraser?). The robot selects P3 as the pre-contact point (bottom left) and D3 as the post-contact point (bottom right) to execute the pushing action. This interaction successfully uncovers and reveals the box of paper clips previously hidden underneath. Four 2x2 images to the right show setup and Grasp keypoints: Top: Robotic setup (left), scene for task VI; Bottom: key points generated for two objects in Task V involving pick-and-place at time step 1 (left) and 2 (right).

Method	Pushing			Grasping		Lifting to Investigate		Observe
Method	Task I	Task II	Task III	Task IV	Task V	Task VI	Task VII	Task VIII
ZS-IP	0.9	0.9	0.6	1.0	0.8	0.8	0.2	0.7

Table 1: ZS-IP performed for eight different tasks with different required levels of complexity from easy (i.e. Task I) to more complex (Task VIII) interactions. Four categories of interaction are considered: ‘Pushing’, ‘Pick-and-place’ or ‘Grasping’, ‘Lifting’ an object to investigate it, and ‘observe’. Each task is performed 10 times, and the results show the success rate out of 10 trials.

Method	Task VII				Task VIII
Method	SR $\uparrow$	TL, TLS $\downarrow m$	PE $\downarrow m$	OSR $\uparrow$	SR $\uparrow$	TL,TLS $\downarrow m$	PE $\downarrow m$	OSR $\uparrow$
MOKA	0.0	{3.88, -}	0.40	7	0.5	{1.96, 1.34}	0.60	4
ZS-IP Observe	-	-	-	-	0.1	{0.48, 0.35}	0.15	1
ZS-IP Push	0.0	{3.25, -}	0.48	7	0.8	{1.48, 1.63}	0.57	4
ZS-IP Grasp	0.0	{3.04, -}	0.39	5	-	-	-	-
ZS-IP	0.2	{3.41, 3.08}	0.47	7	0.7	{1.47, 1.56}	0.64	4

Table 2: Comparing ZS-IP with different features and MOKA. The performance Metrics include SR (Success Rate), TL (Total Length) TLP (Total Length Successful), PE (Position Error) and OSR (Oracle Success Rate).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Robot workspace during pushing actions. A source of light produces a strong shadow. The top row shows Task I, uncovering a box of paper clips under the whiteboard cleaner, the middle row shows Task II pushing an aluminium tin, revealing a small screw, and the bottom row shows Task III a clutter of Lego blocks.

Panel of Figure 4 — Figure 4: Task VII being successfully performed at various time steps (query: ‘What language is the text written in on the black eraser?’): (a) $t=1$ : { $x_{1},o_{1}$ } with pushing action proposed, $EO_{P}$ , (b) $t=2$ : { $x_{2},o_{2}$ } after pushing, red and yellow objects are separated from the eraser. Grasping key points are now proposed $EO_{G}$ , (c) $t=2$ : Cam2 view of eraser that is lifted and moved by the robot in front of the Cam2, (d) Task VIII: brown book partially occludes the view into the cup.

Panel of Figure 5 — Figure 5: (From left to right) First and Third row: Task I to VIII. Second and Fourth row: the robot’s workspace in Task I to VIII. Task I: the robot needs to push the eraser to see the box of paper clips underneath. Task II: The robot needs to pick up the aluminium tin and place it elsewhere to see the screw in the shadow. Task III: Push the Lego blocks to see the pen-drive underneath. Task IV: Pick up the blue block and place it elsewhere to see the text written on the paper under it. Task V: Identify the red block and green Lego structure on the table. Arrange them inside the cardboard box while maintaining their original spatial positions relative to one another. Task VI: grasp the black eraser and bring it in front of the secondary camera to see what is written on the side under it. Task VII: similar to Task VI but eraser is between two objects. Hence ZS-IP requires first pushing the eraser and then grasping to complete the task. Task VIII: ZS-IP should see what is inside the cup covered by the brown book.

Method	Task IV				Task VI				Task VIII
Method	SR $\uparrow$	TL, TLS $\downarrow m$	PE $\downarrow m$	OSR $\uparrow$	SR $\uparrow$	TL,TLS $\downarrow m$	PE $\downarrow m$	OSR $\uparrow$	SR $\uparrow$	TL,TLS $\downarrow m$	PE $\downarrow m$	OSR $\uparrow$
PIVOT w interact perception	0.0	{7.71, -}	0.55	14	0.7	{4.86, 3.81}	0.541	7	0.0	{5.78, -}	0.59	1
PIVOT w active perception	-	-	-	-	-	-	-	-	0.3	{1.06, 0.254}	0.428	0
ZS-IP	1.0	{1.83, 1.83}	0.64	1	0.8	{2.16, 2.11}	0.59	3	0.7	{1.47, 1.56}	0.64	4

Appendix

Table 3: Comparing ZS-IP with different features and PIVOT. The performance Metrics include SR (Success Rate), TL (Total Length) TLP (Total Length Successful), PE (Position Error) and OSR (Oracle Success Rate).

Method	Pushing			Grasping		Lifting to Investigate		Observe
Method	Task I	Task II	Task III	Task IV	Task V	Task VI	Task VII	Task VIII
ZS-IP w GPT-4o (Ours)	0.9	0.9	0.6	1.0	0.8	0.8	0.2	0.7
ZS-IP w Gemini 2.0 Flash	0.5	0.6	-	0.9	-	-	-	-
ZS-IP w Claude 3.5 Sonnet	0.8	1.0	-	1.0	-	-	-	-

Appendix

Table 4: ZS-IP performed on Gemini 2.0 Flash and Claude 3.5 Sonnet

Method	Task III
Method	SR $\uparrow$
GPT-4o	0.6
GPT-4 Turbo	0.0

Appendix

Table 5: Comparison of GPT-4o with Other Models

Panel of Figure 6 — Figure 6: Perception Analyser

Panel of Figure 7 — Figure 7: Memory block of Perception Analyser

Panel of Figure 8 — Figure 8: Output of Perception Analyser

Panel of Figure 9 — Figure 9: Prompt for grasping

Panel of Figure 10 — Figure 10: Prompt to perform camera in-hand movement

Explore similar work

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation
Jun 22, 2026Ali Alabbas, Dipshikha Das, Camillo Murgia +3Zero-Shot Robotic Manipulation Language-Conditioned Robot Manipulation
Department of Electronic, Software and Advanced Manufacturing Engineering, Atlantic Technological University, Galway, Ireland · Atlantic Technological University, Galway, Ireland
Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation
Feb 28, 2026Zilong Xie, Jingyu Gong, Xin Tan +2Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation
School of Computer Science and Technology, East China Normal University, Shanghai, China · Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, Chongqing, China · Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China
Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning
May 8, 2026Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2Large Language Model-Based Robot Planning LLM Prompting
IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1

Jun 22, 2026J/K move · Enter open · S save

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation

Ali Alabbas, Dipshikha Das, Camillo Murgia, Sainul Ansary+2

Department of Electronic, Software and Advanced Manufacturing Engineering, Atlantic Technological University, Galway, Ireland · Atlantic Technological University, Galway, Ireland

Zero-Shot Robotic Manipulation Language-Conditioned Robot Manipulation Large Language Model-Based Robot Planning Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.

Abstract

Explore similar work

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation

Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning