Zero-Shot Robotic Manipulation

Follow this topic Ask about this topic

Zero-shot robotic manipulation studies how robots can carry out new tasks and interact with unfamiliar objects or environments without task-specific demonstrations or fine-tuning. Approaches combine vision-language models and language-guided planning with geometric grounding, affordance representations, object-centric policies, and closed-loop feedback; other work learns transferable skills from human videos or scales synthetic data and simulation for sim-to-real transfer. Evaluation spans tabletop and long-horizon tasks, dexterous and contact-rich manipulation, and transfer across objects, viewpoints, robots, and real-world settings. A central challenge is turning broad semantic generalization into precise, reliable physical control, especially under occlusion, changing contact dynamics, and compounding errors during long tasks.

Latest papers

103 papers26 papers in the last four weeks, up 117% on the four weeks before. 0.2% of all new papers.

Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation
Feb 28, 2026Zilong Xie, Jingyu Gong, Xin Tan +2Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation
School of Computer Science and Technology, East China Normal University, Shanghai, China · Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, Chongqing, China · Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China
LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies
Feb 25, 2026Yue Yang, Shuo Cheng, Yu Fang +4Long-Horizon Robotic Manipulation Robotic Manipulation
University of North Carolina at Chapel Hill · Georgia Institute of Technology · Carnegie Mellon University
NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning
Feb 23, 2026Jiahui Fu, Junyu Nan, Lingfeng Sun +7Long-Horizon Robotic Manipulation Zero-Shot Robotic Manipulation
Robotics and AI Institute · Carnegie Mellon University · Brown University +1
Zero-shot Interactive Perception
Feb 20, 2026Venkatesh Sripada, Frank Guerin, Amir GhalamzanRobotic Perception Zero-Shot Robotic Manipulation
University of Surrey · University of Sheffield
EgoPush: Egocentric Multi-Object Rearrangement for Mobile Robots via Constrained Teacher Observability
Feb 20, 2026Boyuan An, Zhexiong Wang, Yipeng Wang +4Robotic Manipulation Robot Learning
New York University
MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation
Feb 18, 2026Mehrshad Taji, Arad Mahdinezhad Kashani, Iman Ahmadi +3Large Language Model-Based Robot Planning Robotic Manipulation
Department of Electrical Engineering Sharif University of Technology
SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation
Feb 18, 2026Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg +1Zero-Shot Robotic Manipulation Robot Skill Learning
Cornell University · Stanford University
One Hand to Rule Them All: Canonical Representations for Unified Dexterous Manipulation
Feb 18, 2026Zhenyu Wei, Yunchao Yao, Mingyu DingCross-Embodiment Robot Learning Robotic Grasping
University of North Carolina at Chapel Hill
Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation
Feb 17, 2026Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki +1Robotic Manipulation Robot Skill Learning
Carnegie Mellon University
World Action Models are Zero-shot Policies
Feb 17, 2026Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng +33World Action Models Zero-Shot Robotic Manipulation
NVIDIA
UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs
Feb 13, 2026Haichao Liu, Yuanjiang Xue, Yuheng Zhou +5Robotic Manipulation Long-Horizon Robotic Manipulation
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Automation, Tsinghua University, China
DiffMorphGrasp: A Diffusion Framework Conditioned on Hand Morphology for Cross-Embodiment Grasp Generation
Jan 31, 2026Zhiyuan Wu, Xiangyu Zhang, Zhuo Chen +3Robotic Grasping Cross-Embodiment Robot Learning
Department of Engineering, King’s College London, Strand, London, WC2R 2LS, United Kingdom · Imperial College London, London, SW7 2AZ, United Kingdom
Agentic Scene Policies
Sep 23, 2025Sacha Morin, Kumaraditya Gupta, Mahtab Sandhu +4Robot Policy Generalization Robot Learning
Université de Montréal · Mila - Quebec AI Institute · Sapienza University of Rome

Feb 28, 2026J/K move · Enter open · S save

Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation

Zilong Xie, Jingyu Gong, Xin Tan, Zhizhong Zhang+1

School of Computer Science and Technology, East China Normal University, Shanghai, China · Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, Chongqing, China · Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China

Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation VLMs for Robotics Multimodal RAG Object Pose Estimation

Open paper AskPDF arXiv

Existing end-to-end approaches of robotic manipulation often lack generalization to unseen objects or tasks due to limited data and poor interpretability. While recent Multimodal Large Language Models (MLLMs) demonstrate strong commonsense reasoning, they struggle with geometric and spatial understanding required for pose prediction. In this paper, we propose RobMRAG, a 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation (MRAG) framework for zero-shot robotic manipulation. Specifically, we construct a multi-source manipulation knowledge base containing object contact frames, task completion frames, and pose parameters. During inference, a Hierarchical Multimodal Retrieval module first employs a three-priority hybrid retrieval strategy to find task-relevant object prototypes, then selects the geometrically closest reference example based on pixel-level similarity and Instance Matching Distance (IMD). We further introduce a 3D-Aware Pose Refinement module based on 3D Gaussian Splatting into the MRAG framework, which aligns the pose of the reference object to the target object in 3D space. The aligned results are reprojected onto the image plane and used as input to the MLLM to enhance the generation of the final pose parameters. Extensive experiments show that on a test set containing 30 categories of household objects, our method improves the success rate by 7.76% compared to the best-performing zero-shot baseline under the same setting, and by 6.54% compared to the state-of-the-art supervised baseline. Our results validate that RobMRAG effectively bridges the gap between high-level semantic reasoning and low-level geometric execution, enabling robotic systems that generalize to unseen objects while remaining inherently interpretable.