Robot Manipulation Benchmarks

Ask about this topic

Overview

They range from simulated tabletop grasping and placement suites such as LIBERO and RoboCasa to real-robot tests of insertion, dexterous hands, bimanual work, mobile manipulation, and deformable or industrial objects. Common designs pair standardized task protocols and demonstrations with controlled shifts in objects, scenes, cameras, dynamics, or instructions; evaluation increasingly adds stage-level progress, recovery, motion quality, force and tactile interaction, and temporal safety to task success. Researchers use simulation, real-world rollouts, and matched sim-to-real protocols to compare imitation- and reinforcement-learning policies, vision-language-action models, and planning or memory systems. The central challenge is making scores both reproducible and diagnostic: success on fixed tasks may hide shortcuts, unsafe behavior, weak generalization, and gaps between simulated and physical performance.

Latest papers

CardsList

316 papers85 papers in the last four weeks, up 81% on the four weeks before. 0.5% of all new papers.

Feb 20, 2026

Robotic Fruits with Tunable Stiffness and Sensing: Towards a Methodology for Developing Realistic Physical Twins of Fruits

The global agri-food sector faces increasing challenges from labour shortages, high consumer demand, and supply-chain disruptions, resulting in substantial losses of unharvested produce. Robotic harvesting has emerged as a promising alternative; however, evaluating and training soft grippers for delicate fruits remains difficult due to the highly variable mechanical properties of natural produce. This makes it difficult to establish reliable benchmarks or data-driven control strategies. Existing testing practices rely on large quantities of real fruit to capture this variability, leading to inefficiency, higher costs, and waste. The methodology presented in this work aims to address these limitations by developing tunable soft physical twins that emulate the stiffness characteristics of real fruits at different ripeness levels. A fiber-reinforced pneumatic physical twin of a kiwi fruit was designed and fabricated to replicate the stiffness at different ripeness levels. Experimental results show that the stiffness of the physical twin can be tuned accurately over multiple trials (97.35 - 99.43% accuracy). Gripping tasks with a commercial robotic gripper showed that sensor feedback from the physical twin can reflect the applied gripping forces. Finally, a stress test was performed over 50 cycles showed reliable maintenance of desired stiffness (0.56 - 1.10% error). This work shows promise that robotic physical twins could adjust their stiffness to resemble that of real fruits. This can provide a sustainable, controllable platform for benchmarking and training robotic grippers.

Saitarun Nadipineni, Keshav Pandiyan, Kaspar Althoefer +2

School of Engineering and Materials Science, Queen Mary University of London, Mile End Rd, Bethnal Green, London E1 4NS, United Kingdom. · Department of Robotics, Ritsumeikan University, Japan

Soft RoboticsAgricultural RoboticsRobot Manipulation BenchmarksRobotic Grasping
AskPDF
Feb 19, 2026

Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success

3D reconstruction serves as the foundational layer for numerous robotic perception tasks, including 6D object pose estimation and grasp pose generation. Modern 3D reconstruction methods for objects can produce visually and geometrically impressive meshes from multi-view images, yet standard geometric evaluations do not reflect how reconstruction quality influences downstream tasks such as robotic manipulation performance. This paper addresses this gap by introducing a large-scale, physics-based benchmark that evaluates 6D pose estimators and 3D mesh models based on their functional efficacy in grasping. We analyze the impact of model fidelity by generating grasps on various reconstructed 3D meshes and executing them on the ground-truth model, simulating how grasp poses generated with an imperfect model affect interaction with the real object. This assesses the combined impact of pose error, grasp robustness, and geometric inaccuracies from 3D reconstruction. Our results show that reconstruction artifacts significantly decrease the number of grasp pose candidates but have a negligible effect on grasping performance given an accurately estimated pose. Our results also reveal that the relationship between grasp success and pose error is dominated by spatial error, and even a simple translation error provides insight into the success of the grasping pose of symmetric objects. This work provides insight into how perception systems relate to object manipulation using robots.

Varun Burde, Pavel Burget, Torsten Sattler

Faculty of Electrical Engineering, Czech Technical University in Prague, Czechia · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague, Czechia

Robot Manipulation BenchmarksRobotic GraspingObject Pose Estimation3D Reconstruction3D Mesh Reconstruction
AskPDF
Feb 14, 2026

Humanoid Hanoi: Investigating Shared Whole-Body Control for Skill-Based Box Rearrangement

We introduce Humanoid Hanoi, a challenge problem for long-horizon humanoid skill composition that requires a robot to repeatedly sequence pickup, locomotion, and placement skills to rearrange stacked boxes in simulation and on hardware. This symbolically simple task exposes a weakness in a common modular humanoid architecture, where reusable high-level skills are learned on top of a shared, task-agnostic whole-body controller (WBC). While such a WBC provides a unified low-level interface for skill composition, it may not be sufficiently hardened to execute the full collection of learned skills under domain randomization and long-horizon closed-loop execution. We compare strategies for addressing this issue, including skill-specific WBC fine-tuning, residual low-level adaptation, and task-driven data aggregation that expands the shared-WBC training distribution with closed-loop trajectories induced by learned skills while preserving its original tracking objective. In our evaluated system, the shared-WBC approach achieves the strongest long-horizon robustness. A direct transition-smoothing intervention further shows that low-level controller handoffs account for a substantial part of the composition loss observed with non-shared controllers. Project page: https://osudrl.github.io/Humanoid_Hanoi/

Minku Kim, Kuan-Chia Chen, Aayam Shrestha +3

The authors are with the Collaborative Robotics and Intelligent Systems Institute, Oregon State University, Corvallis, Oregon, 97331, USA

Humanoid Robot ControlWhole-Body ControlLong-Horizon Robotic ManipulationRobot Manipulation Benchmarks
AskPDF
Feb 5, 2026

MobileManiBench: Simplifying Model Verification for Mobile Manipulation

Vision-language-action models have advanced robotic manipulation but remain constrained by reliance on the large, teleoperation-collected datasets dominated by the static, tabletop scenes. We propose a simulation-first framework to verify VLA architectures before real-world deployment and introduce MobileManiBench, a large-scale benchmark for mobile-based robotic manipulation. Built on NVIDIA Isaac Sim and powered by reinforcement learning, our pipeline autonomously generates diverse manipulation trajectories with rich annotations (language instructions, multi-view RGB-depth-segmentation images, synchronized object/robot states and actions). MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories. This design enables controlled, scalable studies of robot embodiments, sensing modalities, and policy architectures, accelerating research on data efficiency and generalization. We benchmark representative VLA models and report insights into perception, reasoning, and control in complex simulated environments, with all code, datasets, and models publicly released.

Wenbo Wang, Fangyun Wei, QiXiu Li +5

University of Sydney · 1Microsoft Research Asia · 3Tsinghua University

Mobile ManipulationRobotics SimulationSynthetic Data Generation for RoboticsVision-Language-Action ModelsRobot Manipulation BenchmarksRobot Manipulation Datasets
AskPDF
Jan 29, 2026

DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation

Manipulating dynamic objects remains an open challenge for Vision-Language-Action (VLA) models. Although recent VLAs generalize well in static manipulation, dynamic scenes introduce a latency-induced perception-execution mismatch: object states continue to evolve during inference, making actions predicted from past observations stale at execution time. We present DynamicVLA, a latency-aware VLA model for dynamic object manipulation. It combines a compact 0.4B architecture and convolutional vision encoder for efficient multimodal inference with a continuous inference schedule that overlaps reasoning and execution for non-blocking control. Latent-aware Action Streaming then discards latency-invalid action prefixes and executes only the temporally valid suffix of each predicted chunk, preserving action-time alignment under dynamic object motion. To fill the missing foundation of dynamic manipulation data, we introduce the Dynamic Object Manipulation (DOM) benchmark, built with an automated collection pipeline that gathers 200K synthetic episodes across 2.8K scenes and 206 objects, and enables fast collection of 2K real-world episodes without teleoperation. Extensive evaluations in simulation and on real robots show that DynamicVLA improves dynamic manipulation success under changing object motion, perception-heavy instructions, and unseen motion patterns.

Haozhe Xie, Beichen Wen, Jiarui Zheng +4

Inference-Time OptimizationEfficient VLM InferenceEfficient VLA Model InferenceVision-Language-Action ModelsRobot Manipulation Benchmarks
AskPDF
Jan 29, 2026

From Instruction to Event: Sound-Triggered Mobile Manipulation

Current mobile manipulation research predominantly follows an instruction-driven paradigm, where robots rely on predefined textual commands to execute tasks. However, this setting confines robots to a passive role, limiting robotic autonomy and the ability to react to dynamic environmental events. To address these limitations, we introduce Sound-Triggered Mobile Manipulation (STMM), where robots must actively perceive and interact with sound-emitting objects without explicit action instructions. To support STMM, we develop Habitat-Echo, a simulation platform that integrates sound rendering with physical interaction. We further propose a hierarchical baseline that translates high-level planning into low-level executions, where a task planner predicts a skill chain for policy models to execute sequentially. Experiments indicate the feasibility of perceiving auditory events and executing corresponding physical interactions without explicit instructions. Notably, in challenging multi-event scenarios, the robot successfully isolates the primary sources from overlapping acoustic interference to execute the first interactions, and subsequently proceeds to manipulate the secondary objects. These new challenges of planning-to-execution position STMM as a measurable research direction.The code and datasets will be released upon acceptance.

Hao Ju, Shaofei Huang, Hongyu Li +4

University of Macau · Beihang University · Hefei University of Technology

Robotics SimulationMobile RoboticsRobotic ManipulationRobot Task PlanningRobot Manipulation Benchmarks
AskPDF
Jan 26, 2026

Eval-Actions: Fine-Grained Execution Quality Evaluation for Robotic Manipulation

Although Vision--Action (VA) and Vision--Language--Action (VLA) policies have advanced robotic manipulation, their evaluation remains dominated by binary success rates, which obscure process-level differences among executions that complete the same task. We introduce Eval-Actions, a diagnostic evaluation methodology and real-robot benchmark for fine-grained execution-quality assessment of learned manipulation policies. Eval-Actions combines criteria-based Expert Grading (EG), Rank-Guided (RG) labels that align measurable motion indicators with expert rankings, and Chain-of-Thought-style (CoT) annotations that explain observable quality differences. The benchmark contains 13K+ teleoperated and policy-generated real-robot episodes covering 150+ tasks and approximately 52 hours of recordings with RGB-D videos, robot-state trajectories, task descriptions, and success/failure labels. Its densely annotated subset provides EG/RG/CoT supervision for training and evaluation. We further provide AutoEval, a reference multimodal evaluator that predicts quality scores, task outcomes, and diagnostic explanations from RGB temporal evidence and compact kinematic summaries. On the annotated Eval-Actions test split, AutoEval-S achieves Spearman rank correlations (SRCCs) of 0.81 and 0.84 under EG and RG, with success detection accuracies of 90.6% and 91.0%; AutoEval-P reaches 0.70 SRCC under CoT. Analyses of expert consistency, physical-metric baselines, modality ablations, structured generalization, and offline policy ranking show that Eval-Actions provides standardized, interpretable diagnostic signals complementary to success-rate evaluation.

Mengyuan Liu, Juyi Sheng, Peiming Li +4

Robotic Policy EvaluationRobotic ManipulationRobot Manipulation Benchmarks
AskPDF
Jan 5, 2026

Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot

The development of robust and generalizable robot learning models is critically contingent upon the availability of large-scale, diverse training data and reliable evaluation benchmarks. Collecting data in the physical world poses prohibitive costs and scalability challenges, and prevailing simulation benchmarks frequently suffer from fragmentation, narrow scope, or insufficient fidelity to enable effective sim-to-real transfer. To address these challenges, we introduce Genie Sim 3.0, a unified simulation platform for robotic manipulation. We present Genie Sim Generator, a large language model (LLM)-powered tool that constructs high-fidelity scenes from natural language instructions. Its principal strength resides in rapid and multi-dimensional generalization, facilitating the synthesis of diverse environments to support scalable data collection and robust policy evaluation. We introduce the first benchmark that pioneers the application of LLM for automated evaluation. It leverages LLM to mass-generate evaluation scenarios and employs Vision-Language Model (VLM) to establish an automated assessment pipeline. We also release an open-source dataset comprising more than 10,000 hours of synthetic data across over 200 tasks. Through systematic experimentation, we validate the robust zero-shot sim-to-real transfer capability of our open-source dataset, demonstrating that synthetic data can server as an effective substitute for real-world data under controlled conditions for scalable policy training. For code and dataset details, please refer to: https://github.com/AgibotTech/genie_sim.

Chenghao Yin, Da Huang, Di Yang +16

Robotic Data CollectionRobotics SimulationSynthetic Data Generation for RoboticsSynthetic Benchmark GenerationRobot Manipulation Benchmarks
AskPDF
Dec 30, 2025

World In Your Hands: A Large-Scale and Open-Source Ecosystem for Learning Human-Centric Manipulation in the Wild

We introduce World In Your Hands (WIYH), a large-scale open-source ecosystem comprising over 1,000 hours of human manipulation data collected in-the-wild with millimeter-scale motion accuracy. Specifically, WIYH includes (1) the Oracle Suite, a wearable data collection kit with an auto-labeling pipeline for accurate motion capture; (2) the WIYH Dataset, featuring over 1,000 hours of multimodal manipulation data across hundreds of skills in diverse real-world scenarios; and (3) extensive annotations and benchmarks supporting tasks from perception to action. Furthermore, experiments based on the WIYH ecosystem show that integrating WIYH's human-centric data improves robotic manipulation success rates from 8% to 60% in cluttered scenes. World In Your Hands provides a foundation for advancing human-centric data collection and cross-embodiment policy learning. All data and hardware design will be open-source.

Yupeng Zheng, Jichao Peng, Weize Li +22

TARS Robotics

Human-Object InteractionRobot Skill LearningRobotic Data CollectionCross-Embodiment Robot LearningRobotic ManipulationRobot Manipulation Benchmarks+2
AskPDF
Dec 12, 2025

UniBYD: A Unified Framework for Learning Robotic Manipulation Across Embodiments Beyond Imitation of Human Demonstrations

In embodied intelligence, the embodiment gap between robotic and human hands brings significant challenges for learning from human demonstrations. Although some studies have attempted to bridge this gap using reinforcement learning, they remain confined to merely reproducing human manipulation, resulting in limited task performance. Moreover, current methods struggle to support diverse robotic hand configurations. In this paper, we propose UniBYD, a unified framework that uses a dynamic reinforcement learning algorithm to discover manipulation policies aligned with the robot's physical characteristics. To enable consistent modeling across diverse robotic hand morphologies, UniBYD incorporates a unified morphological representation (UMR). Building on UMR, we design a dynamic PPO with an annealed reward schedule, enabling reinforcement learning to transition from offline-informed imitation of human demonstrations to online-adaptive exploration of policies better adapted to diverse robotic morphologies, thereby going beyond mere imitation of human hands. To address the severe state drift caused by the incapacity of early-stage policies, we design a hybrid Markov-based shadow engine that provides fine-grained guidance to anchor the imitation within the expert's manifold. To evaluate UniBYD, we propose UniManip, the first benchmark for cross-embodiment manipulation spanning diverse robotic morphologies. Experiments demonstrate a 44.08% average improvement in success rate over the current state-of-the-art. Our project page is https://zhanheng-creator.github.io/UniBYD.

Tingyu Yuan, Biaoliang Guan, Wen Ye +8

CASIA · UCAS · XJTU +3

Cross-Embodiment Robot LearningRobotic RLRobot Imitation LearningRobot Manipulation Benchmarks
AskPDF
Nov 10, 2025

SlotVLA: Towards Modeling of Object-Relation Representations in Robotic Manipulation

Inspired by how humans reason over discrete objects and their relationships, we explore whether compact object-centric and object-relation representations can form a foundation for multitask robotic manipulation. Most existing robotic multitask models rely on dense embeddings that entangle both object and background cues, raising concerns about both efficiency and interpretability. In contrast, we study object-relation-centric representations as a pathway to more structured, efficient, and explainable visuomotor control. Our contributions are two-fold. First, we introduce LIBERO+, a fine-grained benchmark dataset designed to enable and evaluate object-relation reasoning in robotic manipulation. Unlike prior datasets, LIBERO+ provides object-centric annotations that enrich demonstrations with box- and mask-level labels as well as instance-level temporal tracking, supporting compact and interpretable visuomotor representations. Second, we propose SlotVLA, a slot-attention-based framework that captures both objects and their relations for action decoding. It uses a slot-based visual tokenizer to maintain consistent temporal object representations, a relation-centric decoder to produce task-relevant embeddings, and an LLM-driven module that translates these embeddings into executable actions. Experiments on LIBERO+ demonstrate that object-centric slot and object-relation slot representations drastically reduce the number of required visual tokens, while providing competitive generalization. Together, LIBERO+ and SlotVLA provide a compact, interpretable, and effective foundation for advancing object-relation-centric robotic manipulation.

Taisei Hanyu, Nhat Chung, Huy Le +10

University of Arkansas, USA · FPT Software AI Center, Vietnam · University of Stuttgart, Germany +5

Representation LearningObject-Centric Representation LearningMulti-Task Robotic ManipulationRobotic ManipulationRobot Manipulation BenchmarksSlot Attention+1
AskPDF
Oct 20, 2025

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems often follow a dual-system paradigm, where System 2 performs high-level reasoning and System 1 handles low-level control. We refer to System 2 as the embodied brain, the cognitive core for decision-making in manipulation. Although evaluating this embodied brain is crucial, existing benchmarks mainly measure execution success or cover only limited aspects of high-level cognition and task realism. We introduce RoboBench, a benchmark for evaluating multimodal large language models (MLLMs) as embodied brains. RoboBench covers five dimensions: Instruction Comprehension, Perception Reasoning, Generalized Planning, Affordance Prediction, and Failure Analysis. It spans 14 capabilities, 25 tasks, and 6,092 QA pairs. To improve realism, it draws from large-scale real robotic data and in-house collection across diverse embodiments, attribute-rich objects, multi-view scenes, and memory-driven navigation. For planning, RoboBench introduces an MLLM-as-world-simulator framework that assesses whether predicted plans can achieve critical object-state changes under physical and visual constraints, enabling more faithful evaluation of long-horizon reasoning than symbolic matching. Experiments on 18 state-of-the-art MLLMs reveal persistent limitations in implicit instruction understanding, spatiotemporal reasoning, cross-scenario planning, fine-grained affordance understanding, and failure diagnosis. We further analyze how embodied cognitive abilities relate to downstream robotic control. RoboBench offers a comprehensive scaffold for quantifying high-level cognition and guiding next-generation MLLMs toward more robust robotic intelligence.

Yulin Luo, Chun-Kai Fan, Menghang Dong +19

State Key Laboratory of Multimedia Information Processing, School of Computer2026 Science, Peking University · Institute for Brain and Intelligence, Fudan University · University of Science andJul Technology Beijing +2

Long-Horizon Robotic ManipulationMultimodal Large Language ModelsRobot Manipulation BenchmarksEmbodied AIMultimodal Model EvaluationEmbodied Reasoning
AskPDF
Oct 9, 2025

Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation

From loco-motion to dextrous manipulation, humanoid robots have made remarkable strides in demonstrating complex full-body capabilities. However, the majority of current robot learning datasets and benchmarks mainly focus on stationary robot arms, and the few existing humanoid datasets are either confined to fixed environments or limited in task diversity, often lacking human-humanoid interaction and lower-body locomotion. Moreover, there are a few standardized evaluation platforms for benchmarking learning-based policies on humanoid data. In this work, we present Humanoid Everyday, a large-scale and diverse humanoid manipulation dataset characterized by extensive task variety involving dextrous object manipulation, human-humanoid interaction, locomotion-integrated actions, and more. Leveraging a highly efficient human-supervised teleoperation pipeline, Humanoid Everyday aggregates high-quality multimodal sensory data, including RGB, depth, LiDAR, and tactile inputs, together with natural language annotations, comprising 10.3k trajectories and over 3 million frames of data across 260 tasks across 7 broad categories. In addition, we conduct an analysis of representative policy learning methods on our dataset, providing insights into their strengths and limitations across different task categories. For standardized evaluation, we introduce a cloud-based evaluation platform that allows researchers to seamlessly deploy their policies in our controlled setting and receive performance feedback. By releasing Humanoid Everyday along with our policy learning analysis and a standardized cloud-based evaluation platform, we intend to advance research in general-purpose humanoid manipulation and lay the groundwork for more capable and embodied robotic agents in real-world scenarios. Our dataset, data collection code, and cloud evaluation website are made publicly available on our project website.

Zhenyu Zhao, Hongyi Jing, Xiawei Liu +7

University of Southern California · Toyota Research Institute

Robot Manipulation DatasetsRobotic Data CollectionWhole-Body Robotic ManipulationRobot Manipulation BenchmarksHumanoid Robotics
AskPDF
Oct 7, 2025

BuilderBench: The Building Blocks of Intelligent Agents

Today's AI models learn primarily through mimicry and refining, so it is not surprising that they struggle to solve problems beyond the limits set by existing data. To solve novel problems, agents should acquire skills by exploring and learning through experience. Finding a scalable learning mechanism for developing agents that learn through interaction remains a major open problem. In this work, we introduce BuilderBench, a benchmark to accelerate research into agent training that centers open-ended exploration. BuilderBench requires agents to learn how to build any structure using blocks. BuilderBench is equipped with (1) a simulator of a robot interacting with various physical blocks, and (2) a task-suite with over 50 diverse target structures that are carefully curated to test an understanding of physics, mathematics, and long-horizon planning. Agents are provided with a target structure at the start, and can interact with the environment for multiple episodes to experiment and learn various skills for building the structure. Solving these tasks requires embodied reasoning in a way that is not reflected in words but rather in actions, experimenting with different strategies and piecing them together. Our experiments with multiple state-of-the-art frontier language model based agents and tabula rasa reinforcement learning algorithms show that these agents cannot solve any of the non-trivial tasks in the BuilderBench. Our analysis throws light on the lack of exploration abilities in these models.

Raj Ghugare, Roger Creus Castanyer, Catherine Ji +4

Princeton University · Mila – Quebec AI Institute · Université de Montréal

Long-Horizon Agent EvaluationRobot Manipulation BenchmarksLLM Agent TrainingEmbodied AIEmbodied Reasoning
AskPDF
Sep 30, 2025

RoboPilot: Generalizable Dynamic Robotic Manipulation with Dual-thinking Modes

Despite rapid progress in robotics, complex or long-horizon tasks remain a fundamental challenge. Most current approaches follow an open-loop paradigm with limited reasoning and no feedback, resulting in poor robustness to environmental changes and severe error accumulation. We present RoboPilot, a dual-thinking closed-loop agentic framework for robotic manipulation that supports adaptive reasoning for complex tasks in real-world dynamic environments. RoboPilot leverages primitive actions for structured task planning and flexible action generation as a agentic system, while introducing feedback to enable replanning from dynamic changes and execution errors. Chain-of-Thought reasoning further enhances high-level task planning and guides low-level action generation. The agentic system dynamically switches between fast and slow thinking to balance efficiency and accuracy. To systematically evaluate the robustness of RoboPilot in diverse robot manipulation scenarios, we introduce RoboPilot-Bench, a benchmark spanning 21 tasks across 10 categories, including infeasible-task recognition and dynamic recovery. Experiments show that RoboPilot outperforms state-of-the-art baselines by 11% in task success rate, and the real-world deployment on an industrial robot further demonstrates its robustness.

Xinyi Liu, Mohammadreza Fani Sani, Zewei Zhou +3

Microsoft, Denmark · Technical University of Denmark · University of California, Los Angeles

Robotic ManipulationDynamic Robotic ManipulationLong-Horizon Robotic ManipulationRobot Task PlanningGeneralization in Robotic ManipulationRobot Manipulation Benchmarks
AskPDF
May 31, 2025

XYZ-IBD: Benchmarking Robust 6D Object Pose Estimation under Real-World Industrial Complexity

While current 6D pose estimation benchmarks have reached near-saturation on household objects, they often fail to capture the stochastic and optical complexities of industrial environments. We introduce XYZ-IBD, a high-precision benchmark for object detection and 6D pose estimation specifically designed for industrial bin-picking. XYZ-IBD addresses the domain gap by providing 75 multi-view real-world scenes containing approximately 273k annotated instances of metallic, symmetrical, and specular objects. Unlike existing datasets, our benchmark features high-density stochastic stacking and multi-instance ambiguity, reflecting authentic robotic manipulation challenges. We employ a rigorous multi-stage and semi-automatic annotation pipeline, ensuring sub-millimeter annotation accuracy. The annotations are validated through our designed error quantification scheme, securing the reliability of the annotation quality. In addition to real-world evaluation data, we provide a large-scale complementary synthetic training set that is rendered under a realistic bin-picking simulation. Benchmarking state-of-the-art (SOTA) methods for 2D detection and 6D pose estimation reveals a significant performance degradation compared to standard household benchmarks, highlighting the unsolved challenges of industrial vision. XYZ-IBD establishes a new frontier for robust pose estimation in complex, high-occlusion, and reflective scenarios. The dataset and benchmark are publicly available at https://xyz-ibd.github.io.

Junwen Huang, Jiaqi Hu, Peter KT Yu +3

Technical University of Munich · Munich Center for Machine Learning (MCML) · XYZ Robotics +2

Object Pose EstimationRobotic ManipulationRobot Manipulation BenchmarksRobotic Perception
AskPDF