cs.RONov 9, 2026

Benchmarking Real-World Robotic Manipulation through Stochastic Challenges

Authors: Elias Gläßer, Gülhan Şikaroğlu, Pavlo Kupyn, Marc Pollefeys, Zuria Bauer, Hermann Blum

Organizations: University of Bonn · Lamarr Institute · ETH Zürich · Microsoft

Abstract

A challenging bottleneck in manipulation research is reproducibility and comparability of methods in real-world environments. Since manipulation is an interactive task, it cannot be evaluated on a static dataset of observations. Benchmarking in simulation is a common remedy, but cannot cover the whole complexity of real-world tests. This paper proposes a new paradigm of stochastic challenges. Like the well-known YCB objects, it is based on reproducible real-world scenes and therefore allows system-level innovation and comparability between different platforms. Due to their stochastic nature, our challenges however, mimic the rigor and surprise of hidden test sets. To this end, we show how different robots in different labs can be compared based on their success rate over different random task samples, if these samples come from the same distribution. We propose stochastic challenges as a general paradigm for robot benchmarking and implement this paradigm in a novel benchmark for mobile manipulation, on which we compare two robots tested in two different labs.

Figures & tables

Explore similar work

CardsList
  1. ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning

    Mar 4, 2026Yiting Chen, Kenneth Kimble, Edward H. Adelson +20Robotic ManipulationRobot Manipulation Benchmarks

  2. What Are We Actually Benchmarking in Robot Manipulation?

    Jun 2, 2026Tianchong Jiang, Xiangshan Tan, Samuel Wheeler +3Benchmark DesignBenchmark Contamination