Benchmarking Robots for Everyday Environments: From Lab Experiments to Real-World Operations
Organizations: Autonomous Intelligent Systems, University of Bonn, Germany · FZI Forschungszentrum Informatik, Germany · School of Business and Economics, Freie Universität Berlin, Germany · Institute for Occupational Safety and Health of the German Social Accident Insurance (IFA), Germany · Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Germany · Ruhr West University of Applied Sciences, Germany · Human Factors and Engineering Psychology, Johannes Gutenberg University Mainz, Germany · Human-Factors-Consult GmbH, Germany · T ÜV S ÜD Product Service GmbH, Germany · Institute for Information Processing Technologies, Karlsruhe Institute of Technology, Germany · Machine Intelligence and Robotics Lab, Karlsruhe Institute of Technology, Germany
Abstract
This study introduces an interdisciplinary framework for benchmarking robots deployed in public environments, addressing the gap between traditional laboratory metrics and real-world benchmarking requirements. We evaluate three distinct robots across diverse use cases - outdoor park cleaning, pedestrian underpass cleaning, and interactive library assistance - each representing unique challenges in public daily life. Over a three-year benchmarking process (2023-2025) comprising seven benchmarking events, a consensus workshop and six on-site evaluations (two per use case), we utilized realistic indoor and outdoor test environments to assess not only technical performance but also the broader implications of deploying robots in unstructured, human-centric settings. An expert panel, spanning robotics, human-robot interaction, safety, and economics, systematically developed and refined an evaluation concept to analyze the transition from laboratory prototypes to operational systems. Our findings highlight critical factors for successful deployment, including task fulfillment, interaction quality, safety, and economic feasibility. This work provides actionable insights for researchers and practitioners aiming to bridge the gap between robotic innovation and real-world applicability.