A Checklist for Deploying Robots in Public: Articulating Tacit Knowledge in the HRI Community

Capability claims in robot manipulation are unverifiable, and existing leaderboards are each closed, simulated, or gameable. We propose HATCH, a robot escape room: an openly specified, physically replicable, sensor-scored, and unrehearsable benchmark. Tasks span five skill categories at three difficulty levels and are chained so that the sensor confirming one task's completion unlocks the next, turning a room into a single long-horizon evaluation. Success is confirmed by instrumentation rather than human judgment, scoring is embodiment-agnostic, and object placements are randomized by a run-day seed. Entrants submit attested self-runs (uncut video, sensor logs, and the seed), with occasional organized events for calibration. Task specifications, build guide, bill of materials, and scoring rubrics are released under an open license.

As the presence of autonomous robots in public spaces increases-whether navigating campus walkways or neighborhood sidewalks-understanding how to carefully study these robots becomes critical. While HRI research has conducted field studies in public spaces, these are often limited to controlled experiments with prototype robots or structured observational methods, such as the Wizard of Oz technique. However, the autonomous mobile robots we encounter today, particularly delivery robots, operate beyond the control of researchers, navigating dynamic routes and unpredictable environments. To address this challenge, a more deliberate approach is required. Drawing inspiration from public realm ethnography in urban studies, geography, and sociology, this paper proposes the Walk-Along with Robots (WawR) methodology. We outline the key features of this method, the steps we applied in our study, the unique insights it offers, and the ways it can be evaluated. We hope this paper stimulates further discussion on research methodologies for studying autonomous robots in public spaces.

This study presents an interdisciplinary process for benchmarking service robots in public everyday environments, addressing the gap between laboratory metrics and real-world operation at technology readiness levels 6-7. Three robots were evaluated in three use cases: outdoor park cleaning, pedestrian underpass cleaning, and interactive assistance in a public library. Over three years (2023-2025), seven benchmarking events (a consensus workshop and six on-site evaluations, two per use case) assessed the robots in increasingly realistic settings, from controlled conditions to public operation. An interdisciplinary panel of experts in robotics, human-robot interaction, safety, and economic viability consolidated use-case-specific metrics into a shared evaluation concept refined across three phases. Rather than ranking the robots, the evaluation examined how task fulfillment, interaction quality, safety, and economic viability interact in the transition from laboratory prototype to operational system. Direct quantitative comparability across the heterogeneous use cases remained limited, characteristic of benchmarking at this maturity level. The process yielded cross-case insights: increasing environmental realism introduces challenges that uniform metrics capture only in part; the interaction-quality guideline works best as a modular instrument adapted to each use case; safety was assessed by occupational-safety and product-certification experts who adapted industrial test procedures for public settings; and economic viability proved a cross-cutting dimension. We conclude that benchmarking at higher technology readiness levels is well served by a shared procedure for selecting and documenting metrics across these four categories, with safety as mandatory, rather than by a fixed set of metrics: generalizability lies in the procedure rather than in identical metrics.