While robots deployed in real-world environments inevitably experience interaction failures, understanding how users respond through verbal and non-verbal behaviors remains under-explored in human-robot interaction (HRI). This gap is particularly significant in healthcare-inspired settings, where interaction failures can directly affect task performance and user trust. We present the Robot Failures in Medical HRI (RFM-HRI) Dataset, a multimodal dataset capturing dyadic interactions between humans and robots embodied in crash carts, where communication failures are systematically induced during item retrieval tasks. Through Wizard-of-Oz studies with 41 participants across laboratory and hospital settings, we recorded responses to four failure types (speech, timing, comprehension, and search) derived from three years of crash-cart robot interaction data. The dataset contains 214 interaction samples including facial action units, head pose, speech transcripts, and post-interaction self-reports. Our analysis shows that failures significantly degrade affective valence and reduce perceived control compared to successful interactions. Failures are strongly associated with confusion, annoyance, and frustration, while successful interactions are characterized by surprise, relief, and confidence in task completion. Emotional responses also evolve across repeated failures, with confusion decreasing and frustration increasing over time. This work contributes (1) a publicly available multimodal dataset (RFM-HRI), (2) analysis of user responses to different failure types and preferred recovery strategies, and (3) a crash-cart retrieval scenario enabling systematic comparison of recovery strategies with implications for safety-critical failure recovery. Our findings provide a foundation for failure detection and recovery methods in embodied HRI.
Figures & tables
Figure 1. Data collection environment at university campus laboratory (left) and at the hospital (right) Participants faces are blacked out in compliance with our approved IRB protocol.
Dataset
Annotations
Dataset Modalities
Healthcare Domain?
Experts and Non-Experts?
Human Robot Interaction?
Controlled Failure Injection?
Reactions to Failures?
Oops! ( Epstein et al., 2020 )
Videos Annotated for Action Intention Variability
Video
×
×
×
×
✓
RAF-DB ( Li et al., 2017 )
Compound Expression Images and Labels in the Wild
Image
×
×
×
×
×
CK+ ( Lucey et al., 2010 )
Facial Action Coding System, Action Units, and Labels for Posed and Unposed Expressions
Video, Image, FACS/AUs
×
×
×
×
×
Oulu-CASIA ( Zhao et al., 2011 )
Facial Expression Dataset, Across Illumination Conditions
Image (Infrared and Visible Light)
×
×
×
×
×
AFEW 7.0 ( Dhall et al., 2012 )
Emotion-Labeled Movie Clips
Video, Audio
×
×
×
×
×
BP4D-Spontaneous ( Zhang et al., 2013 )
2D and 3D Dataset Containing Facial Action Units, Video Features, and Head Poses
Video (2D and 3D), FACS/AUs, Pose
×
×
×
×
×
Table 1. Comparison of representative datasets spanning facial expression analysis, conversational errors, and human–robot interaction. In contrast to prior work, RFM-HRI (Ours) provides a healthcare-centered HRI dataset featuring a novel crash cart robot, controlled failure injection, and rich multimodal annotations of human reactions, including speech, FACS and AUs, head pose, gaze estimation, and post-hoc survey responses—collected from both expert and non-expert participants.
Figure 2. Robotic crash cart platforms in real-world hospital deployment (left) and controlled university campus laboratory evaluation (right).
Razor, Medical Tape, Surgical Gloves, Line Draw Arterial Blood Sample Syringe
Red pencils, wires, blue markers, blue pencils
Drawer 4
Blanket, Scissors, Face Masks
White board cleaner, circuit board, calculator, gloves, playstation controller, mouse
Drawer 5
1 mL Tuberculin Syringe, 6 mL Syringe, 500 mL IV Bag
Scissors, pliers, book, wipes, orange multimeter, hand sanitizer
Drawer 6
Drawer 6 is Empty in the Hospital Cart
No Drawer 6 on In-Lab Cart
Table 2. Drawer-by-drawer inventory comparison of crash cart contents in the lab prototype vs hospital reference cart, supporting multimodal data collection and failure-induction studies.
Figure 3. The Wizard-of-Oz (WoZ) Interface components (Part 1 of 4).
Figure 4. The Wizard-of-Oz (WoZ) Interface components (Part 2 of 4).
Figure 5. Post-study survey form used to collect affective responses, including categorical emotion selection and Self-Assessment Manikin (SAM) ratings, as well as participants’ preferred recovery strategies following each trial.
Figure 6. Dataset organization and per-trial multimodal composition. The cleaned dataset is organized into five trial outcome categories (four injected robot failure types and success). For each trial, we release a self-contained sample comprising time-indexed visual behavioral features (MediaPipe-derived facial action units, gaze, pose, and landmarks), structured post-trial survey responses, and a natural-language task transcript. An overview of the dataset structure and accessible modalities is illustrated in Fig. 6 . Together, these views summarize both the dataset-level structure and the multimodal signals available for modeling trial outcomes and recovery preferences, while abstracting away file-level organization.
Understanding how users perceive and respond to robot failures is essential for building robust and trustworthy robot systems. Prior work, however, (i) often treats failures as independent events, (ii) emphasizes binary failure detection, (iii) with rule-based recovery modeling. We present REPAIR-Bench, built on 214 interaction trials from 41 participants, the benchmark spans four induced failure types and provides synchronized facial action units, head pose, speech transcripts, and post-interaction affect and recovery reports. The benchmark spans three novel evaluation tasks that jointly capture the lifecycle of failure in human-robot interaction (HRI): (i) failure detection over inter-dependent interaction sessions, modeling longitudinal user adaptation across repeated failures; (ii) visual failure-type classification beyond binary success/failure formulations; and (iii) user-centered recovery prediction, inferring users' preferred recovery strategies from interaction context rather than relying on manually designed or rule-based strategies. In baseline experiments, hierarchical recurrent modeling improved failure detection over a single-session model (strict F1: 0.80 vs. 0.68), achieved a failure localization mean signed error of -0.51 s, median absolute error of 2.97 s and, for recovery prediction, a QLoRA-tuned Mistral-7B reached Hit@5=0.76 and F1@5=0.32. REPAIR-Bench provides both the HRI and Medical HRI communities with a standardized framework for (1) evaluating robot failures and (2) building transparent, adaptive, and trustworthy recovery systems.
Robots can recover from failures by asking bystanders for help, but effective human-in-the-loop recovery requires communication that accounts for differences in people's knowledge. Prior inverse-semantics work generates requests using a single listener model, leaving differences in listener knowledge untested. We introduce Listener Differences in Human-Robot Interaction (LD-HRI), a game, dataset, and benchmark that evaluates speakers through human listener performance. Our evaluation examines request properties, large language model (LLM) speakers, and inverse-semantics request-selection algorithms under controlled differences in listener information. The corpus contains 446 human-written requests and 1{,}302 listener trials. We additionally evaluated 24 frozen LLM-written requests with 70 human listeners across 560 trials. Novice success is descriptively higher with model-written requests across all four tasks, yet both request sources leave substantial expert--novice gaps, including 16 percentage points for LLM requests. LD-HRI makes these gaps measurable, providing a foundation for designing more robust communication in human-robot and human-agent interaction.
Promise Ekpo, Teju Vijay, Dhruv Mandalik +5
Cornell University, New York, New York, USA · Cornell Tech, New York, New York, USA · Brown University, Providence, Rhode Island, USA
Robots fail, potentially leading to a loss in the robot's perceived reliability (PR), a measure correlated with trustworthiness. In this study we examine how various kinds of failures affect the PR of the robot differently, and how this measure recovers without explicit social repair actions by the robot. In a preregistered and controlled online video study, participants were asked to predict a robot's success in a pick-and-place task. We examined manipulation failures (slips), freezing (lapses), and three types of incorrect picked objects or place goals (mistakes). Participants were shown one of 11 videos -- one of five types of failure, one of five types of failure followed by a successful execution in the same video, or a successful execution video. This was followed by two additional successful execution videos. Participants bet money either on the robot or on a coin toss after each video. People's betting patterns along with a qualitative analysis of their survey responses highlight that mistakes are less damaging to PR than slips or lapses, and some mistakes are even perceived as successes. We also see that successes immediately following a failure have the same effect on PR as successes without a preceding failure. Finally, we show that successful executions recover PR after a failure. Our findings highlight which robot failures are in higher need of repair in a human-robot interaction, and how trust could be recovered by robot successes.
Andrew Violette, Zhanxin Wu, Haruki Nishimura +5
Department of Computer Science, Cornell University · Toyota Research Institute (TRI). · Sibley School of Mechanical and Aerospace Engineering, Cornell University