RFM-HRI : A Multimodal Dataset of Medical Robot Failure, User Reaction and Recovery Preferences for Item Retrieval Tasks

Yashika Batra; Giuliano Pioldi; Promise Ekpo; Arman Sayatqyzy; Purnjay Maruur; Shalom Otieno; Kevin Ching; Angelique Taylor

cs.ROMar 5, 2026

RFM-HRI : A Multimodal Dataset of Medical Robot Failure, User Reaction and Recovery Preferences for Item Retrieval Tasks

Authors: Yashika Batra, Giuliano Pioldi, Promise Ekpo, Arman Sayatqyzy, Purnjay Maruur, Shalom Otieno, Kevin Ching, Angelique Taylor

Organizations: Cornell University, Cornell Tech, United States of America · Weill Cornell Medicine, United States of America

Abstract

While robots deployed in real-world environments inevitably experience interaction failures, understanding how users respond through verbal and non-verbal behaviors remains under-explored in human-robot interaction (HRI). This gap is particularly significant in healthcare-inspired settings, where interaction failures can directly affect task performance and user trust. We present the Robot Failures in Medical HRI (RFM-HRI) Dataset, a multimodal dataset capturing dyadic interactions between humans and robots embodied in crash carts, where communication failures are systematically induced during item retrieval tasks. Through Wizard-of-Oz studies with 41 participants across laboratory and hospital settings, we recorded responses to four failure types (speech, timing, comprehension, and search) derived from three years of crash-cart robot interaction data. The dataset contains 214 interaction samples including facial action units, head pose, speech transcripts, and post-interaction self-reports. Our analysis shows that failures significantly degrade affective valence and reduce perceived control compared to successful interactions. Failures are strongly associated with confusion, annoyance, and frustration, while successful interactions are characterized by surprise, relief, and confidence in task completion. Emotional responses also evolve across repeated failures, with confusion decreasing and frustration increasing over time. This work contributes (1) a publicly available multimodal dataset (RFM-HRI), (2) analysis of user responses to different failure types and preferred recovery strategies, and (3) a crash-cart retrieval scenario enabling systematic comparison of recovery strategies with implications for safety-critical failure recovery. Our findings provide a foundation for failure detection and recovery methods in embodied HRI.

Figures & tables

Figure 1. Data collection environment at university campus laboratory (left) and at the hospital (right) Participants faces are blacked out in compliance with our approved IRB protocol.

Dataset	Annotations	Dataset Modalities	Healthcare Domain?	Experts and Non-Experts?	Human Robot Interaction?	Controlled Failure Injection?	Reactions to Failures?
Oops! ( Epstein et al., 2020 )	Videos Annotated for Action Intention Variability	Video	$\times$	$\times$	$\times$	$\times$	$\checkmark$
RAF-DB ( Li et al., 2017 )	Compound Expression Images and Labels in the Wild	Image	$\times$	$\times$	$\times$	$\times$	$\times$
CK+ ( Lucey et al., 2010 )	Facial Action Coding System, Action Units, and Labels for Posed and Unposed Expressions	Video, Image, FACS/AUs	$\times$	$\times$	$\times$	$\times$	$\times$
Oulu-CASIA ( Zhao et al., 2011 )	Facial Expression Dataset, Across Illumination Conditions	Image (Infrared and Visible Light)	$\times$	$\times$	$\times$	$\times$	$\times$
AFEW 7.0 ( Dhall et al., 2012 )	Emotion-Labeled Movie Clips	Video, Audio	$\times$	$\times$	$\times$	$\times$	$\times$
BP4D-Spontaneous ( Zhang et al., 2013 )	2D and 3D Dataset Containing Facial Action Units, Video Features, and Head Poses	Video (2D and 3D), FACS/AUs, Pose	$\times$	$\times$	$\times$	$\times$	$\times$

Table 1. Comparison of representative datasets spanning facial expression analysis, conversational errors, and human–robot interaction. In contrast to prior work, RFM-HRI (Ours) provides a healthcare-centered HRI dataset featuring a novel crash cart robot, controlled failure injection, and rich multimodal annotations of human reactions, including speech, FACS and AUs, head pose, gaze estimation, and post-hoc survey responses—collected from both expert and non-expert participants.

Figure 2. Robotic crash cart platforms in real-world hospital deployment (left) and controlled university campus laboratory evaluation (right).

Drawer #	Hospital Data Collection Supplies	In-Lab Data Collection Supplies
Drawer 1	Phenylephrine HCI, Epinephrine Injection, Metoprolol, Naloxone, Nitroglycerin, Norepinephrine, Adenosine	Blue tape, white tape, SD cards, LEDs
Drawer 2	Sodium Bicarbonate Injection USP, Lidocaine, 50% Dextrose Injection, 10% Calcium Chloride Injection	Orange pencils, yellow pencils, batteries, measuring tape
Drawer 3	Razor, Medical Tape, Surgical Gloves, Line Draw Arterial Blood Sample Syringe	Red pencils, wires, blue markers, blue pencils
Drawer 4	Blanket, Scissors, Face Masks	White board cleaner, circuit board, calculator, gloves, playstation controller, mouse
Drawer 5	1 mL Tuberculin Syringe, 6 mL Syringe, 500 mL IV Bag	Scissors, pliers, book, wipes, orange multimeter, hand sanitizer
Drawer 6	Drawer 6 is Empty in the Hospital Cart	No Drawer 6 on In-Lab Cart

Table 2. Drawer-by-drawer inventory comparison of crash cart contents in the lab prototype vs hospital reference cart, supporting multimodal data collection and failure-induction studies.

Panel of Figure 3 — Figure 3. The Wizard-of-Oz (WoZ) Interface components (Part 1 of 4).

Panel of Figure 4 — Figure 4. The Wizard-of-Oz (WoZ) Interface components (Part 2 of 4).

Figure 5. Post-study survey form used to collect affective responses, including categorical emotion selection and Self-Assessment Manikin (SAM) ratings, as well as participants’ preferred recovery strategies following each trial.

Panel of Figure 6 — Figure 6. Dataset organization and per-trial multimodal composition. The cleaned dataset is organized into five trial outcome categories (four injected robot failure types and success). For each trial, we release a self-contained sample comprising time-indexed visual behavioral features (MediaPipe-derived facial action units, gaze, pose, and landmarks), structured post-trial survey responses, and a natural-language task transcript. An overview of the dataset structure and accessible modalities is illustrated in Fig. 6 . Together, these views summarize both the dataset-level structure and the multimodal signals available for modeling trial outcomes and recovery preferences, while abstracting away file-level organization.

Condition	Participants	# Samples	All emotions (count)	Recovery preferences (count)
Comprehension	38 Total (11 Expert, 27 Non-Expert)	42 (13 Expert, 29 Non-Expert)	Annoyance (11) Confusion (11) Frustration (10) Curiosity (4) Surprise (3) Confidence (2) Relief (1)	Verbal (111) Nonverbal (1) Multimodal (17) No recovery needed (0)
Search	39 Total (12 Expert, 27 Non-Expert)	43 (15 Expert, 28 Non-Expert)	Confusion (16) Annoyance (14) Frustration (5) Stress / pressure (3) Confidence (2) Curiosity (2) None / neutral (1)	Verbal (110) Nonverbal (2) Multimodal (17) No recovery needed (0)
Speech	40 Total (11 Expert, 29 Non-Expert)	48 (15 Expert, 33 Non-Expert)	Confusion (17) Annoyance (9) Frustration (6) Confidence (5) Surprise (4) Relief (2) Stress / pressure (2) Amused (1) Curiosity (1) None / neutral (1)	Verbal (104) Nonverbal (5) Multimodal (36) No recovery needed (1)
Success	37 Total (10 Expert, 27 Non-Expert)	41 (13 Expert, 28 Non-Expert)	Surprise (12) Relief (10) Confidence (6) Confusion (4) Annoyance (3) Frustration (2) Curiosity (1) Happy (1) In doubt (1) None / neutral (1)	Verbal (23) Nonverbal (0) Multimodal (7) No recovery needed (2)
Timing	36 Total (10 Expert, 26 Non-Expert)	40 (12 Expert, 28 Non-Expert)	Frustration (12) Confusion (11) Annoyance (10) Curiosity (2) Surprise (2) Interest (1) Confidence (1) Relief (1)	Verbal (113) Nonverbal (2) Multimodal (13) No recovery needed (0)

Table 3. Dataset overview across entire dataset (214 samples)

Emotion	Count	Percentage (%)
Confusion	49	25.9
Annoyance	42	22.2
Frustration	33	17.5
Surprise	19	10.1
Confidence	16	8.5
Relief	12	6.3

Table 4. Overall distribution of primary emotions across analyzed samples (

N=189

Figure 7. Primary emotion distributions by outcome (Success vs. Failure)

Figure 8. Permutation distributions for emotion – outcome (left) and emotion – condition (right) associations.

Figure 9. Primary emotion distributions across interaction conditions

Figure 10. Overall and temporal distributions of primary emotions

Dimension	Median	IQR	Mean	SD
Valence	4.0	[2.0–4.0]	3.28	1.38
Arousal	3.0	[2.0–4.0]	2.92	1.18
Control	3.0	[2.0–4.0]	2.83	1.33

Table 5. Overall affective ratings across all trials (

N=189

Dimension	W	p-value	Rank-biserial $r$	Median (Succ vs Fail)
Valence	1.0	$1.73\times 10^{-6}$	0.996	1.0 vs 3.8
Arousal	81.0	0.0158	0.538	2.0 vs 3.0
Control	88.0	0.00165	$-0.645$	4.0 vs 2.5

Table 6. Paired Wilcoxon comparisons of affective ratings between Success and Failure outcomes (participant-level summaries,

n=34

Dimension	$\chi^{2}$	df	p-value	Kendall’s $W$
Valence	49.27	4	$5.13\times 10^{-10}$	0.440
Arousal	11.20	4	0.0244	0.100
Control	13.89	4	0.00765	0.124

Table 7. Friedman tests for affective differences across five interaction conditions (

n=28

participants with complete data).

		Mdn	IQR	Mdn	IQR	Mdn	IQR
Condition	$n$	Valence		Arousal		Control
Comprehension Failure	37	4.0	[3.0–4.0]	3.0	[2.0–4.0]	2.0	[2.0–4.0]
Search Failure	39	4.0	[3.0–5.0]	3.0	[2.0–4.0]	2.0	[1.0–3.5]
Speech Failure	41	4.0	[3.0–4.0]	3.0	[2.0–4.0]	3.0	[2.0–4.0]
Timing Failure	36	4.0	[3.0–4.0]	3.0	[2.0–4.0]	3.0	[1.75–3.0]
Success	36	1.0	[1.0–2.0]	2.0	[1.0–4.0]	4.0	[3.0–5.0]

Table 8. Median and interquartile range (IQR) of affective ratings by interaction condition.

Figure 11. Affective ratings distribution across interaction conditions

Figure 12. Affective ratings across trial order

Modality	Mentions	Percent (%)
Verbal	431	63.9
Nonverbal	219	32.5
Multimodal	24	3.6
None needed	0	0.0
Total	674	100.0

Table 9. Recovery modality distribution (eligible subset;

N=153

failure trials). Percentages are computed over all recovery mentions.

Verbal strategy (normalized label)	Mentions
Clarifying prompt (e.g., request repetition/clarification)	74
Self-correction & acknowledgment	59
Specific re-direction (explicit drawer/item instruction)	45
Transparency cue (state/explanation, e.g., recalibrating)	42
Confidence check (explicit confirmation)	42
Guided reset (restart/close drawers and retry)	41

Table 10. Top verbal recovery strategies by number of mentions (eligible subset;

N=153

failure trials).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 23 — Figure A.1 . The Wizard-of-Oz (WoZ) Interface components (Part 3 of 4).

Panel of Figure 24 — Figure A.2 . The Wizard-of-Oz (WoZ) Interface components (Part 4 of 4).

Explore similar work

REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery
Jun 29, 2026Giuliano Pioldi, Yashika Batra, Arman Ibrayeva +4Robot Failure Detection Robot Failure Recovery
Cornell University Cornell Tech New York, NY
Toward Human-in-the-Loop Robot Failure Recovery: Bridging Communication Gaps in Human-Robot Collaboration
Sep 21, 2026Promise Ekpo, Teju Vijay, Dhruv Mandalik +5Human-in-the-Loop Evaluation Robot Failure Recovery
Cornell University, New York, New York, USA · Cornell Tech, New York, New York, USA · Brown University, Providence, Rhode Island, USA
Impact of Different Failures on a Robot's Perceived Reliability
Mar 9, 2026Andrew Violette, Zhanxin Wu, Haruki Nishimura +5Robot Failure Recovery Human-Robot Interaction
Department of Computer Science, Cornell University · Toyota Research Institute (TRI). · Sibley School of Mechanical and Aerospace Engineering, Cornell University

Jun 29, 2026J/K move · Enter open · S save

REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery

Giuliano Pioldi, Yashika Batra, Arman Ibrayeva, Yuanchen Bai+3

Cornell University Cornell Tech New York, NY

Robot Failure Detection Robot Failure Recovery Human-Robot Interaction Anomaly Detection

Open paper AskPDF arXiv

Understanding how users perceive and respond to robot failures is essential for building robust and trustworthy robot systems. Prior work, however, (i) often treats failures as independent events, (ii) emphasizes binary failure detection, (iii) with rule-based recovery modeling. We present REPAIR-Bench, built on 214 interaction trials from 41 participants, the benchmark spans four induced failure types and provides synchronized facial action units, head pose, speech transcripts, and post-interaction affect and recovery reports. The benchmark spans three novel evaluation tasks that jointly capture the lifecycle of failure in human-robot interaction (HRI): (i) failure detection over inter-dependent interaction sessions, modeling longitudinal user adaptation across repeated failures; (ii) visual failure-type classification beyond binary success/failure formulations; and (iii) user-centered recovery prediction, inferring users' preferred recovery strategies from interaction context rather than relying on manually designed or rule-based strategies. In baseline experiments, hierarchical recurrent modeling improved failure detection over a single-session model (strict F1: 0.80 vs. 0.68), achieved a failure localization mean signed error of -0.51 s, median absolute error of 2.97 s and, for recovery prediction, a QLoRA-tuned Mistral-7B reached Hit@5=0.76 and F1@5=0.32. REPAIR-Bench provides both the HRI and Medical HRI communities with a standardized framework for (1) evaluating robot failures and (2) building transparent, adaptive, and trustworthy recovery systems.

Abstract

Explore similar work

REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery

Toward Human-in-the-Loop Robot Failure Recovery: Bridging Communication Gaps in Human-Robot Collaboration

Impact of Different Failures on a Robot's Perceived Reliability