Bridging the Awareness Gap: Socially Mediated State Externalization for Transparent Distributed Home Robots

Wenzheng Zhao; Manideep Duggi; Fengpei Yuan

cs.ROMar 12, 2026

Bridging the Awareness Gap: Socially Mediated State Externalization for Transparent Distributed Home Robots

Authors: Wenzheng Zhao, Manideep Duggi, Fengpei Yuan

Organizations: Faculty of Robotics Engineering Department, Worcester Polytechnic Institute, 100 Institute Rd, Worcester, MA, USA

Abstract

Distributed multi-robot systems for the home often require robots to operate out of the user's sight, creating a state awareness gap that can diminish trust and perceived transparency and control. This paper investigates whether real-time, socially mediated state externalization can bridge this gap without compromising task performance. We developed a system where a co-located social mediator robot (Pepper) externalizes the hidden execution states of an out-of-sight mobile manipulator (Stretch~3) for voice-driven object retrieval and delivery, where task-level states are synchronized and externalized through verbal updates and visual progress display. In a counterbalanced within-subject study (N=30), we compared a baseline of Autonomous Hidden Execution against Socially Mediated State Externalization. Our results show that externalization significantly increases user task-focused attention (from 15.8% to 84.6%, p<.001) and substantially improves perceived perspicuity, dependability, stimulation, and attractiveness (all p<.001). Furthermore, 83% of participants preferred the externalized condition, and this improvement in user experience was achieved without a statistically significant increase in end-to-end task completion time (p=.271). The results suggest that socially mediated state externalization is an effective architectural mechanism for designing more transparent and trustworthy distributed robot systems, ultimately enhancing user experience without sacrificing performance in distributed home robot deployments.

Figures & tables

Fig. 1: Overview of the distributed execution architecture and state externalization framework. The user interacts with a co-located social mediator, while task execution is performed by a mobile manipulation robot operating in a physically separated workspace. A coordination server synchronizes task-level execution states (e.g., NAVIGATING, GRASPING, RECOVERING) and externalizes them to the user through verbal and visual feedback. This architecture enables real-time state awareness despite spatial separation.

Fig. 2: System architecture of the distributed robot interaction framework. User voice commands are captured by the Pepper social mediator, which forwards structured intents to a coordination server. The server dispatches task commands to the Stretch mobile manipulator for navigation and object retrieval. During execution, the robot publishes task-level state updates (e.g., NAVIGATING, SEARCHING, GRASPING), which are externalized to the user through Pepper via verbal and visual feedback. When a failure is detected, a closed-loop recovery process is triggered: the failure event is externalized to the user, user confirmation is collected, and a recovery command is issued to the execution robot.

State	Description
IDLE	The robot is awaiting task assignment.
NAVIGATING	The robot is moving toward a target location.
SEARCHING	The robot is locating the requested object.
GRASPING	The robot is attempting object manipulation.
FAILED	A navigation or manipulation failure has been detected.
RECOVERING	The robot is executing a recovery strategy.

TABLE I: Task-level execution state definitions.

Fig. 3: Experimental layout of the distributed task execution setup under two conditions. (a) Condition A : Autonomous Hidden Execution: the execution robot operates in a physically separated workspace without communicating intermediate task states to the user. (b) Condition B : Socially Mediated State Externalization: a co-located social mediator provides real-time verbal and visual updates about the execution robot’s task states. In both conditions, the execution robot performs object retrieval in a separate room, while the user remains spatially separated from task execution. The purple arrow indicates the execution robot’s navigation path during retrieval and delivery tasks.

Fig. 4: Example end-to-end trial timeline for Condition B (Socially Mediated State Externalization). (a) The participant issues a request via the social mediator (Pepper) and confirms the task. (b) While the execution robot operates in a separate workspace, the mediator externalizes task progress through verbal updates and a visual progress display. (c) The execution robot retrieves the target object and returns. (d) The item is delivered to the participant, and the mediator concludes the interaction.

Metric	Description
Task Initiation Time	Duration from system readiness for speech input to dispatch of a structured task command.
Execution Time	Time from task dispatch to completion (or failure), including navigation and manipulation phases.
End-to-End Task Time	Total duration from user request initiation to final task completion or termination.
Number of Grasp Attempts	Total grasp trials executed during a task session.
Task Success	Binary indicator of successful object retrieval and delivery.
Task-Focused Attention Ratio	Proportion of execution time during which participants oriented toward task-relevant areas, coded from video.

TABLE II: Objective Performance Metrics

Dimension	Description
Perspicuity	Understanding of system behavior and awareness of ongoing task states.
Dependability	Perceived reliability and sense of control during execution.
Stimulation	Engagement and feeling accompanied during task progress.
Attractiveness	Overall enjoyment of the interaction.

TABLE III: Subjective HRI Questionnaire Dimensions

Fig. 5: Objective performance and interaction metrics under Hidden Execution and Socially Mediated State Externalization. (a) Time-based metrics, including task initiation time, execution time, and total end-to-end duration. (b) Average number of grasp attempts per task session. (c) Task-focused attention ratio during execution. Error bars represent $\pm$ 1 SEM. Asterisks denote statistical significance ( ${}^{*}p<0.05$ , ${}^{***}p<0.001$ ). .

Metric	A Mean (SD)	B Mean (SD)	t(29)	p
Init. (s)	33.47 (7.84)	49.93 (9.12)	9.21	$<0.001^{***}$
Exec. (s)	162.63 (64.54)	137.33 (35.87)	-2.56	$0.016^{*}$
Total (s)	196.10 (67.78)	187.26 (38.40)	-1.12	$0.271$
Grasp Att.	2.20 (1.21)	1.78 (0.93)	-1.54	$0.134$
Attention (%)	15.8 (8.3)	84.6 (6.5)	21.47	$<0.001^{***}$
Suc. Rate (%)	86.7	90.0	–	$0.754$

TABLE IV: Objective performance under Autonomous Hidden Execution (A) and Socially Mediated State Externalization (B).

Fig. 6: Object-level task distribution and manipulation success rates. (a) Percentage of object requests across categories (Water, Chips, Fruit). (b) Object-wise task success rates aggregated across conditions. Although success rates varied across object types, object requests were moderately distributed and no condition-specific interaction was observed, indicating that object composition did not confound the transparency effect.

Metric	A Mean (SD)	B Mean (SD)	t(29)	p
Perspicuity	3.12 (0.74)	4.31 (0.62)	6.84	$<0.001^{***}$
Dependability	3.36 (0.69)	4.18 (0.58)	4.97	$<0.001^{***}$
Stimulation	3.08 (0.77)	4.45 (0.51)	8.12	$<0.001^{***}$
Attractiveness	3.24 (0.72)	4.36 (0.55)	6.21	$<0.001^{***}$

TABLE V: Subjective ratings for Autonomous Hidden Execution (A) vs Socially Mediated State Externalization (B).

Outcome	Effect	$\beta$	Test stat.	$p$
Total task time	Condition	7.98	$z=0.70$	0.484
Total task time	Period	-2.85	–	0.803
Grasp attempts	Condition	0.44	$z=1.89$	0.059
Grasp attempts	Period	-0.26	–	0.274

TABLE VI: Linear mixed-effects analysis of order effects. Fixed effects include Condition (Hidden vs. External) and Period (first vs. second exposure); participant is a random intercept.

Fig. 7: Failure mode breakdown for failed trials under both experimental conditions. Failure events were categorized into navigation errors, grasp failures, system hangs, Pepper-related errors, and other/unknown causes. Although the total number of failed trials differed slightly, the relative composition of failure types remained comparable across conditions, indicating no systematic shift in underlying failure mechanisms.

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 449 papers

What Is My Robot Thinking? Design Considerations for Transparent and Trustworthy Shared Autonomy
Jun 5, 2026Atharv Belsare, Zohre Karimi, Connor Mattson +2Trust in HRI Shared Autonomy
Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA. · Robotics Center, University of Utah, Salt Lake City, UT, USA.
Seeing Less Is Not Seeing Safely: Privacy Leakage from Task-Scoped Robot Perception Exports
Sep 2, 2026Yuqiao Xu, Erman AydayPrivacy-Preserving ML Robotic Perception
Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH, USA
OmniRobotHome: A Multi-Camera Home Platform for Real-Time Human-Robot Interaction
Apr 30, 2026Junyoung Lee, Inhee Lee, Kyungwon Cho +9Human Motion Prediction Human-Robot Interaction
Seoul National University · RLWRLD

Jun 5, 2026J/K move · Enter open · S save

What Is My Robot Thinking? Design Considerations for Transparent and Trustworthy Shared Autonomy

Atharv Belsare, Zohre Karimi, Connor Mattson, Rushiil Nakka+1

Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA. · Robotics Center, University of Utah, Salt Lake City, UT, USA.

Trust in HRI Shared Autonomy Assistive Robotics Human-Robot Interaction

Open paper AskPDF arXiv

Assistive robots operating under shared autonomy must balance user control with autonomous assistance. Because robot actions depend on internal intent inference that is not directly observable, mismatches between inferred and intended goals can undermine coordination and trust. We investigate how interface-level transparency, including feedback modality (visual vs. auditory) and information richness (sparse vs. rich), shapes interaction in a vision-based shared autonomy system. In a user study with N=25 participants across two assistive manipulation tasks, we evaluate how these designs influence coordination and trust. Providing feedback significantly improves intent alignment and reduces corrective intervention, indicating that making the inferred goal legible accelerates convergence in shared control. Participants preferred visual over auditory feedback, while preferences for sparse versus rich information depended on task complexity. We also found that revealing the full belief distribution did not consistently improve alignment or trust. Together, these findings indicate that effective transparency enhances coordination primarily through goal legibility, while trust depends on task-appropriate information exposure rather than maximal disclosure. Based on these results, we outline guidelines for designing transparent shared autonomy systems.

Abstract

Human-Robot Interaction

Explore similar work

What Is My Robot Thinking? Design Considerations for Transparent and Trustworthy Shared Autonomy

Seeing Less Is Not Seeing Safely: Privacy Leakage from Task-Scoped Robot Perception Exports

OmniRobotHome: A Multi-Camera Home Platform for Real-Time Human-Robot Interaction