Multimodal Socially Aware Framework for Human-Robot Interaction in Virtual Reality

Richard Xu; Dakarai Crowder; Alexis E. Block; Wenzhen Yuan

cs.ROOct 25, 2026

Multimodal Socially Aware Framework for Human-Robot Interaction in Virtual Reality

Authors: Richard Xu, Dakarai Crowder, Alexis E. Block, Wenzhen Yuan

Organizations: University of Illinois at Urbana-Champaign, Champaign, IL, USA · Case Western Reserve University, Cleveland, OH, USA

Abstract

Social robots must coordinate speech and movement while interpreting multimodal human input. We present a multimodal framework that integrates speech, vision, and touch to generate contextually appropriate robot speech and movement. We evaluated the framework in virtual reality with 18 participants across five social interaction scenarios, comparing its performance against a human-teleoperated baseline. Although participants rated the autonomous system above the neutral midpoint for both naturalness and social appropriateness, the teleoperated baseline received significantly higher ratings on both measures. Behavioral coding and qualitative feedback revealed that conversational timing, clarification of ambiguous requests, and spatial coordination were important factors contributing to this performance gap. These findings highlight key challenges in developing autonomous robots capable of natural, multimodal social interaction.

Figures & tables

Fig. 1: We created a multimodal socially aware framework that can interact with humans. The system takes touch, vision, and speech as input and then produces socially appropriate robot behaviors.

Fig. 2: System overview. Speech, touch, and vision inputs are composed into a VLM prompt; the VLM produces action–speech pairs for execution and speech, with an interaction history conditioning subsequent turns.

Fig. 3: Contact sensing between the tracked hand and the simulated Reachy. (a) The hand (translucent) resting on the robot’s shoulder; red markers are the physics engine’s contact points on the robot surface at a single frame. (b) The same hand open, palm toward the viewer, with its colliders at true scale: three per finger ( $r=7$ mm) and a $3\times 3$ palm grid ( $r=10$ mm). (c) The participant was equipped with haptic gloves that indicated when contact occurred with the robot.

Fig. 4: The eight touch gestures, captured in VR. Translucent mesh: the tracked hand at first contact. Markers: all contact points recorded over the gesture episode, on the robot surface, colored by contact strength on a shared scale.

Type	Content (text-only)	Added to history when
Action-Speech Pair	Motion primitive and speech content from VLM output (started/finished)	Motion primitive execution and speech output started/finished
Image Description	Content of image captured by robot eye camera	VLM outputs description for image captured at time of prompt generation
Touch Event	Touch gesture type with confidence, number of touches, and body part touched	Touch on the robot finishes and triggers prompt generation
Speech Event	Transcription of spoken message from human	Human speech finishes and triggers prompt generation

TABLE I: Interaction History Entries

Fig. 5: Scenarios: participants completed five different scenarios. (S1) The participant is prompted to ask Reachy for a joke. (S2) Reachy is cleaning the wall but standing in front of the bathroom (the doorway on the right); the participant is prompted to get Reachy to move. (S3) Reachy is depicted as sad, and the participant is asked to cheer the robot up. (S4) Reachy is vacuuming, and the vacuum is making a loud noise; the participant is asked to get the robot’s attention. (S5) There are drinks on the counter, and the participant is prompted to ask Reachy to pass them a drink.

Fig. 6: Questionnaire and behavioral coding results. Per-scene ratings of (A) the appropriateness and (B) the naturalness of the autonomous system and the teleoperated baseline. (C) Interaction events coded for both the participant and the robot. The first 12 behaviors are considered interaction breakdowns, whereas the final three are considered positive interaction behaviors. (D) Whether a participant touched the robot under a condition for each scenario. There were no instances in which only the teleoperated baseline was touched.

	Mean (SD)
	Autonomous	Teleoperated
Naturalness (N)	0.253 (0.554)	0.620 (0.413)
Appropriateness (A)	0.605 (0.443)	0.812 (0.276)

TABLE II: Rating Means and LME Results

Dimension	Mean	95% CI	p
Naturalness	$+0.25$	$[+0.08,\,+0.42]$	$.004$
Appropriateness	$+0.61$	$[+0.48,\,+0.73]$	$<.001$

TABLE III: Autonomous Ratings vs. Scale Midpoint

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 449 papers

A Multimodal Framework for Human-Multi-Agent Interaction
Mar 24, 2026Shaid Hasan, Breenice Lee, Sujan Sarker +1Human-Robot Interaction Human-AI Interaction
University of Virginia
Validating Virtual Reality for Studying Multimodal Human-Robot Interaction in Socially Aware Robot Navigation
Jul 10, 2026Hariharan Arunachalam, Phani Teja Singamaneni, Rachid AlamiVirtual Reality Social Robot Navigation
LAAS-CNRS, Universite de Toulouse, France · Inria Nancy, France
Immersive Social Interaction with VR and LLM-Assisted Humanoids
Jul 8, 2026Niraj Pudasaini, Geeta Chandra Raju Bethala, Pranav Doma +2Robot Teleoperation Humanoid Robot Control
New York University Abu Dhabi (NYUAD), UAE

Mar 24, 2026J/K move · Enter open · S save

A Multimodal Framework for Human-Multi-Agent Interaction

Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal

University of Virginia

Human-Robot Interaction Human-AI Interaction Multimodal Agents Multi-Agent Coordination

Open paper AskPDF arXiv

Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.

Abstract

Human-Robot Interaction

Explore similar work

A Multimodal Framework for Human-Multi-Agent Interaction

Validating Virtual Reality for Studying Multimodal Human-Robot Interaction in Socially Aware Robot Navigation

Immersive Social Interaction with VR and LLM-Assisted Humanoids