ExpressMM: Expressive Mobile Manipulation Behaviors in Human-Robot Interactions

Souren Pashangpour; Haitong Wang; Matthew Lisondra; Goldie Nejat

cs.ROApr 7, 2026

ExpressMM: Expressive Mobile Manipulation Behaviors in Human-Robot Interactions

Authors: Souren Pashangpour, Haitong Wang, Matthew Lisondra, Goldie Nejat

Organizations: Autonomous Systems and Biomechatronics Laboratory (ASBLab), Department of Mechanical and Engineering, University of Toronto, Toronto, ON M5S 3G8, Canada

Abstract

Mobile manipulators are increasingly deployed in human-centered environments to perform tasks. While completing such tasks, they should also be able to communicate their intent to the people around them using expressive robot behaviors. Prior work on expressive robot behaviors has used preprogrammed or learning-from-demonstration-based expressive motions and large language model generated high-level interactions. The majority of these existing approaches have not considered human-robot interactions (HRI) where users may interrupt, modify, or redirect a robot's actions during task execution. In this paper, we develop the novel ExpressMM framework that integrates a high-level language-guided planner based on a vision-language model for perception and conversational reasoning with a low-level vision-language-action policy to generate expressive robot behaviors during collaborative HRI tasks. Furthermore, ExpressMM supports interruptible interactions to accommodate updated or redirecting instructions by users. We demonstrate ExpressMM on a mobile manipulator assisting a human in a collaborative assembly scenario and conduct audience-based evaluation of live HRI demonstrations. Questionnaire results show that the ExpressMM-enabled expressive behaviors helped observers clearly interpret the robot's actions and intentions while supporting socially appropriate and understandable interactions. Participants also reported that the robot was useful for collaborative tasks and behaved in a predictable and safe manner during the demonstrations, fostering positive perceptions of the robot's usefulness, safety, and predictability during the collaborative tasks.

Explore similar work

Jul 15, 2026

Anatomy of Uncertainty: Expressive Descriptors of Robotic Manipulator Motion for Non-verbal Communication in Human-Robot Collaboration

Robots operating in human-robot collaboration must communicate not only their intended actions but also uncertainty arising from incomplete or ambiguous perception. This work introduces a mathematical framework for expressing perceptual uncertainty through robotic manipulator motion. Drawing on Laban Movement Analysis, robot behavior is organized in a Commitment-Vigilance state space that maps uncertainty-related states - confidence, curiosity, hesitance, fear, and inactivity - to distinct Laban Effort signatures. Five motion primitives - approach, pause, retreat, exploration, and oscillation - are then parameterized using eleven kinematic and geometric descriptors, including acceleration, pause and retreat characteristics, gaze angles, tilt, and shiver amplitude. A video-based human-subject study evaluated recognition of four expressive trajectories and the influence of individual descriptors on perceived intensity. Participants reliably identified the intended behavioral states, while several descriptors significantly modulated expressiveness. The results establish a perceptually grounded basis for encoding robot uncertainty in motion and support future autonomous trajectory generation using parametric movement representations for collaborative tasks in shared environments. Code, videos, questionnaire and appendices are available at "https://bit.ly/github-aou".

Ridhima Bector, Souravik Dutta, Poornima Ramachandran +4

School Of Art, Design and Media, Nanyang Technological University, 81 Nanyang Drive, Singapore 637458 · School of Mechanical and Aerospace Engineering, Nanyang Technological University, 50 Nanyang Ave, Block N3, Nanyang Avenue, 639798

Robot Motion Generation Human-Robot Collaboration Robotic Manipulation Human-Robot Interaction

Ask PDF

Jun 4, 2026

A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI models

This paper presents a distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack. Language understanding, visual grounding, orchestration, and motion execution run as separate ROS 2 nodes, enabling flexible deployment across distributed hardware while maintaining a responsive control loop. From free-form user commands, the system generates structured action requests for pick, place, and handover. It uses a VLM to return image-space targets, which are converted into metric robot-frame goals using depth and calibration. A web dashboard exposes intermediate intent and grounding overlays (pixel, depth, and robot-frame) and requires explicit operator confirmation before any motion is executed. Experiments on a Franka FR3 platform evaluate end-to-end task reliability and latency under increasing working table scene ambiguity and compare alternative LLM/VLM configurations in the same pipeline. Code and full documentation are available at github.com/cogrob-tuni/franka-llm.

Arash Ghasemzadeh Kakroudi, Roel Pieters

Automation Technology and Mechanical Engineering, Tampere University, 33720, Tampere, Finland

Language-Conditioned Robot Manipulation Large Language Model-Based Robot Planning Human-Robot Collaboration VLMs for Robotics

Ask PDF

Jun 7, 2026

Safe, Fluent and Acceptable Motion Generation and Execution for Human--Robot Interaction in Manufacturing Environments

Robots operating in human environments must not only ensure physical safety but also exhibit behaviors that are understandable, fluent, and acceptable to human partners. This paper investigates motion generation strategies that combine safety guarantees with interaction quality considerations, such as motion smoothness and human comfort. While the design of robots capable of ensuring safety in shared human-robot environments has enabled closer and more advanced forms of interaction, these new proximity-based tasks require moving beyond purely technical considerations. In particular, robot behavior must also be addressed from psycho-cognitive and social perspectives. In this context, we argue for the relevance of integrating social-aware motion control into robotic systems. First, we identify the motion parameters that influence human perception and operator experience. Then, we implement a Model Predictive Control (MPC) framework that generates four distinct socially-informed robot behaviors. Finally, we conduct a user study to evaluate and validate these behaviors and assess their social impact on non-expert participants. The results demonstrate that variations in robot behavior significantly affect the perceived social acceptability of the system. These findings highlight the importance of incorporating human-centered considerations into motion generation strategies for robots operating in shared environments.

Thibaut Lopez, Olivier Aycard, Pierre-Brice Wieber +2

GIPSA Lab and Grenoble Institute of Technology, 38000 Grenoble, France · Inria, 38000 Grenoble, France · LIP/PC2S and Univ. Grenoble Alpes, Univ. Savoie Mont Blanc, 38000 Grenoble, France

Robot Motion Generation Safe Motion Planning Human-Robot Interaction

Ask PDF