Vision-Language System using Open-Source LLMs for Gestures in Medical Interpreter Robots

Thanh-Tung Ngo; Emma Murphy; Robert J. Ross

cs.ROMar 5, 2026

Vision-Language System using Open-Source LLMs for Gestures in Medical Interpreter Robots

Authors: Thanh-Tung Ngo, Emma Murphy, Robert J. Ross

Organizations: Technological University Dublin Dublin, Ireland

Abstract

Effective communication is vital in healthcare, especially across language barriers, where non-verbal cues and gestures are critical. This paper presents a privacy-preserving vision-language framework for medical interpreter robots that detects specific speech acts (consent and instruction) and generates corresponding robotic gestures. Built on locally deployed open-source models, the system utilizes a Large Language Model (LLM) with few-shot prompting for intent detection. We also introduce a novel dataset of clinical conversations annotated for speech acts and paired with gesture clips. Our identification module achieved 0.90 accuracy, 0.93 weighted precision, and a 0.91 weighted F1-Score. Our approach significantly improves computational efficiency and, in user studies, outperforms the speech-gesture generation baseline in human-likeness while maintaining comparable appropriateness.

Figures & tables

Figure 1. Overview of our proposed vision-language system for gesturing medical interpreter robots. A flowchart illustrating the system architecture. A 'User' provides two inputs: speech containing the text 'Hello!', and visual input. The speech input enters a decision diamond labeled 'GSD'. If the GSD output is 'No' (black path), the system proceeds to 'Speech-Gesture Generation' to create a 'Generated Gesture'. If the GSD output is 'Yes' (red path), the system performs 'Pose Estimation' on the user's visual input to create an 'Estimated Gesture'. Both paths converge to drive the final 'Robot Motion' on a Pepper robot.

Figure 2. Pipeline for using an LLM to create the clinical conversation dataset. A data processing pipeline diagram starting with 'Clinical training videos' showing a doctor and patient. The video audio is processed by 'Whisper' to produce 'Conversation transcripts'. These transcripts are input into an 'LLM' using 'Few-shot prompting' to generate classification labels. The final outputs are three categories labeled 'Neither,' 'Consent,' and 'Instruction'.

Figure 3. The on-device gesture sentence detection module. A classification flowchart starting with a 'User sentence' input. The text is processed by an 'Edge LLM' via 'Few-shot prompting classification'. The system outputs one of three labels: 'Neither' results in 'Gesture trigger: No', while both 'Consent' and 'Instruction' result in 'Gesture trigger: Yes'.

Figure 4. Conversation videos to robot motions workflow. A four-stage pipeline diagram illustrating human-to-robot motion transfer. The process begins with an Input Video of two people, flows to Human Pose Estimation, where skeletal keypoints are detected on the subjects, proceeds to Keypoint Mapping, showing a 3D skeletal rig with labeled joints (e.g., Head, Spine, Arms), and concludes with Robot motion displayed on a Pepper humanoid robot.

Model	Accuracy $\uparrow$	Precision $\uparrow$	F1-Score $\uparrow$	Memory (GB) $\downarrow$
qwen3:8b	0.90	0.93	0.91	7.2
deepseek-r1:8b	0.83	0.87	0.84	7.2
llama3.1:8b	0.67	0.83	0.69	6.8
mistral:7b	0.70	0.83	0.72	6.5
gemma3:4b	0.63	0.81	0.69	5.2
llama3.2:3b	0.56	0.83	0.60	4.0

Table 1. Evaluation of LLMs on gesture sentence detection

Figure 5. Confusion matrix illustrating the performance of qwen3:8b . A confusion matrix with the following values. True Consent row: 101 predicted Consent, 0 Instruction, 6 Neither, 10 Unknown. True Instruction row: 28 predicted Consent, 711 Instruction, 135 Neither, 38 Unknown. True Neither row: 26 predicted Consent, 86 Instruction, 2567 Neither, 28 Unknown. The matrix uses a blue color scale where darker blue indicates higher frequencies, peaking at 2567.

Approach	Human-likeness $\uparrow$	Appropriateness $\uparrow$	GPU RAM (MB) $\downarrow$
SG ( Zhang et al., 2024 )	5.24	4.76	2260
Ours	5.78	5.20	3

Table 2. Comparison of human-likeness, appropriateness and memory usage between our approach and SG (mean values).

Figure 6. Boxplots visualizing the rating distributions. Side-by-side boxplots with a y-axis 'Rating Score' ranging from 0 to 10. Appropriateness: 'Ours' (Median 5.50, Q1 4.00, Q3 7.00, Max 8.00, Min 0.00); 'SG' (Median 5.00, Q1 3.00, Q3 6.00, Max 8.00, Min 0.00). Human-likeness: 'Ours' (Median 6.00, Q1 5.00, Q3 7.00, Max 9.00, Min 1.00, with an outlier at 1.00); 'SG' (Median 5.00, Q1 4.00, Q3 6.75, Max 9.00, Min 0.00).

Healthcare Robotics

See all 24 papers

EmoPose: Vision-Language Model Guided Emotion-Aware Gesture Generation for Humanoid Robots
Sep 20, 2026Daojie Peng, Bingtao Wang, Fulong Ma +3Large Language Model-Based Robot Planning Humanoid Robot Control
The Hong Kong University of Science and Technology (Guangzhou) · The Shandong University · RoboScience
GIVE: Grounding Human Gestures in Vision-Language-Action Models
Jun 11, 2026Pengfei Liu, Gen Li, Junqiao Fan +4Language-Conditioned Robot Manipulation Goal-Conditioned Robotic Manipulation
1MARS Lab, Nanyang Technological University
Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech
Apr 13, 2026Edwin C. Montiel-Vazquez, Christian Arzate Cruz, Stefanos Gkikas +3Co-Speech Gesture Generation Transformer
School of Engineering and Sciences Tecnologico de Monterrey State of Mexico, Mexico · Honda Research Institute Japan Wako City, Japan · Department of Electronic Engineering Hellenic Mediterranean University Chania, Greece

Sep 20, 2026J/K move · Enter open · S save

EmoPose: Vision-Language Model Guided Emotion-Aware Gesture Generation for Humanoid Robots

Daojie Peng, Bingtao Wang, Fulong Ma, Wenjun Yue+2

The Hong Kong University of Science and Technology (Guangzhou) · The Shandong University · RoboScience

Large Language Model-Based Robot Planning Humanoid Robot Control Co-Speech Gesture Generation Human-Robot Interaction

Open paper AskPDF arXiv

Socially competent humanoid robots must communicate affect and intent through gesture as well as speech, yet open-ended interaction must become motion that is both expressive and executable on a specific body. This demands semantic flexibility for contextual social intent while preserving deterministic, embodiment-aware robot control. We present EmoPose, a vision-language model (VLM)-guided framework that bridges this gap through an executable semantic interface. Given language, dialogue history, and optional visual context, the VLM selects an ordered gesture plan containing a communicative class, library variant, intensity, and speech anchor. A scalable robot-owned motion library defines the available expressive vocabulary and the source of 14-DoF joint targets. Pose Studio supports automatic trajectory generation, MuJoCo preview, and automatic synchronization of new library entries with the VLM guide; deterministic robot-side modules validate plans, construct trajectories, schedule gestures, and manage queueing and interruption. This division lets the interaction repertoire grow for new social contexts without changing the control interface or delegating raw joint commands to the foundation model. On the EmoPose-Bench, structured GPT-5.5 planning reaches

98.25\pm0.52\%

on the Easy tier and

76.50\pm0.54\%

overall, exceeding same-model direct-label prompting. Further tests validate dialogue-context use and ordered multi-action composition. The system completes the nominal MuJoCo suite and realizes all 29 authored variants on the physical Unitree G1. A four-stop laboratory tour demonstrates expressive narration with interruption, camera-grounded dialogue, and navigation.

Abstract

Healthcare Robotics

Explore similar work

EmoPose: Vision-Language Model Guided Emotion-Aware Gesture Generation for Humanoid Robots

GIVE: Grounding Human Gestures in Vision-Language-Action Models

Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech