Development of a Humanoid Robot Prototype for Multimodal Human-Robot Interaction

Thang Tran Viet; Thanh Nguyen Canh; Huy Uong Gia; Phuc Dinh Van; Son Tran Duc; Ngoc Minh Do; Xiem HoangVan

cs.ROSep 4, 2026

Development of a Humanoid Robot Prototype for Multimodal Human-Robot Interaction

Authors: Thang Tran Viet, Thanh Nguyen Canh, Huy Uong Gia, Phuc Dinh Van, Son Tran Duc, Ngoc Minh Do, Xiem HoangVan

Organizations: University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam · School of Information Science, Japan Advanced Institute of Science and Technology

Abstract

Human-robot interaction (HRI) enables intuitive and intelligent collaboration between humans and robots in real-world environments. This paper introduces a humanoid robot prototype designed as a flexible testbed for developing and integrating artificial intelligence (AI) modules in HRI tasks. The system features a 12 degree-of-freedom (DOFs) dual-arm mechanism and a 2 DOFs head with an expressive LCD screen to express facial emotions. All hardware components are controlled by a custom-designed controller board with real-time AI processing supported by an onboard Jetson module. The system incorporates three AI modules: (1) gesture recognition using MediaPipe Pose and an LSTM classifier, (2) object detection with YOLO and 3D localization, and (3) voice-command processing through speech recognition and large language model(LLM)-based semantic parsing. The platform is validated through experiments on positioning accuracy, with results showing average manipulation errors of approximately 1.83 cm. To demonstrate its versatility, experimental results show over 90% task accuracy, with gesture recognition reaching 96%, speech recognition reaching 92%. The results confirm the effectiveness of the proposed system as a reproducible and accessible humanoid platform for research and prototyping in HRI.

Figures & tables

Fig. 1 : Illustration of our hardware design.

Servo	Weight	Voltage Range	Max. Torque
DS5160 SSG	158 g	6.0–8.4 V	65.0 kg·cm
FT5330M	67 g	6.0–7.4 V	35.5 kg·cm
TD-8120MG	56–68 g	4.8–8.4 V	22.8 kg·cm
MG996R	55 g	4.8–7.2 V	11.0 kg·cm

TABLE I : Main specifications of servomotors used in the humanoid robot

Panel of Figure 3 — Fig. 2 : Simulation and real-world PCB implemented on the robot

Device	Parameters	Application
Battery	Input: 220V AC; Output: 36V 10Ah.	Power Supply
Jetson Xavier AGX	CPU: 8-core ARM v8.2; GPU: 512-core Volta Tensor Core; RAM: 32 GB LPDDR4x; Storage: 32 GB eMMC; Power: 10–30W.	High-level controller
Arduino UNO	MCU: ATmega328P; $14$ digital I/O; $6$ analog input; UART/I2C/SPI; $16$ MHz	Low-level controller
Arduino Mega 2560	MCU: ATmega2560; $54$ digital I/O; $16$ analog input; $4$ UART; $16$ MHz	Low-level controller
FSR402 Force Sensor	Force range: $0.2$ – $20$ N; Resistance range: $\sim 1.8$ k $\Omega$ –10M $\Omega$ ; Diameter: $18$ mm	Grasp feedback
ZED 2 Stereo Camera	Dual $4$ MP; FoV: $110^{\circ}$ ; Depth sensing: $0.2$ - $20$ m; Integrated IMU + Barometer sensor; USB $3.0$	Vision

TABLE II : Technical specification of components

Fig. 4 : Overview of the proposed software architecture

Panel of Figure 7 — Fig. 5 : Illustration of human-robot interaction when recognizing love gesture action.

Panel of Figure 8 — Fig. 6 : Robot recognizing and interacting with objects.

Times	Left arm	Right arm	Integrated system
1	1.05	1.07	1.30
2	1.28	0.88	1.24
3	1.29	0.96	1.84
4	1.55	1.36	3.48
5	1.56	1.68	3.82
6	1.25	1.24	1.51

TABLE III : Results of performance evaluation in position error of each arm (cm)

Humanoid Robotics

Humanoid robotics develops human-shaped robots that move, manipulate objects, and interact with people in human environments.

See all 151 papers

An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction
Feb 23, 2026Guanting Shen, Zi TianHuman-Robot Interaction Language-Conditioned Robot Manipulation
The authors are with the Dalian University of Technology. Dalian, China.
Articulated Humanoid Head for a Robot Receptionist Capable of Natural Human Interaction
Jul 19, 2026Tharusha Fonseka, Charuka Bandara, Moshintha Hewavitharana +4Robotics Human-Robot Interaction
Dept. of ECE National University of Singapore Singapore · Dept. of ENTC University of Moratuwa Moratuwa, Sri Lanka
A Multimodal Framework for Human-Multi-Agent Interaction
Mar 24, 2026Shaid Hasan, Breenice Lee, Sujan Sarker +1Human-Robot Interaction Human-AI Interaction
University of Virginia

Feb 23, 2026J/K move · Enter open · S save

An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction

Guanting Shen, Zi Tian

The authors are with the Dalian University of Technology. Dalian, China.

Human-Robot Interaction Language-Conditioned Robot Manipulation Language-Conditioned Robot Control VLMs for Robotics

Open paper AskPDF arXiv

Interpreting human intent accurately is a central challenge in human-robot interaction (HRI) and a key requirement for achieving more natural and intuitive collaboration between humans and machines. This work presents a novel multimodal HRI framework that combines advanced vision-language models, speech processing, and fuzzy logic to enable precise and adaptive control of a Dobot Magician robotic arm. The proposed system integrates Florence-2 for object detection, Llama 3.1 for natural language understanding, and Whisper for speech recognition, providing users with a seamless and intuitive interface for object manipulation through spoken commands. By jointly addressing scene perception and action planning, the approach enhances the reliability of command interpretation and execution. Experimental evaluations conducted on consumer-grade hardware demonstrate a command execution accuracy of 75%, highlighting both the robustness and adaptability of the system. Beyond its current performance, the proposed architecture serves as a flexible and extensible foundation for future HRI research, offering a practical pathway toward more sophisticated and natural human-robot collaboration through tightly coupled speech and vision-language processing.

Abstract

Humanoid Robotics

Explore similar work

An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction

Articulated Humanoid Head for a Robot Receptionist Capable of Natural Human Interaction

A Multimodal Framework for Human-Multi-Agent Interaction