Organizations: University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam · School of Information Science, Japan Advanced Institute of Science and Technology
Human-robot interaction (HRI) enables intuitive and intelligent collaboration between humans and robots in real-world environments. This paper introduces a humanoid robot prototype designed as a flexible testbed for developing and integrating artificial intelligence (AI) modules in HRI tasks. The system features a 12 degree-of-freedom (DOFs) dual-arm mechanism and a 2 DOFs head with an expressive LCD screen to express facial emotions. All hardware components are controlled by a custom-designed controller board with real-time AI processing supported by an onboard Jetson module. The system incorporates three AI modules: (1) gesture recognition using MediaPipe Pose and an LSTM classifier, (2) object detection with YOLO and 3D localization, and (3) voice-command processing through speech recognition and large language model(LLM)-based semantic parsing. The platform is validated through experiments on positioning accuracy, with results showing average manipulation errors of approximately 1.83 cm. To demonstrate its versatility, experimental results show over 90% task accuracy, with gesture recognition reaching 96%, speech recognition reaching 92%. The results confirm the effectiveness of the proposed system as a reproducible and accessible humanoid platform for research and prototyping in HRI.
Figures & tables
Fig. 1 : Illustration of our hardware design.
Servo
Weight
Voltage Range
Max. Torque
DS5160 SSG
158 g
6.0–8.4 V
65.0 kg·cm
FT5330M
67 g
6.0–7.4 V
35.5 kg·cm
TD-8120MG
56–68 g
4.8–8.4 V
22.8 kg·cm
MG996R
55 g
4.8–7.2 V
11.0 kg·cm
TABLE I : Main specifications of servomotors used in the humanoid robot
Fig. 2 : Simulation and real-world PCB implemented on the robot
Fig. 3 : The components in our robot
Device
Parameters
Application
Battery
Input: 220V AC; Output: 36V 10Ah.
Power Supply
Jetson Xavier AGX
CPU: 8-core ARM v8.2; GPU: 512-core Volta Tensor Core; RAM: 32 GB LPDDR4x; Storage: 32 GB eMMC; Power: 10–30W.
High-level controller
Arduino UNO
MCU: ATmega328P; 14 digital I/O; 6 analog input; UART/I2C/SPI; 16 MHz
Low-level controller
Arduino Mega 2560
MCU: ATmega2560; 54 digital I/O; 16 analog input; 4 UART; 16 MHz
Low-level controller
FSR402 Force Sensor
Force range: 0.2 – 20 N; Resistance range: ∼1.8 k Ω –10M Ω ; Diameter: 18 mm
Grasp feedback
ZED 2 Stereo Camera
Dual 4 MP; FoV: 110∘ ; Depth sensing: 0.2 - 20 m; Integrated IMU + Barometer sensor; USB 3.0
Vision
TABLE II : Technical specification of components
Fig. 4 : Overview of the proposed software architecture
Fig. 5 : Illustration of human-robot interaction when recognizing love gesture action.
Fig. 6 : Robot recognizing and interacting with objects.
Times
Left arm
Right arm
Integrated system
1
1.05
1.07
1.30
2
1.28
0.88
1.24
3
1.29
0.96
1.84
4
1.55
1.36
3.48
5
1.56
1.68
3.82
6
1.25
1.24
1.51
TABLE III : Results of performance evaluation in position error of each arm (cm)
Interpreting human intent accurately is a central challenge in human-robot interaction (HRI) and a key requirement for achieving more natural and intuitive collaboration between humans and machines. This work presents a novel multimodal HRI framework that combines advanced vision-language models, speech processing, and fuzzy logic to enable precise and adaptive control of a Dobot Magician robotic arm. The proposed system integrates Florence-2 for object detection, Llama 3.1 for natural language understanding, and Whisper for speech recognition, providing users with a seamless and intuitive interface for object manipulation through spoken commands. By jointly addressing scene perception and action planning, the approach enhances the reliability of command interpretation and execution. Experimental evaluations conducted on consumer-grade hardware demonstrate a command execution accuracy of 75%, highlighting both the robustness and adaptability of the system. Beyond its current performance, the proposed architecture serves as a flexible and extensible foundation for future HRI research, offering a practical pathway toward more sophisticated and natural human-robot collaboration through tightly coupled speech and vision-language processing.
Guanting Shen, Zi Tian
The authors are with the Dalian University of Technology. Dalian, China.
Humanoid robots have become increasingly popular in applications such as social interaction, education, and service roles, which drives the need for more natural and efficient human-robot interactions. However, currently available humanoid heads often face limitations, including high costs, mechanical complexity, and limited adaptability across diverse environments. To address these challenges, we present an articulated humanoid robot head designed for a receptionist role, integrating a mechanical structure with 21 degrees of freedom (DoF), including mechanisms for the mouth, eyes, eyebrows, and neck, and covered with realistic silicone skin to achieve a human-like appearance and expression. The system integrates a model-based architecture that combines SCRFD, ArcFace, and ByTetrack for face recognition and Llama and Whisper for natural language processing, with hardware support enabling real-time operations and human re-identification. The conversational ability and re-identification capabilities of the humanoid robot head were quantitatively measured, while its emotional expressiveness and human likeness were evaluated through a user study, achieving an average human likeness score of 4.13 out of 5.
Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.