Robots interacting with people must recognize not only explicit commands, but also social cues such as invitations, refusals, and unavailability. In real deployments, these cues must be inferred from noisy onboard perception under partial occlusion, changing viewpoints, and strict latency constraints. We present SocioGesture, a real-time adaptive social gesture perception system for human-robot interaction (HRI). SocioGesture uses a compact confidence-aware body-hand skeleton representation and a lightweight dual-stream model that fuses body motion with hand articulation for low-latency onboard recognition. To improve deployment robustness, we train the model with occlusion-aware skeleton corruption, exposing it to missing hands, occluded arms, and temporally unstable keypoints without increasing the inference cost. On a social gesture dataset collected in mixed indoor-outdoor HRI scenarios, SocioGesture achieves strong held-out-subject recognition, substantially improves robustness under structured joint occlusion, and runs in real time on a robot-mounted edge device. During deployment, uncertain interaction segments are saved for offline labeling and adaptation, enabling SocioGesture to expand its gesture vocabulary while preserving performance in the original classes. These results demonstrate a practical path toward robust, efficient, and adaptive social perception for interactive robots.
Figures & tables
Figure 1: SocioGesture enables real-time and adaptive social gesture perception for human-robot interaction. Confident predictions are mapped to conservative robot behaviors, while uncertain interaction segments are saved for offline labeling and model adaptation.
Figure 2: System overview. The online path converts RGB video into confidence-aware skeleton windows and predicts gestures with a dual-stream body-hand model. Confident predictions are mapped to conservative robot behaviors, while uncertain interaction segments are saved with paired RGB video and skeleton data. The offline path labels saved videos, consolidates labels, fine-tunes the skeleton model, and redeploys the updated recognizer.
Training
Clean
Hand
Arm
Temporal-Hand
Standard
96.6±2.8
31.8±0.0
28.3±0.2
90.5±0.1
Occlusion Aug.
95.7±3.2
84.9±0.1
79.5±0.3
92.6±0.1
Occlusion Consistency
95.9±3.2
84.6±0.1
80.2±0.3
92.7±0.1
Table 1: Clean recognition and structured occlusion evaluation on the HRI dataset. The Clean column reports LOSO accuracy as mean ± standard deviation across 10 held-out-subject folds. Occlusion columns report mean ± standard deviation over three evaluation passes with independent occlusion realizations while keeping the trained models and test set fixed. Hand masks both hands, Arm masks one arm and its connected hand, and Temporal-Hand masks hand joints over a contiguous temporal span. All values are accuracies in percent.
Module
Latency
Deployment
YOLO11n person detection
5 ms
TensorRT FP16
BoT-SORT tracking
1 ms
CPU/GPU tracking
RTMW-l pose estimation
8 ms
TensorRT FP16
Skeleton gesture model
11 ms
PyTorch
Total, 1 person
25 ms
real-time @ 30 FPS
Total, 3 persons
42 ms
real-time @ 24 FPS
Table 2: Real-time deployment latency on the robot-mounted NVIDIA Jetson Thor. YOLO11n and RTMW-l are converted to TensorRT FP16 engines. The deployed pipeline operates at the 30 FPS camera rate; reported latencies reflect per-frame compute time.
Metric
Base
Fine-tuned
Δ
Overall 10-class clip accuracy
64.6
87.9
+23.3
Clip macro F1
56.7
88.5
+31.8
Original 7-class clip accuracy
98.5
98.5
+0.0
New 3-class clip accuracy
–
67.6
–
Table 3: Offline deployment adaptation on a held-out 5-subject 10-class test set. The base model recognizes only the original 7 classes and is evaluated in the expanded 10-class label space; new-class accuracy is therefore unsupported before adaptation. Fine-tuning expands the classifier using pseudo-labeled deployment skeletons.
Figure 3: Robot-view recognition examples. The examples include original social gestures, newly added gestures recognized after offline adaptation, and multi-person deployment scenes. Faces are blurred for privacy; the deployed system operates on the original stream.
Method
Passes
Params (M)
GFLOPs
X-Sub Acc.
CTR-GCN [ 3 ]
4
5.8
7.9
92.4%
InfoGCN [ 4 ]
6
9.4
10.0
93.0%
BlockGCN [ 5 ]
4
5.2
6.5
93.1%
PoseConv3D [ 6 ]
1
4.0
31.8
94.1%
BHaRNet-P [ 7 ]
2
9.06
3.42
96.3%
SocioGesture (ours)
1
1.3
2.3
94.6%
Table 4: Computational efficiency on NTU RGB+D 60 under the cross-subject protocol. Baseline values are reported from the corresponding papers. “Passes” denotes the number of inference passes or ensemble streams used for prediction. SocioGesture has body and hand branches internally, but is evaluated as a single forward pass for low-latency robot deployment.
Gesture
Robot behavior
Recognition
Behavior success
come
Approach
15/15
13/15
wave
Wave back
15/15
14/15
stop
Halt
15/15
14/15
phone call
Wait
13/15
11/15
occupied
Wait
15/15
15/15
idle (facing)
Approach (engage)
14/15
13/15
Table 5: Live closed-loop deployment on the Unitree G1 with five unseen participants, 150 trials, fixed confidence threshold 0.7 . Recognition reports correctly recognized gestures regardless of confidence; behavior success reports trials in which the robot executed the correct behavior.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Input
Backbone
Training
Clean
Structured occlusion
Random stress test
Avg. Occl.
Hand
Arm
Temp-Hand
R20
R40
R60
8ch
ST-GCN++
Standard
94.3
79.5
49.3
91.0
19.6
4.5
2.9
41.1
8ch
ST-GCN++
Occlusion Aug.
94.3
88.1
82.6
92.3
90.5
79.8
49.1
80.4
8ch
ST-GCN++
Occlusion Consistency
94.4
88.2
83.5
92.7
91.6
83.3
55.5
82.5
9ch
ST-GCN++
Standard
94.4
81.8
27.9
91.5
12.3
3.4
2.5
36.6
9ch
ST-GCN++
Occlusion Aug.
94.5
83.3
81.9
92.1
91.3
80.4
48.2
79.5
Appendix
Table 6: Occlusion robustness ablation on NTU RGB+D 60 using RTMW-extracted 2D 59-joint skeletons under the cross-subject protocol. The 9ch input uses the full confidence-aware representation in Eq. ( 1 ); the 8ch input removes the confidence channel. Structured occlusions approximate common robot-mounted pose failures, while random joint masking is included as a controlled stress test. Avg. Occl. is the mean of the six occlusion columns.
Setting
Value
Notes
Architecture
d (feature width)
256
Per-stream feature dimension
Cross-attention heads
4
Multi-head temporal attention
JB/JH (HRI)
11 / 42
Upper-body / hand joints
JB/JH (NTU)
17 / 42
Full-body / hand joints
Face feature dim.
3
Yaw, pitch, confidence
Appendix
Table 7: Hyperparameters used for SocioGesture training, deployment, and adaptation. Symbols match the Method section.
Enabling humanoid robots to respond to human speech with synchronized and semantically meaningful gestures is fundamental to natural human-robot interaction. However, this task faces three critical barriers: the scarcity of semantically rich datasets, the "modality eclipse" where models ignore audio cues in favor of kinematic inertia, and the sim-to-real gap regarding physical safety. We propose RoboGesture, a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system in which the robot listens, responds, and gestures in real time. We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale collision-free, robot-specific audio-motion pairs. Our architecture features a Hierarchical Semantic-Acoustic Aligner that extracts multi-granular prosodic and semantic cues directly from raw audio tokens. These cues drive a Streaming Conditional Motion Generator based on a diffusion transformer with conditional flow matching. To ensure high responsiveness, we introduce Anti-Inertia CFG Masking, which prevents the model from collapsing into repetitive historical patterns by compelling it to proactively mine control signals from the audio modality. Finally, an MPC-based safety filter ensures real-time, collision-free execution on physical hardware. Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.
Zifan Wang, Ziang Ren, Pengyang Shi +7
Tsinghua University · Galbot Inc. · Beijing Institute of Technology +3
Robust human-robot interaction in complex environments requires accurate gesture perception, semantic scene understanding, and reliable task planning under limited onboard computing resources. This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents. The proposed detector, incorporates the Convolutional Block Attention Module (CBAM) into the neck and replaces the baseline bounding-box regression objective with Distance-IoU (DIoU) loss. These modifications improve feature discrimination and localization for small or partially occluded gestures in complex backgrounds. The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback. Experiments on a public gesture dataset and a custom dataset show that YOLO-DC achieves precision values of 98.9% and 95.0%, with [email protected] values of 90.7% and 92.7%, respectively. System-level evaluation yields success rates of 95%, 88%, and 82% for single-action, composite-action, and vision-dependent tasks. A 30 participant evaluation yields an overall mean satisfaction score of 3.69 out of 5. These results demonstrate the feasibility of combining refined gesture detection with multimodal agents for resource-constrained robotic interaction.
Meaningful human-robot interaction (HRI) requires a robot to continuously assess user engagement through persistent user tracking. However, state-of-the-art Multi-Object Tracking models are heavily optimized for surveillance or autonomous driving. A social robot faces distinct egocentric challenges, such as humans moving in unpredictable nonlinear patterns, obstructing each other, or leaving and reentering the scene. These dynamics trigger frequent identity switches (IDSW), causing the robot to lose its footing mid-conversation. To address this, we introduce a focused, custom-annotated egocentric dataset collected via the Furhat robot. We present a systematic evaluation isolating detection errors from tracking logic, comparing face versus body tracking, and assessing the impact of extended memory and appearance re-identification (ReID). Results indicate that increasing temporal memory mitigates prolonged occlusions but fails on complex dynamic events. Integrating ReID resolves complex switches but exhibits opposing effects: it substantially improves body tracking stability, yet causes facial IDSW to spike due to profile angle sensitivity. Ultimately, our optimized pipeline reduces IDSW by 49% compared to a standard tracking-by-detection baseline, effectively mitigating interaction breakdowns. As standard benchmarks lack dense, close-quarter occlusions, this work highlights the critical need for natively captured social dynamics to truly validate HRI perception models.
Jessica Wenninger, Gabriel Skantze
Furhat Robotics, Stockholm, Sweden · University of Naples Federico II, Naples, Italy