RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction

Zifan Wang; Ziang Ren; Pengyang Shi; Zirui Wang; Chenghuai Lin; Tianze Wang; Zekun Qi; Liangliang Zhao

cs.ROAug 27, 2026

RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction

Authors: Zifan Wang, Ziang Ren, Pengyang Shi, Zirui Wang, Chenghuai Lin, Tianze Wang, Zekun Qi, Liangliang Zhao, +2 more

Organizations: Tsinghua University · Galbot Inc. · Beijing Institute of Technology · Harbin Institute of Technology · Peking University · Shanghai Qi Zhi Institute

Abstract

Enabling humanoid robots to respond to human speech with synchronized and semantically meaningful gestures is fundamental to natural human-robot interaction. However, this task faces three critical barriers: the scarcity of semantically rich datasets, the "modality eclipse" where models ignore audio cues in favor of kinematic inertia, and the sim-to-real gap regarding physical safety. We propose RoboGesture, a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system in which the robot listens, responds, and gestures in real time. We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale collision-free, robot-specific audio-motion pairs. Our architecture features a Hierarchical Semantic-Acoustic Aligner that extracts multi-granular prosodic and semantic cues directly from raw audio tokens. These cues drive a Streaming Conditional Motion Generator based on a diffusion transformer with conditional flow matching. To ensure high responsiveness, we introduce Anti-Inertia CFG Masking, which prevents the model from collapsing into repetitive historical patterns by compelling it to proactively mine control signals from the audio modality. Finally, an MPC-based safety filter ensures real-time, collision-free execution on physical hardware. Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.

Figures & tables

Model	Streaming?	Semantics?	Text-free?	Hand?	Motion?
LivelySpeaker [ 51 ]	✗	✓	✗	✗	Continuous
DiffSHEG [ 9 ]	✓	✗	✓	✓	Continuous
Semantic Gesticulator [ 50 ]	✗	✓	✗	✓	Discrete
SemTalk [ 47 ]	✗	✓	✗	✓	Discrete
Ours	✓	✓	✓	✓	Continuous

Table 1: Comparison with related methods. Streaming : whether the model has a design for streaming support. Semantics : whether the model focuses on modeling special semantic gestures. Text-free : whether the model works without text as input or intermediate modality. Hand : whether hand modeling is considered. Motion : motion generation approach.

Figure 1: Overview of RoboGesture. (a) The Semantic-Acoustic Aligner is pre-trained to decouple multi-granular audio cues from streaming tokens. (b) The Motion Generator synthesizes motions via Conditional Flow Matching, jointly conditioned on hierarchical audio features and historical motion to maintain autoregressive consistency. (c) During inference, the pipeline seamlessly integrates with upstream Speech-LLMs and employs a safety filter to drive physical humanoid robots in real-time.

Figure 2: Visualization of RoboGesture Dataset. RoboGesture refines the SeG dataset by correcting motion penetration issues (e.g., avoiding arm–body collisions), augments the EgoGesture dataset with full-body motion, and additionally incorporates newly collected ones. Each sample is annotated with detailed semantic labels.

	BEAT					SemanticBEAT
Method	FGD $\downarrow$	BC $\uparrow$	MSE $\downarrow$	DIV $\uparrow$	Col. $\downarrow$	BC $\uparrow$	DIV $\uparrow$	Col. $\downarrow$
LivelySpeaker [ 51 ]	3.0350	0.1811	0.1861	0.1485	21.41	0.2852	0.1384	13.36
DiffSHEG [ 9 ]	2.2316	0.1851	0.1753	0.1218	0.85	0.2859	0.1209	1.20
SemTalk [ 47 ]	7.9328	0.1828	0.3931	0.1781	52.82	0.2913	0.1440	42.65
Semantic Gesticulator [ 50 ]	3.0147	0.1771	0.2375	0.2818	13.11	0.2906	0.2831	13.84
Ours	0.8452	0.1866	0.1347	0.2075	0.88	0.2950	0.2041	0.13

Table 2: Quantitative comparison of our model with other methods on the BEAT and SemanticBEAT datasets. Since the SemanticBEAT dataset does not provide ground-truth motion or semantic annotations, we omit FGD and MSE for this dataset. Best results are shown in bold , and second-best results are underlined .

Figure 3: Qualitative comparison of generated co-speech gestures. We present three representative scenarios to highlight the advantages of our method. Left: Baseline methods (e.g., SG) occasionally suffer from severe self-penetration (highlighted by the orange box), whereas our model ensures physical safety and strictly avoids self-collisions. Middle: Given the explicit semantic cue “OK”, our model successfully synthesizes the precise, fine-grained “OK” hand gesture, while baselines fail to capture this specific semantic alignment. Right: For emphatic speech contexts (“seize the chance”), our model generates highly expressive body language, such as confidently patting the chest, demonstrating superior contextual richness and overall expressiveness.

	BEAT				SemanticBEAT
Model	RA $\uparrow$	SA $\uparrow$	PHC $\uparrow$	OP $\uparrow$	RA $\uparrow$	SA $\uparrow$	PHC $\uparrow$	OP $\uparrow$
LivelySpeaker [ 51 ]	-0.2223	-0.3994	-0.4304	-0.3064	-0.2126	-0.3015	-0.3466	-0.3497
Diffsheg [ 9 ]	-0.0058	-0.2691	-0.1440	-0.0992	-0.0090	-0.0817	-0.0458	-0.0626
SemTalk [ 47 ]	0.0128	0.0151	0.1679	-0.0132	0.0650	-0.0012	0.1207	0.0970
SemanticGesticulator [ 50 ]	0.0819	0.2264	-0.0273	0.1137	0.0296	0.1605	-0.0347	0.0640
Ours	0.1334	0.4270	0.4338	0.3050	0.1269	0.2239	0.3066	0.2514

Table 3: Human Evaluation on BEAT and SemanticBEAT datasets. We evaluate models across four key dimensions: Rhythmic Alignment (RA), Semantic Accuracy (SA), Physical & Hand Consistency (PHC), and Overall Preference (OP). All values are reported as mean

\pm

standard error. Higher values indicate better performance.

Configuration	SA $\uparrow$	HD $\uparrow$	HN $\uparrow$	BM $\uparrow$
Data Scale
Ours w/o Semi-data	4.281	2.321	4.945	4.628
Ours (1/4 Semi-data)	6.316	5.980	6.882	6.892
Architecture & Strategy
Ours w/o Context Motion	4.237	4.263	2.192	1.928
Ours w/o FiLM Injection	5.181	4.389	4.506	4.589

Table 4: Ablation study on the key components of our method. SA: Semantic Action Score, HD: Hand Detail Score, HN: Human-likeness & Naturalness, BM: Beat Matching Score. Bold indicates the best performance among all configurations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Real-world deployment pipeline of our humanoid co-speech interaction system. Human speech is first processed by an interaction pipeline including ASR, a LoRA-tuned Qwen module, and TTS. The synthesized speech is then fed into the audio tokenizer, semantic-acoustic aligner, and motion generator in a streaming manner. The generated motion is further refined by an MPC-based safety filter before being executed on the deployed humanoid platform, i.e., Unitree G1 equipped with BrainCo dexterous hands.

Configuration	SA $\uparrow$	HD $\uparrow$	HN $\uparrow$	BM $\uparrow$
Data Scale
Ours w/o Semi-data	4.281	2.321	4.945	4.628
Ours (1/4 Semi-data)	6.316	5.980	6.882	6.892
Architecture & Strategy
Ours w/o Context Motion	4.237	4.263	2.192	1.928
Ours w/o FiLM Injection	5.181	4.389	4.506	4.589

Appendix

Table 6: Ablation study on the key components of our method. SA: Semantic Action Score, HD: Hand Detail Score, HN: Human-likeness & Naturalness, BM: Beat Matching Score. Bold indicates the best performance among all configurations.

Model	Audio-swap MSE	History-swap MSE	Hist./Audio Ratio
Full Model	$0.0619$	$0.1261$	$2.04$
Only-DiT	$0.0312$	$0.1428$	$4.57$

Appendix

Table 7: Controlled perturbation analysis on 50 validation samples. We report the output MSE when swapping the incoming audio while fixing past motion, or swapping past motion while fixing the audio. A larger History/Audio ratio indicates a stronger tendency to rely on historical motion rather than incoming audio cues.

	Audio-shuffle FGD			Gradient
Variant	Normal	Shuffled	$\Delta~\uparrow$	$G_{hist}$	$G_{audio}$	$G_{hist}/G_{audio}~\downarrow$
Only-DiT	1.325	1.440	$+0.11$	0.365	0.075	4.88
w/o CFG	1.504	1.170	$-0.33$	0.042	0.074	0.57
Full	0.624	9.151	$\mathbf{+8.53}$	0.134	0.245	0.55

Appendix

Table 8: Direct evidence for modality eclipse on BEAT. Audio-shuffle FGD measures how much performance degrades when the input audio is shuffled (

\Delta=

Shuffled

-

Normal; larger is better). The gradient analysis reports the sensitivity of the loss to the history vs. audio conditions.

Figure 5: Visualization of RoboGesture Dataset. RoboGesture is a comprehensive full-body gesture dataset comprising over 300 high-quality motion classes collected from EgoGesture, SeG, full-body mocap recordings, online sources, and user surveys. All gestures are retargeted, optimized for physical feasibility, and validated on a real humanoid robot, ensuring human-like expressiveness while remaining fully executable on hardware.

Method	Rhythm Align.	Semantic Acc.	Physical Acc. & Cons.	Overall Pref.
Ours	$0.1334\pm 0.0624$	$0.4270\pm 0.0702$	$0.4338\pm 0.0693$	$0.3050\pm 0.0673$
SemanticGesticulator	$0.0819\pm 0.1693$	$0.2264\pm 0.1834$	$-0.0273\pm 0.1708$	$0.1137\pm 0.1751$
SemTalk	$0.0128\pm 0.1659$	$0.0151\pm 0.1741$	$0.1679\pm 0.1827$	$-0.0132\pm 0.1624$
DiffSHEG	$-0.0058\pm 0.1526$	$-0.2691\pm 0.1972$	$-0.1440\pm 0.1943$	$-0.0992\pm 0.1885$
LivelySpeaker	$-0.2223\pm 0.1632$	$-0.3994\pm 0.1846$	$-0.4304\pm 0.1836$	$-0.3064\pm 0.1737$

Appendix

Table 10: Results on the BEAT Benchmark

Method	Rhythm Align.	Semantic Acc.	Physical Acc. & Cons.	Overall Pref.
Ours	$0.1269\pm 0.0875$	$0.2239\pm 0.0880$	$0.3066\pm 0.0917$	$0.2514\pm 0.0913$
SemanticGesticulator	$0.0296\pm 0.0869$	$0.1605\pm 0.0873$	$-0.0347\pm 0.0922$	$0.0640\pm 0.0360$
SemTalk	$0.0650\pm 0.0348$	$-0.0012\pm 0.0348$	$0.1207\pm 0.0903$	$0.0970\pm 0.0901$
DiffSHEG	$-0.0090\pm 0.0888$	$-0.0817\pm 0.0890$	$-0.0458\pm 0.0360$	$-0.0626\pm 0.0922$
LivelySpeaker	$-0.2126\pm 0.0878$	$-0.3015\pm 0.0883$	$-0.3466\pm 0.0913$	$-0.3497\pm 0.0915$

Appendix

Table 11: Results on the SemanticBEAT Benchmark

Method	FGD $\downarrow$	BC $\uparrow$	DIV $\uparrow$	Sem. BC $\uparrow$	Sem. DIV $\uparrow$
DiffSHEG (retargeted)	2.232	0.185	0.122	0.286	0.121
DiffSHEG (direct-trained)	2.195	0.179	0.164	0.290	0.167
SG (retargeted)	3.015	0.177	0.282	0.291	0.283
SG (direct-trained)	2.781	0.165	0.258	0.291	0.259
RoboGesture (robot-space)	0.845	0.187	0.208	0.295	0.204

Appendix

Table 12: Retargeting is not a confounder. For each baseline we compare the retargeted variant against a variant re-trained directly in robot joint space. The two setups yield nearly identical scores, and both trail RoboGesture, indicating that our gains are not attributable to retargeting artifacts.

	Top-1 $\uparrow$	Top-5 $\uparrow$	F1 $\uparrow$
Strict	64.71	87.00	70.31
Semantic-equivalent	72.76	95.67	–

Appendix

Table 13: Accuracy of the 300-class semantic head of our aligner on the test set. “Strict” requires the exact class, while “Semantic-equivalent” credits semantically interchangeable classes.

Explore similar work

WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots
Jun 15, 2026Thang Tran Viet, Thanh Nguyen Canh, Gia Huy Uong +4Human Motion Generation Audio-Visual Synchronization
University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom. +1
ECHO-G: Embodied Co-speech Humanoid mOtion Generation
Sep 30, 2026Yizhao Li, Pusen Gao, Ming Wang +3Text-to-Motion Generation Human Motion Generation
Beihang University · Mondo Robotics · The Hong Kong University of Science and Technology +1
Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot
Oct 5, 2026Jin Jiang, Kun Li, Jiancong Ma +1Robot Motion Generation Co-Speech Gesture Generation
College of Computing and Artificial Intelligence (CCAI), United Arab Emirates University (UAEU), Al Ain, Abu Dhabi, United Arab Emirates

Jun 15, 2026J/K move · Enter open · S save

WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots

Thang Tran Viet, Thanh Nguyen Canh, Gia Huy Uong, Phuc Van Dinh+3

University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom.+1

Human Motion Generation Audio-Visual Synchronization Robot Motion Generation Co-Speech Gesture Generation Human-Robot Interaction Humanoid Robotics

Open paper AskPDF arXiv

Expressive co-speech gestures are crucial for natural human--robot interaction, yet generating them on physical humanoid robots remains challenging because, unlike virtual avatars, robots must synchronize gestures with speech under strict kinematic and actuator constraints. We present WaveSync, a hybrid framework in which a Large Language Model decomposes dialogue responses into structured semantic schemas and assigns per-word importance weights, forming a continuous Semantic Importance Wave. Gesture trajectories are shaped through Dynamic Movement Primitives to ensure kinematic feasibility while enhancing expressiveness. A Wavefront Optimization stage aligns gesture stroke peaks with speech emphasis peaks and resolves residual temporal conflicts through gesture-duration compression and forward propagation. Experimental evaluation across five dialogue scenarios demonstrates effective gesture--speech alignment and favorable performance in both objective and subjective evaluations. The results further show that the key components of WaveSync contribute to producing gestures that are expressive, semantically grounded, and kinematically feasible. The code, resources, and videos are available at WaveSync.

Abstract

Explore similar work

WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots

ECHO-G: Embodied Co-speech Humanoid mOtion Generation

Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot