Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot

Jin Jiang; Kun Li; Jiancong Ma; Shengcai Liao

cs.ROOct 5, 2026

Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot

Authors: Jin Jiang, Kun Li, Jiancong Ma, Shengcai Liao

Organizations: College of Computing and Artificial Intelligence (CCAI), United Arab Emirates University (UAEU), Al Ain, Abu Dhabi, United Arab Emirates

Abstract

Expressive humanoid interaction requires speech, facial animation, and body gestures to form a coherent response. However, many full-body humanoid robots produce speech and gestures without a visually expressive face, while talking-face animation and robot gesture generation are typically developed separately. We present Talk, Render, Act (TRABot), an agent-based framework comprising specialized agents for motion-atom construction, dialogue generation, motion planning, and facial animation. First, to produce natural and semantically meaningful gestures, we construct Robot-Ready Semantic Motion Atoms by segmenting long-form, G1-retargeted BEAT2 motion into units with natural gesture boundaries, human-verified communicative functions, and feasible trajectories. Second, to preserve semantic order and coordinate body motion with the spoken response, we introduce a Semantic-Conditioned Compositional Planner. Given an ordered semantic function sequence and an estimated response duration, the planner selects approved atoms to realize the longest feasible action sequence while accounting for transitions and neutral recovery. Finally, we deploy a Streaming Face-Speech-Body Integration system on a physical G1 humanoid, combining streaming dialogue audio, audio-driven facial animation, and semantically planned body motion in a unified real-time interaction loop. Quantitative and qualitative experiments demonstrate that TRAbot achieves the best overall performance among all compared conditions in terms of naturalness, expressiveness, and multimodal coherence.

Figures & tables

Fig. 1: Overview of TRABot. Offline Motion Atom Construction Agent converts G1-retargeted BEAT2 motion into an approved catalog of Robot-Ready Semantic Motion Atoms. Online the Dialogue Agent generates streaming reply audio and a semantic motion request. The Motion Planning Agent composes approved atoms and constructs state-dependent transitions, while the Facial Animation Agent generates facial frames from the same audio stream. The Streaming Face-Speech-Body Integration System executes speech, facial animation, and robot body motion concurrently.

Category	Communicative functions	Common family
emphasis	emphasize	beat
enumeration	enumerate	beat / metaphoric
contrast	contrast_compare	metaphoric
reference	refer_self , refer_other , refer_external	deictic
depiction	depict_size_shape , depict_direction_motion	iconic
turn management	turn_open , turn_yield	beat / metaphoric / emblem

TABLE I: Planner-facing communicative vocabulary. Communicative functions are used as online planning labels, whereas common gesture families are inherited from the original BEAT2 annotations and retained as offline metadata.

Case	User utterance	Planned communicative-function sequence
1	Introduce artificial intelligence to me.	turn_open $\rightarrow$ emphasize $\rightarrow$ depict_size_shape $\rightarrow$ neutral
2	Nice to meet you. Bye-bye.	greet_farewell $\rightarrow$ neutral
3	What are three ways to relax?	enumerate $\rightarrow$ enumerate $\rightarrow$ enumerate $\rightarrow$ neutral
4	Do you like swimming?	refer_self $\rightarrow$ deny $\rightarrow$ turn_yield $\rightarrow$ neutral
5	What is the shape of a basketball?	depict_size_shape $\rightarrow$ certainty $\rightarrow$ neutral
6	Which way does the sun move across the sky?	depict_direction_motion $\rightarrow$ neutral

TABLE II: The 12 dialogue cases used in the end-to-end evaluation and their planned communicative-function sequences. Arrows indicate execution order, and the terminal neutral state denotes recovery after motion execution.

Fig. 2: Qualitative visualizations of action-speech alignment from the user study. Each row shows a user utterance, the robot’s spoken reply, and body-motion keyframes under Simple Motion, Random Motion, and Full (TRABot). Blue-highlighted words or phrases indicate the speech content corresponding to the displayed actions.

Face offset (ms) $\downarrow$	Body offset (ms) $\downarrow$	Speech–body overlap (%) $\uparrow$	Completed without failure (%) $\uparrow$
37.5	$-666.6$	84.72	100

TABLE III: Overall system performance on 12 utterances.

Method	Binary BAS $\uparrow$	Gaussian BAS $\uparrow$
Simple Motion	0.297 $\pm$ 0.206	0.210 $\pm$ 0.129
Random Motion	0.281 $\pm$ 0.197	0.190 $\pm$ 0.145
Ours (Full model)	0.308 $\pm$ 0.141	0.225 $\pm$ 0.130

TABLE IV: Beat alignment score (12 utterances per condition). Binary BAS counts postural apexes inside model-independent emphasis windows; Gaussian BAS uses

\sigma=0.30

Condition	Natural $\uparrow$	Express $\uparrow$	Coherence $\uparrow$	Gesture-speech $\uparrow$	Preference (%) $\uparrow$
Speech	1.0958	1.1042	1.0958	N/A	0.83
Speech+Face	2.1042	2.0958	2.0833	N/A	0.42
Simple Motion	3.1458	3.1333	3.1583	3.1458	0.42
Random Motion	3.8667	3.9083	3.8708	3.8625	13.75
Full model	4.7875	4.7583	4.7917	4.7708	84.58

TABLE V: User study results on different conditions (12 utterances per condition). Mean within-turn rank (1 = worst, 5 = best).

Method	Planned Atoms per reply $\uparrow$	Unused budget (s) $\downarrow$
Random Match	1.451	2.376
Ours (Full model)	1.705	0.993

TABLE VI: Evaluation on 200 fixed replies.

Explore similar work

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation
Sep 27, 2026Chengqun Yang, Tengjie Zhu, Liang Xu +10Human Motion Generation Humanoid
Shanghai Jiao Tong University · ZTE Corporation
WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots
Jun 15, 2026Thang Tran Viet, Thanh Nguyen Canh, Gia Huy Uong +4Co-Speech Gesture Generation Importance
University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom. +1
ECHO-G: Embodied Co-speech Humanoid mOtion Generation
Sep 30, 2026Yizhao Li, Pusen Gao, Ming Wang +3Co-Speech Gesture Generation Human Motion Generation
Beihang University · Mondo Robotics · The Hong Kong University of Science and Technology +1

Sep 27, 2026, cs.ROJ/K move · Enter open · S save

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation

Chengqun Yang, Tengjie Zhu, Liang Xu, Fulong Liu+9

Shanghai Jiao Tong University · ZTE Corporation

Human Motion Generation Humanoid Whole-Body Control Motion

Open paper AskPDF arXiv

Humanoid robots are increasingly expected to serve as embodied social agents that communicate naturally with humans through face-to-face interaction. During such communication, humanoid robots require body behaviors that are synchronized with speech, affectively expressive, and suitable for real-time execution. However, existing co-speech methods are primarily developed for digital humans and lack joint support for affective control and low-latency continuous generation on physical embodiments. To bridge this gap, we present SocialHumanoid, a system for expressive humanoid behavior via one-step co-speech motion generation. Given response speech and a specified affective condition, SocialHumanoid generates each full-body motion window in a single forward pass and connects successive windows through motion-history conditioning. The generated human motion is further converted online into embodiment-compatible robot references and tracked by a whole-body controller for physical execution. To provide explicit supervision for affective body expression, we further introduce AffectMoCap, a 4-hour dataset captured from two professional actors, containing synchronized speech, body motion, fine-grained hand motion, and emotion annotations. On BEAT2, SocialHumanoid achieves the best FGD among the compared generation methods, competitive speech-motion synchrony, and approximately

6\times

faster inference than GestureLSM under the same protocol. Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution. Our project page is https://rex0191.github.io/SocialHumanoid/.

Abstract

Explore similar work

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation

WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots

ECHO-G: Embodied Co-speech Humanoid mOtion Generation