ECHO-G: Embodied Co-speech Humanoid mOtion Generation

Yizhao Li; Pusen Gao; Ming Wang; Shaojie Shen; Shuo Yang; Hao Xu

cs.ROSep 30, 2026

ECHO-G: Embodied Co-speech Humanoid mOtion Generation

Authors: Yizhao Li, Pusen Gao, Ming Wang, Shaojie Shen, Shuo Yang, Hao Xu

Organizations: Beihang University · Mondo Robotics · The Hong Kong University of Science and Technology · Nanjing University

Abstract

Generating full-body co-speech motion for humanoid robots requires coordinating speech prosody, linguistic content, and embodiment-specific motion. To this end, we present ECHO-G, a framework jointly conditioned on speech audio and timed transcripts. Its Speech-Grounded Diffusion Transformer (SGDiT) combines frame-aligned acoustic features with token-level linguistic context, preserving their distinct granularities. Trained with rectified flow matching, it models one-to-many utterance-motion relationships directly in robot space. To support training and evaluation, we introduce a BEAT2-derived audio-text-robot dataset and a benchmark covering co-speech characteristics, robot-motion quality, and runtime efficiency. Comparative evaluation supports direct robot-space generation over the evaluated human-motion generation and retargeting pipelines, while modality ablations highlight the benefits of joint audio-text conditioning. We further demonstrate deployment on a physical humanoid robot. A complementary video-rating study also favors joint conditioning over the alternatives. The dataset and training, inference, and evaluation code are available through our project page.

Figures & tables

Fig. 2: SGDiT architecture and tracking interface. Frame-aligned acoustic features are combined with noisy motion-frame features to form motion tokens. Contextual transcript embeddings provide a shared key–value memory for global and temporally biased text attention. Their attention distributions are fused within each transformer block before value aggregation and residual injection into the motion stream. Rectified-flow sampling produces robot-motion references, whose joint-position components are passed to a fixed whole-body motion tracker for execution.

Fig. 3: Temporal conditioning in the local transcript-attention branch. Orange and purple heatmaps show global and local attention weights, respectively. The blue heatmap represents the clipped Gaussian log-prior $\widetilde{b}_{\tau n}$ derived from motion-frame times $u_{\tau}$ and token-center times $m_{n}$ . Multiplying global weights by the exponentiated log-prior and renormalizing yields the local weights. Matrices are transposed for display; colored and dashed arrows indicate stronger and weaker attention, respectively.

Fig. 4: Conditioning comparison on an utterance outside BEAT2. Rows show joint audio–text conditioning (Ours), text-only, and audio-only outputs at matched frame indices. In the selected frames, joint conditioning exhibits broader arm extensions, whereas the unimodal outputs generally keep the hands closer to the torso. Colored arrows and circles highlight selected arm and hand movements.

Fig. 5: Real-robot execution on an utterance outside BEAT2. ECHO-G’s predicted joint angles are supplied as joint-position references to a fixed SONIC motion tracker while the corresponding speech is played. Frames progress from left to right through the book-recommendation utterance shown below, illustrating changes in arm extension, hand height, and torso posture. Colored arrows highlight selected arm movements, while the orange skeletal overlays outline the body configuration.

Method	Co-Speech Motion Characteristics				Robot Motion Quality			Efficiency
Method	FGD $\downarrow$	$\Delta$ Div $\downarrow$	MM $\uparrow$	$\Delta$ BA $\downarrow$	$\Delta$ Jerk $\downarrow$	Foot Err. (m) $\downarrow$	C-Slide (m/s) $\downarrow$	E2E Time (ms/frame) $\downarrow$	Peak RAM $\Delta$ (MB) $\downarrow$
EMAGE+GMR	4.976	0.749	0	0.172	26.951	0.013	0.163	21.3	2661
GestureLSM+GMR	5.008	0.561	1.016	0.158	24.444	0.010	0.169	20.5	3804
Human-Retargeted	4.725	0.408	1.498	0.161	8.001	0.003	0.050	6.56	18714
Ours	2.278	0.320	1.786	0.063	9.039	0.008	0.052	5.96	18542

TABLE I: Comparison of direct robot-space generation with human-motion generation followed by retargeting or learned mapping on the BEAT2 speaker-held-out evaluation data.

\Delta

Div,

\Delta

BA, and

\Delta

Jerk denote absolute deviations from matched ground-truth statistics over each method’s evaluated motion range. Best and second-best results are shown in bold and underlined text, respectively.

Method	FGD $\downarrow$	$\Delta$ Div $\downarrow$	MM $\uparrow$	$\Delta$ BA $\downarrow$	$\Delta$ Jerk $\downarrow$	Foot Err. (m) $\downarrow$	C-Slide (m/s) $\downarrow$
Audio-only	2.360	0.360	1.702	0.081	4.497	0.008	0.038
Text-only	2.436	0.429	1.681	0.113	5.940	0.009	0.049
Ours	2.278	0.320	1.786	0.063	9.039	0.008	0.052

TABLE II: Ablation of conditioning modalities for direct robot-space generation on the BEAT2 speaker-held-out evaluation data.

\Delta

Div,

\Delta

BA, and

\Delta

Jerk denote absolute deviations from the shared ground-truth statistics. Best and second-best results are shown in bold and underlined text, respectively. Ties at the displayed precision receive identical highlighting.

Method	Overall	Human- likeness	Rhythm Matching	Motion Quality
EMAGE+GMR	1.80	1.87	1.95	1.59
Human-Retargeted	2.65	2.67	2.38	2.90
Unimodal (pooled)	3.05	3.01	3.13	3.01
Ours	3.49	3.45	3.31	3.70

TABLE III: Mean user-study ratings from 45 participants on a five-point scale. Higher is better. Unimodal scores are pooled as described in the text. Best and second-best means are shown in bold and underlined text, respectively.

Explore similar work

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation
Sep 27, 2026Chengqun Yang, Tengjie Zhu, Liang Xu +10Human Motion Generation Humanoid
Shanghai Jiao Tong University · ZTE Corporation
PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generation
Jun 18, 2026Zhangzhao Liang, Xiaofen Xing, Mingyue Yang +2Human Motion Generation Human Motion
South China University of Technology · DexForce Technology · Foshan University
WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots
Jun 15, 2026Thang Tran Viet, Thanh Nguyen Canh, Gia Huy Uong +4Co-Speech Gesture Generation Importance
University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom. +1

Sep 27, 2026, cs.ROJ/K move · Enter open · S save

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation

Chengqun Yang, Tengjie Zhu, Liang Xu, Fulong Liu+9

Shanghai Jiao Tong University · ZTE Corporation

Human Motion Generation Humanoid Whole-Body Control Motion

Open paper AskPDF arXiv

Humanoid robots are increasingly expected to serve as embodied social agents that communicate naturally with humans through face-to-face interaction. During such communication, humanoid robots require body behaviors that are synchronized with speech, affectively expressive, and suitable for real-time execution. However, existing co-speech methods are primarily developed for digital humans and lack joint support for affective control and low-latency continuous generation on physical embodiments. To bridge this gap, we present SocialHumanoid, a system for expressive humanoid behavior via one-step co-speech motion generation. Given response speech and a specified affective condition, SocialHumanoid generates each full-body motion window in a single forward pass and connects successive windows through motion-history conditioning. The generated human motion is further converted online into embodiment-compatible robot references and tracked by a whole-body controller for physical execution. To provide explicit supervision for affective body expression, we further introduce AffectMoCap, a 4-hour dataset captured from two professional actors, containing synchronized speech, body motion, fine-grained hand motion, and emotion annotations. On BEAT2, SocialHumanoid achieves the best FGD among the compared generation methods, competitive speech-motion synchrony, and approximately

6\times

faster inference than GestureLSM under the same protocol. Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution. Our project page is https://rex0191.github.io/SocialHumanoid/.

Abstract

Explore similar work

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation

PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generation

WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots