cs.ROSep 27, 2026
SaveSocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation
Organizations: Shanghai Jiao Tong University · ZTE Corporation
Abstract
Humanoid robots are increasingly expected to serve as embodied social agents that communicate naturally with humans through face-to-face interaction. During such communication, humanoid robots require body behaviors that are synchronized with speech, affectively expressive, and suitable for real-time execution. However, existing co-speech methods are primarily developed for digital humans and lack joint support for affective control and low-latency continuous generation on physical embodiments. To bridge this gap, we present SocialHumanoid, a system for expressive humanoid behavior via one-step co-speech motion generation. Given response speech and a specified affective condition, SocialHumanoid generates each full-body motion window in a single forward pass and connects successive windows through motion-history conditioning. The generated human motion is further converted online into embodiment-compatible robot references and tracked by a whole-body controller for physical execution. To provide explicit supervision for affective body expression, we further introduce AffectMoCap, a 4-hour dataset captured from two professional actors, containing synchronized speech, body motion, fine-grained hand motion, and emotion annotations. On BEAT2, SocialHumanoid achieves the best FGD among the compared generation methods, competitive speech-motion synchrony, and approximately faster inference than GestureLSM under the same protocol. Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution. Our project page is https://rex0191.github.io/SocialHumanoid/.
Figures & tables
Fig. 1: Expressive humanoid behavior during a robot response. SocialHumanoid converts a spoken response and an affective condition into continuous full-body co-speech motion through one-step inference, then retargets and executes the motion online on a physical humanoid.
Fig. 2: SocialHumanoid for expressive humanoid behavior. (a) AffectMoCap provides professional affective full-body motion supervision for training. (b) During a robot response, audio features, an affective condition, Gaussian noise, and preceding motion context drive the affect-conditioned one-step generator, which produces each SMPL-X motion window with NFE and links past and current chunks through history tokens. (c) Online retargeter and whole-body control convert the generated stream into synchronized behavior on the physical humanoid. (d) The interactive communication frontend maps user speech to an LLM response and TTS audio; after one response is executed, the system returns to the next user utterance and repeats the response pipeline.
Fig. 3: One-step co-speech motion generation. The improved MeanFlow module predicts an interval velocity in the part-wise latent space, enabling one-step inference for each affect-conditioned motion window.
| Dataset | Body Scope | Affect Label | Scale |
|---|---|---|---|
| Trinity Mocap [ 59 ] | Full body (3D) | – | 4 h |
| S2G-2D [ 60 ] | Upper body (2D) | – | 60 h |
| TED-2D [ 61 ] | Upper body (2D) | – | 97 h |
| TWH Mocap [ 62 ] | Full body (3D) | – | 20 h |
| TED-3D [ 1 ] | Upper body (3D) | – | 97 h |
| S2G-3D [ 63 ] | Upper body (3D) | – | 38 h |
TABLE I: Data supervision for expressive co-speech body behavior. We compare AffectMoCap with representative datasets in terms of body scope, affect/style annotation, and recording scale. PGT-Mesh denotes SMPL or SMPL-X motion predicted or fitted from video, whereas MC-Mesh denotes mesh motion converted from motion-capture data. For the Embody 3D dataset [ 58 ] , we report statistics only for the Dyadic Conversations section.
Fig. 4: Representative samples from AffectMoCap. The two rows show examples from the two professional actors. Each sample pairs the motion-capture skeleton with its fitted SMPL-X mesh, illustrating varied arm configurations, articulated hand poses, and full-body postures in the dataset.
Fig. 5: Duration distribution of AffectMoCap. The left pie chart reports recording time by actor gender, and the right reports recording time across the eight emotion labels.
Fig. 6: AffectMoCap capture setup. The composite figure shows the hardware components used for synchronized affective co-speech capture: the OptiTrack optical tracking camera, the actor wearing a full-body marker suit, the hand motion-capture gloves, and the wireless microphone kit. All actors provided informed consent for data collection, analysis, and publication of anonymized results.
| Methods | FGD ( ) | BC ( ) | Diversity ( ) | NFE ( ) |
|---|---|---|---|---|
| Ground-Truth | – | 0.703 | 11.97 | – |
| HA2G [ 4 ] | 12.32 | 0.677 | 8.626 | 30 |
| DisCo [ 27 ] | 9.417 | 0.643 | 9.912 | 1 |
| CaMN [ 3 ] | 6.644 | 0.676 | 10.86 | 1 |
| DiffSHEG [ 8 ] | 7.141 | 0.743 | 8.21 | 25 |
| TalkShow [ 26 ] | 6.209 | 0.695 | 13.47 | 64 |
TABLE II: Motion quality and one-step sampling on BEAT2. We report FGD , BC, Diversity, and the number of forward process steps (NFE) required to generate a 128-frame motion window.
| Methods | AIST (s) ( ) |
|---|---|
| DiffSHEG [ 8 ] | 4.4136 |
| TalkShow [ 26 ] | 0.9279 |
| ProbTalk [ 29 ] | 0.1472 |
| EMAGE [ 7 ] | 0.1121 |
| MambaTalk [ 30 ] | 0.1580 |
| SynTalker [ 36 ] | 10.8981 |
TABLE III: Controlled generation-runtime comparison on BEAT2. AIST is measured using the same 128-frame protocol on a single NVIDIA RTX 3090. Only methods with compatible full-body checkpoints are included.
Fig. 7: One-step co-speech motion deployed on the physical humanoid. We compare methods using three BEAT2 test clips. Baseline motions are generated offline, whereas SocialHumanoid generates motion online; all outputs are evaluated through the same retargeting and whole-body control pipeline. Each column group shares the same speech clip, and red words indicate the corresponding speech content. Additional comparisons are provided on the project page.
Fig. 8: Affect-conditioned humanoid behavior under fixed speech input. Rows correspond to different emotion labels, while speech audio, speaker identity, and displayed timestamps are fixed. Each column therefore compares robot poses at the same temporal position. Contempt is an additional affect label inherited from BEAT2 and is not included in the eight-label AffectMoCap annotation set.
| Source | Stimulus | Label Source | Match Acc (%) ( ) |
|---|---|---|---|
| BEAT2 [ 7 ] | Motion only | BEAT emotion label | 76.4 |
| BEAT2 [ 7 ] | Audio + motion | BEAT emotion label | 78.2 |
| ZeroEGGS [ 20 ] | Motion only | Dataset style label | 86.8 |
| ZeroEGGS [ 20 ] | Audio + motion | Dataset style label | 84.2 |
| AffectMoCap (ours) | Motion only | Actor emotion label | 88.6 |
| AffectMoCap (ours) | Audio + motion | Actor emotion label | 92.4 |
TABLE IV: Perceptual evaluation of expressive co-speech motion. Paired audio+motion and motion-only conditions separate speech cues from body-motion cues, while the BEAT2-only variant isolates the effect of AffectMoCap fine-tuning.
| System setting | Naturalness ( ) | Emotion Fit ( ) | Delay Impact ( ) |
|---|---|---|---|
| w/o affective condition | 4.06 | 3.64 | 3.18 |
| w/o AffectMoCap fine-tuning | 3.94 | 3.30 | 3.22 |
| w/o speech playback | 4.12 | 3.62 | 3.56 |
| MeanFlow retraining and sampling | 4.10 | 3.98 | 3.28 |
| SocialHumanoid | 4.23 | 3.92 | 3.14 |
TABLE V: Perceptual evaluation of humanoid response behavior. Naturalness and emotion fit are rated on a five-point scale, while delay impact measures perceived disruption from response latency on the same scale (lower is better).
| Variant | FGD ( ) | BC ( ) | Diversity ( ) | NFE ( ) |
|---|---|---|---|---|
| w/o temporal attention | 4.648 | 0.787 | 12.14 | 1 |
| w/o spatial attention | 5.031 | 0.783 | 12.51 | 1 |
| w/o affective condition | 4.730 | 0.768 | 12.88 | 1 |
| MeanFlow retraining and sampling | 4.078 | 0.769 | 13.89 | 1 |
| Shortcut retraining and sampling | 4.241 | 0.728 | 13.75 | 8 |
| Full model | 3.929 | 0.770 | 12.36 | 1 |
TABLE VI: Ablation study on BEAT2. We report FGD , BC, Diversity, and NFE for model-architecture variants and retraining with MeanFlow or Shortcut.
| Module | Time (s) |
|---|---|
| Motion generation | 0.006 |
| Online retargeting | 1.248 |
| Whole-body controller computation | 0.008 |
TABLE VII: Runtime of the deployed motion path. The processing time of each motion-related stage is measured during real-robot operation for a 128-frame generation window on a single NVIDIA RTX 4090.
Fig. 9: Training loss comparison. MeanFlow and improved MeanFlow are compared over training epochs under the same training setup.
| Hyperparameter | Value |
|---|---|
| History length | 4 latent tokens (16 motion frames) |
| Motion window length | 128 frames (32 latent tokens) |
| Streaming hop length | 112 frames (28 latent tokens) |
| Training clip stride | 20 motion frames |
| RVQ temporal downsampling | |
| Residual quantizer stages | 6 per body region |
TABLE VIII: Key co-speech generation hyperparameters. Values follow the improved MeanFlow training and streaming inference implementation.
Fig. 10: Effect of classifier-free guidance scale on FGD. The scale is varied from 1.0 to 2.0 while all other inference settings remain fixed.
Fig. 11: Predefined communicative robot actions. Three representative action clips are executed on the physical humanoid. The clips use the same retargeting and control pipeline as generated co-speech motion and are not outputs of the one-step generator.
Fig. 12: Multilingual speech-driven execution. Real-robot sequences are driven by synthesized speech in English, Chinese, Japanese, and Spanish. Each block presents consecutive frames from one language condition using the same motion generation, retargeting, and control pipeline.