CFGComposer: Compositional Audio-Text-to-Motion Generation via Copula Score Correction
Organizations: Tsinghua University · Galbot · CFCS, School of Computer Science, Peking University · State Key Laboratory of General Artificial Intelligence ·
Abstract
Generating whole-body gestures that are simultaneously rhythmically synchronized with speech audio and semantically aligned with text descriptions is a fundamental challenge in embodied AI and social robotics. We tackle this Audio-Text-to-Motion (AT2M) problem by composing two independently-trained diffusion models at inference time and refine bias in joint inference by modeling the copula score . A rigorous theoretical analysis reveals that naive epsilon blending corresponds to a Product-of-Experts formulation that omits the copula score : the cross-modal interaction between audio and text that neither marginal model can capture. We propose CFGComposer: a jointly-trained AT2M diffusion model that estimates via a CFG analogy and injects it as an -space correction. Experiments on BEAT demonstrate best-in-class semantic fidelity (SRGR , FID ) with GT-level beat alignment (BA vs. GT ), and 76-83% win rate in blind pairwise human evaluation against all prior methods. Physical deployment on a Unitree G1 humanoid confirms direct sim-to-real transfer.
Figures & tables
| Method | Automatic | Human Evaluation | ||||||||
| G-MSE | F-MSE | BA | SRGR | FID | DIV | HN | SA | BM | HD | |
| Ground Truth | – | – | 0.1743 | 1.0000 | – | 17.07 | – | – | – | – |
| DiffSHEG [ 1 ] | 0.0178 | 0.0163 | 0.2228 | 0.7749 | 92.21 | 8.82 | ||||
| SemTalk [ 6 ] | 0.0402 | 0.0411 | 0.1808 | 0.7354 | 177.91 | 11.28 | ||||
| Semantic Gesticulator [ 7 ] | 0.0254 | 0.0245 | 0.2127 | 0.7216 | 104.56 | 16.31 | ||||
| RoboGesture [ 8 ] | 0.0159 | 0.0138 | 0.1903 | 0.7851 | 489.16 | 24.51 | ||||
| Variant | G-MSE | F-MSE | BA | SRGR | FID | DIV |
|---|---|---|---|---|---|---|
| Ground Truth | – | – | 0.1743 | 1.0000 | – | 17.07 |
| DiffSHEG (pretrained) | 0.0178 | 0.0163 | 0.2228 | 0.7749 | 92.21 | 8.82 |
| AT2M fine-tune | 0.0139 | 0.0118 | 0.1846 | 0.7985 | 68.45 | 11.18 |
| BEAT+AT2M mixed | 0.0154 | 0.0140 | 0.1838 | 0.7793 | 62.31 | 13.39 |
| RoboGesture (pretrained) | 0.0159 | 0.0138 | 0.1903 | 0.7851 | 489.16 | 24.51 |
| AT2M fine-tune | 0.0183 | 0.0179 | 0.2085 | 0.8018 | 179.52 | 0.01 |
Co-Speech Gesture Generation
Co-speech gesture generation creates body movements that express and synchronize with spoken language.
See all 56 papers