TalkFlow: Structured Latent Flow Matching for Speech Driven Holistic Motion Generation
Organizations: Stony Brook University, Stony Brook, NY, USA · Futurewei Technologies Inc, Santa Clara, CA, USA
Abstract
Holistic co-speech motion generation aims to synthesize coordinated facial expressions, hand gestures, and body movements from speech, which is challenging due to heterogeneous part-specific dynamics, long-range temporal coherence, and cross-part coordination. Directly modeling high-dimensional motion can be unstable, while overly compact or unstructured latent spaces may lose fine-grained motion details. We present TalkFlow, a structured latent flow framework for speech-driven holistic motion generation. TalkFlow learns a continuous three-stream latent representation for upper-body motion, lower-body motion, and facial expressions, preserving part-specific dynamics without forcing heterogeneous motion components into a single latent space. To maintain holistic coordination, lightweight cross-stream interactions are incorporated in both latent representation learning and flow-based generation. Conditioned on multi-level speech features, TalkFlow performs speech-conditioned latent flow matching, learning a continuous transport process from source latents to realistic motion latents. Experiments on the BEAT2 dataset demonstrate strong performance across facial expression accuracy, body gesture quality, and speech-motion synchronization. Qualitative comparisons and user studies further show that TalkFlow generates more natural, expressive, and speech-aligned holistic motion than state-of-the-art methods. The project page is available at https://xinmuwang.github.io/TalkFlow/.
Figures & tables
| Methods | FGD | BCS | DIV | MSE | LVD |
| Ground Truth | 0 | 6.86 | 13.05 | 0 | 0 |
| Facial Expression Generation | |||||
| FaceFormer ( Fan et al., 2022 ) | - | - | - | 7.79 | 7.59 |
| CodeTalker ( Xing et al., 2023 ) | - | - | - | 8.03 | 7.77 |
| Body Gesture Generation | |||||
| S2G ( Ginosar et al., 2019 ) | 28.15 | 4.68 | 5.97 | - | - |
| Variant | FGD | BCS | DIV | MSE | LVD |
|---|---|---|---|---|---|
| Single latent | 13.21 | 7.45 | 8.67 | 7.20 | 7.76 |
| Dual latent | 11.48 | 7.72 | 9.07 | 6.66 | 7.16 |
| Triple latent | |||||
| w/o Int. | 6.40 | 7.80 | 10.94 | 6.11 | 6.79 |
| w/o Flow Int. | 5.30 | 7.73 | 11.20 | 6.09 | 6.75 |
| w/o AE Int. | 5.40 | 7.72 | 11.01 | 6.30 | 7.09 |
Human Motion Generation
Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.
See all 309 papers