Holistic co-speech motion generation aims to synthesize coordinated facial expressions, hand gestures, and body movements from speech, which is challenging due to heterogeneous part-specific dynamics, long-range temporal coherence, and cross-part coordination. Directly modeling high-dimensional motion can be unstable, while overly compact or unstructured latent spaces may lose fine-grained motion details. We present TalkFlow, a structured latent flow framework for speech-driven holistic motion generation. TalkFlow learns a continuous three-stream latent representation for upper-body motion, lower-body motion, and facial expressions, preserving part-specific dynamics without forcing heterogeneous motion components into a single latent space. To maintain holistic coordination, lightweight cross-stream interactions are incorporated in both latent representation learning and flow-based generation. Conditioned on multi-level speech features, TalkFlow performs speech-conditioned latent flow matching, learning a continuous transport process from source latents to realistic motion latents. Experiments on the BEAT2 dataset demonstrate strong performance across facial expression accuracy, body gesture quality, and speech-motion synchronization. Qualitative comparisons and user studies further show that TalkFlow generates more natural, expressive, and speech-aligned holistic motion than state-of-the-art methods. The project page is available at https://xinmuwang.github.io/TalkFlow/.
Figures & tables
Figure 1. TalkFlow generates coordinated body, hand, and facial motion from speech via structured latent flow matching. Conditioned on speech audio and aligned text features, our method synthesizes speech-aligned holistic motion: each frame shows the generated body pose with zoomed views of the face and hand. Frames are sampled at the onset of content words across a 9 -second clip, with the corresponding waveform shown at the bottom. Test teaser description.
Figure 2. Overall architecture of TalkFlow. Stage I first encodes full-body pose and expression as two spatial branches and then forms upper-body, lower-body, and expression latent streams through temporal encoding. Stage II encodes speech features into branch-specific conditions and generates the three motion latents using a structured flow backbone. Lightweight FiLM-gated interactions coordinate upper–lower and upper–expression streams without direct lower–expression interaction. The frozen Stage-I decoder reconstructs the final SMPL-X motion.
Methods
FGD ↓
BCS ↑
DIV ↑
MSE ↓
LVD ↓
Ground Truth
0
6.86
13.05
0
0
Facial Expression Generation
FaceFormer ( Fan et al., 2022 )
-
-
-
7.79
7.59
CodeTalker ( Xing et al., 2023 )
-
-
-
8.03
7.77
Body Gesture Generation
S2G ( Ginosar et al., 2019 )
28.15
4.68
5.97
-
-
Table 1. Quantitative comparison on the BEAT2 dataset. Following prior work, methods are grouped into facial expression generation, body gesture generation, and holistic motion generation. FGD is reported in 10−1 scale, BCS in 10−1 scale, MSE in 10−8 scale, and LVD in 10−5 scale. Bold indicates the best result, and underline indicates the second-best result.
Figure 3. Perceptual user study. N=37 participants ranked TalkFlow against three baselines on four explicitly labeled criteria. Left: mean rank per criterion, with error bars denoting one standard deviation across participants; lower is better. Right: percentage shares of Rank-1 to Rank-4 votes for each method. TalkFlow obtains the lowest mean rank in realism, semantic consistency, and speech-motion synchronization, and the largest overall share of first-place votes.
Variant
FGD ↓
BCS ↑
DIV ↑
MSE ↓
LVD ↓
Single latent
13.21
7.45
8.67
7.20
7.76
Dual latent
11.48
7.72
9.07
6.66
7.16
Triple latent
w/o Int.
6.40
7.80
10.94
6.11
6.79
w/o Flow Int.
5.30
7.73
11.20
6.09
6.75
w/o AE Int.
5.40
7.72
11.01
6.30
7.09
Table 2. Ablation study on the BEAT2 dataset. “Single latent”, “Dual latent”, and “Triple latent” denote the number of latent streams. “Int.” denotes cross-stream interaction, and “Rollout Rec.” denotes training with ODE rollout reconstruction loss.
Figure 4. Facial qualitative comparison on selected spoken words. Each column corresponds to a spoken word, with the highlighted syllable shown in red. Red boxes indicate evident lip-shape errors relative to the ground truth, such as an open mouth versus a closed mouth. TalkFlow generally remains closest to the ground-truth mouth shape across the selected comparisons.
Figure 5. Qualitative comparison on representative word-level speech contents. The first row shows ground-truth motions and the corresponding speech contents, followed by results from different methods. Orange boxes highlight representative ground-truth motion regions, green boxes indicate generated motions closest to the corresponding references, and red boxes mark visible artifacts or semantically inconsistent motions. For EchoMask on “talking loudly,” the two hands intersect. For MambaTalk on “terrible” and EMAGE on “watching,” the fists intersect the torso; MambaTalk also exhibits abrupt whole-body rotation and imbalance.
Figure 6. Qualitative comparison on representative word-level speech contents. The first row shows ground-truth motions and the corresponding speech contents, followed by results from different methods. Orange boxes highlight representative ground-truth motion regions, green boxes indicate generated motions closest to the corresponding references, and red boxes mark visible artifacts or semantically inconsistent motions. In the selected examples, EMAGE exhibits abrupt whole-body rotation on “wow” and “especially,” together with visible mesh deformation on “in the street.” These cases illustrate that larger-amplitude motions may still be accompanied by mesh deformation or temporal jitter.
An embodied speaker's expressiveness depends not only on what is said, but also on how the voice sounds and how the body moves. However, existing joint speech-motion generation models are largely driven implicitly by transcript semantics or intermediate speech representations, offering little explicit control over motion hierarchy, body parts, or vocal style. To bridge this gap, we construct MotionSpeech, a structured benchmark for controllable joint speech-motion generation that includes transcripts, voice-style descriptions, and motion conditions at two hierarchical levels. Building on this benchmark, we propose DuoFlow, a dual-stream latent flow transformer that jointly denoises speech and motion under a shared noise level and a unified second-based temporal coordinate. Modality-specific residual connectors inject structured conditions into the corresponding streams, layer-wise bidirectional cross-modal attention coordinates their generation, and a monotonic phoneme pathway combines normalized progress anchors with CTC supervision to strengthen speech temporal modeling. DuoFlow provides a unified interface for fine-grained motion and vocal-style control and demonstrates the ability to synthesize high-quality, synchronized, and instruction-consistent speech-motion sequences. Project page, https://jundong-dev.github.io/DuoFlow-project-page.
Embodied conversational agents require synchronized full-body motion (body gestures and facial expressions) that aligns with speech and emotional state. Omni-modal large language models excel at multimodal understanding but produce only linguistic outputs, leaving a critical gap in embodied response generation. We identify and address a failure of emotion conditioning: like other conditional generators that under-use weak conditioning signals, a flow-matching model given both a rich audio embedding and a discrete emotion label suppresses the emotion, generating near-identical motion regardless of the specified emotion. We present EMODY Flow, a lightweight (around 35M parameters) flow-matching framework that attaches to a frozen Qwen-3 Omni model and reuses its internal Mimi audio-codecs to condition two parallel DiT generators - one for SMPL-X body pose, one for FLAME facial expressions. A training-time auxiliary emotion classifier restores emotion sensitivity by forcing generated motion to be emotion-identifiable. EMODY Flow sets a new state of the art on BEAT2 gesture quality, with FGD 0.302, Beat Correlation 0.853, and Diversity 24.62 - improving over the best prior results by 26%, 5%, and 62% respectively - and transfers to zero-shot facial animation on TFHP without domain-specific fine-tuning. Beyond these quantitative gains, the classifier yields clearly emotion-separated motion, which we demonstrate qualitatively through a multidimensional-scaling analysis of the generated gestures.
Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR
Pinxin Liu, Haiyang Liu, Jiahao Luo +3
University of Rochester · University of Tokyo · University of California Santa Cruz +2