Organizations: Faculty of Information Engineering and Automation Kunming University of Science and Technology Kunming, China · Yunnan Key Laboratory of Artificial Intelligence · Utrecht University Utrecht, Netherlands · The Max Planck Institute for Psycholinguistics Nijmegen, Netherlands · Zhejiang University Hangzhou, China
Co-speech gesture generation in dyadic conversations for creating immersive virtual avatars remains highly challenging. Existing models often struggle to balance kinematic stability with rhythmic responsiveness and suffer from semantic entanglement, leading to either generic mean-pose regression or unnatural micro-jitters. To address these challenges, we propose SynerGest, a high-fidelity dyadic gesture generation framework grounded in Contrastive Flow Matching (CFM). Specifically, our CFM objective explicitly penalizes mismatched semantic pairs during training, discouraging spatial shortcuts and over-smoothed predictions while improving conditional semantic alignment. Furthermore, to overcome architectural bottlenecks, we introduce the Impulse-Trend Synergistic Attention (ITSA) module. Grounded in the key insight of decoupling "what moves" from "when it moves," ITSA models sharp, speech-driven rhythmic impulses separately from continuous kinematic trends, supporting precise audio-motion synchronization and global structural stability. Extensive qualitative and quantitative evaluations on two public datasets demonstrate that SynerGest achieves state-of-the-art performance, delivering contextually responsive and kinematically fluid dyadic interactions.
Figures & tables
Figure 1. Overview of SynerGest . ITSA selects rhythmic motion cues, while CFM separates mismatched audio–motion pairs to preserve synchrony and motion diversity. A conceptual diagram of the SynerGest framework divided into left and right panels. The left panel shows 'Dyadic Audio Extraction' and 'ITSA' for spatial and temporal gating. The right panel shows 'CFM Generation' where a ghosted 'Mismatched Sample' is pushed away to generate highly synchronized active gestures and suppress resting trends.
Figure 2. Architecture of SynerGest : (a) RVQ-based guide key-frame generation, (b) the ITSA-equipped dyadic flow estimator, and (c) audio–lip and audio–pose condition fusion. A detailed three-panel architecture diagram of the SynerGest framework featuring dyadic-conditioned flow matching.
Figure 3. ITSA decouples trend-feature and impulse-temporal gating, then adaptively fuses both branches with global multi-head attention. A block diagram showing the architecture of the Impulse-Trend Synergistic Attention module. It details the parallel Trend Feature Gating and Impulse Temporal Gating branches, followed by an MHA block and a Soft Mix addition operation to produce the final output.
Figure 4. Conceptual audio–motion CFM at d=0 : x1−x0 is the matched target direction, and the loss contrasts matched and shuffled-condition velocity predictions. A contrastive flow matching diagram comparing velocity fields conditioned on matched and mismatched audio-motion pairs. Red arrows indicate the repulsion applied to mismatched semantic conditions.
Dataset
Method
FGD ↓
BC ↑
Div ↑
Tinf (s) ↓
Audio2Photoreal
Ground Truth
–
0.703
12.15
–
EMAGE ( Liu and others, 2024 )
0.908
0.725
12.81
0.332
GestureLSM ( Liu et al., 2025b )
0.951
0.721
16.51
0.116
TalkSHOW ( Yi and others, 2023 )
3.824
0.674
13.97
0.410
Audio2Photoreal (A2P) ( Ng and others, 2024 )
2.743
0.623
13.32
0.378
Ours
0.741
0.782
14.14
0.127
Table 1. Quantitative comparison on Audio2Photoreal and Seamless Interaction. Tinf is the average inference time per sample; best generated results are bold.
Figure 5. Qualitative comparison with baselines and GT. Red highlights show that SynerGest captures semantically aligned gestures while avoiding mean-pose regression and kinematic jitter. Frame sequences comparing ground truth, A2P, GestureLSM, and SynerGest. Red annotations highlight semantically aligned gestures and visible differences in pose quality and temporal stability.
Figure 6. User-study MOS with 95% CIs. SynerGest significantly outperforms both baselines. A chart of mean opinion scores with 95 percent confidence intervals for naturalness, audio-motion synchronization, and overall quality across SynerGest and baseline methods.
Dataset
Model Variant
FGD ↓
BC ↑
Div ↑
Audio2Photoreal
Full Model (Ours)
0.741
0.782
14.14
w/ Diffusion (replace FM)
2.170
0.639
13.38
w/o Contrastive Loss (Vanilla FM)
1.440
0.693
13.80
w/o ITSA (use standard MHA)
1.334
0.712
17.78
Seamless Interaction
Full Model (Ours)
2.380
0.753
13.50
w/ Diffusion (replace FM)
3.219
0.726
7.026
Table 2. Component ablations on Audio2Photoreal and Seamless Interaction.
Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures. This limits semantic grounding, speech-motion alignment, and kinematic smoothness. We propose DuoGesture, a neuro-inspired and biomechanically informed approach that decomposes co-speech gesture synthesis into semantic and beat streams. The two streams are coordinated by a Semantic Variational Information Bottleneck, a stochastic frame-level gate that learns when semantic gestures should override rhythmic beat motion. The semantic stream is controlled by Motion-Grounded Semantic Conditioning, which replaces purely linguistic word embeddings with motion-language representations to provide motion-aligned semantic priors for long-tailed lexical triggers of gestures. The beat stream is further regularised by an Inertial Beat Prior, an anthropometry-weighted arm-chain module that reduces jitter and improves rhythmic consistency without constraining semantic frames. Objective evaluations and subjective experiments show that DuoGesture outperforms strong baselines, while component ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.
Ferdinand Paar, Lanmiao Liu, Aslı Özyürek +2
Radboud University, Nijmegen · Max Planck Institute for Psycholinguistics · Utrecht University
Generating realistic conversational gestures are essential for achieving natural, socially engaging interactions with digital humans. However, existing methods typically map a single audio stream to a single speaker's motion, without considering social context or modeling the mutual dynamics between two people engaging in conversation. We present DyaDiT, a multi-modal diffusion transformer that generates contextually appropriate human motion from dyadic audio signals. Trained on Seamless Interaction Dataset, DyaDiT takes dyadic audio with optional social-context tokens to produce context-appropriate motion. It fuses information from both speakers to capture interaction dynamics, uses a motion dictionary to encode motion priors, and can optionally utilize the conversational partner's gestures to produce more responsive motion. We evaluate DyaDiT on standard motion generation metrics and conduct quantitative user studies, demonstrating that it not only surpasses existing methods on objective metrics but is also strongly preferred by users, highlighting its robustness and socially favorable motion generation. Code and models will be released upon acceptance.
Yichen Peng, Jyun-Ting Song, Siyeol Jung +7
Institute of Science Tokyo · Carnegie Mellon Unversity · UNIST +2
We propose LiveGesture, the first fully streamable, speech-driven full-body gesture generation framework that operates with zero look-ahead and supports arbitrary sequence length. Unlike existing co-speech gesture methods, which are designed for offline generation and either treat body regions independently or entangle all joints within a single model, LiveGesture is built from the ground up for causal, region-coordinated motion generation. LiveGesture consists of two main modules: the Streamable Vector Quantized Motion Tokenizer (SVQ) and the Hierarchical Autoregressive Transformer (HAR). The SVQ tokenizer converts the motion sequence of each body region into causal, discrete motion tokens, enabling real-time, streamable token decoding. On top of SVQ, HAR employs region-expert autoregressive (xAR) transformers to model expressive, fine-grained motion dynamics for each body region. A causal spatio-temporal fusion module (xAR Fusion) then captures and integrates correlated motion dynamics across regions. Both xAR and xAR Fusion are conditioned on live, continuously arriving audio signals encoded by a streamable causal audio encoder. To enhance robustness under streaming noise and prediction errors, we introduce autoregressive masking training, which leverages uncertainty-guided token masking and random region masking to expose the model to imperfect, partially erroneous histories during training. Experiments on the BEAT2 dataset demonstrate that LiveGesture produces coherent, diverse, and beat-synchronous full-body gestures in real time, matching or surpassing state-of-the-art offline methods under true zero look-ahead conditions.
Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyoanuntapong +6
University of North Carolina · University of Central Florida