SynerGest: Temporal-Spatial Synergistic Gesture Generation for Dyadic Conversations

Chengxiang Guo; Jian Wang; Lanmiao Liu; Zhan Tang; Junhua Fei; Xiao Li

cs.CVOct 16, 2026

SynerGest: Temporal-Spatial Synergistic Gesture Generation for Dyadic Conversations

Authors: Chengxiang Guo, Jian Wang, Lanmiao Liu, Zhan Tang, Junhua Fei, Xiao Li

Organizations: Faculty of Information Engineering and Automation Kunming University of Science and Technology Kunming, China · Yunnan Key Laboratory of Artificial Intelligence · Utrecht University Utrecht, Netherlands · The Max Planck Institute for Psycholinguistics Nijmegen, Netherlands ·

Abstract

Co-speech gesture generation in dyadic conversations for creating immersive virtual avatars remains highly challenging. Existing models often struggle to balance kinematic stability with rhythmic responsiveness and suffer from semantic entanglement, leading to either generic mean-pose regression or unnatural micro-jitters. To address these challenges, we propose SynerGest, a high-fidelity dyadic gesture generation framework grounded in Contrastive Flow Matching (CFM). Specifically, our CFM objective explicitly penalizes mismatched semantic pairs during training, discouraging spatial shortcuts and over-smoothed predictions while improving conditional semantic alignment. Furthermore, to overcome architectural bottlenecks, we introduce the Impulse-Trend Synergistic Attention (ITSA) module. Grounded in the key insight of decoupling "what moves" from "when it moves," ITSA models sharp, speech-driven rhythmic impulses separately from continuous kinematic trends, supporting precise audio-motion synchronization and global structural stability. Extensive qualitative and quantitative evaluations on two public datasets demonstrate that SynerGest achieves state-of-the-art performance, delivering contextually responsive and kinematically fluid dyadic interactions.

Figures & tables

Figure 1. Overview of SynerGest . ITSA selects rhythmic motion cues, while CFM separates mismatched audio–motion pairs to preserve synchrony and motion diversity. A conceptual diagram of the SynerGest framework divided into left and right panels. The left panel shows 'Dyadic Audio Extraction' and 'ITSA' for spatial and temporal gating. The right panel shows 'CFM Generation' where a ghosted 'Mismatched Sample' is pushed away to generate highly synchronized active gestures and suppress resting trends.

Figure 2. Architecture of SynerGest : (a) RVQ-based guide key-frame generation, (b) the ITSA-equipped dyadic flow estimator, and (c) audio–lip and audio–pose condition fusion. A detailed three-panel architecture diagram of the SynerGest framework featuring dyadic-conditioned flow matching.

Figure 3. ITSA decouples trend-feature and impulse-temporal gating, then adaptively fuses both branches with global multi-head attention. A block diagram showing the architecture of the Impulse-Trend Synergistic Attention module. It details the parallel Trend Feature Gating and Impulse Temporal Gating branches, followed by an MHA block and a Soft Mix addition operation to produce the final output.

Figure 4. Conceptual audio–motion CFM at $d=0$ : $x_{1}-x_{0}$ is the matched target direction, and the loss contrasts matched and shuffled-condition velocity predictions. A contrastive flow matching diagram comparing velocity fields conditioned on matched and mismatched audio-motion pairs. Red arrows indicate the repulsion applied to mismatched semantic conditions.

Dataset	Method	FGD $\downarrow$	BC $\uparrow$	Div $\uparrow$	$\mathcal{T}_{\text{inf}}$ (s) $\downarrow$
Audio2Photoreal	Ground Truth	–	0.703	12.15	–
	EMAGE ( Liu and others, 2024 )	0.908	0.725	12.81	0.332
	GestureLSM ( Liu et al., 2025b )	0.951	0.721	16.51	0.116
	TalkSHOW ( Yi and others, 2023 )	3.824	0.674	13.97	0.410
	Audio2Photoreal (A2P) ( Ng and others, 2024 )	2.743	0.623	13.32	0.378
	Ours	0.741	0.782	14.14	0.127

Table 1. Quantitative comparison on Audio2Photoreal and Seamless Interaction.

\mathcal{T}_{\text{inf}}

is the average inference time per sample; best generated results are bold.

Figure 5. Qualitative comparison with baselines and GT. Red highlights show that SynerGest captures semantically aligned gestures while avoiding mean-pose regression and kinematic jitter. Frame sequences comparing ground truth, A2P, GestureLSM, and SynerGest. Red annotations highlight semantically aligned gestures and visible differences in pose quality and temporal stability.

Figure 6. User-study MOS with 95% CIs. SynerGest significantly outperforms both baselines. A chart of mean opinion scores with 95 percent confidence intervals for naturalness, audio-motion synchronization, and overall quality across SynerGest and baseline methods.

Dataset	Model Variant	FGD $\downarrow$	BC $\uparrow$	Div $\uparrow$
Audio2Photoreal	Full Model (Ours)	0.741	0.782	14.14
	w/ Diffusion (replace FM)	2.170	0.639	13.38
	w/o Contrastive Loss (Vanilla FM)	1.440	0.693	13.80
	w/o ITSA (use standard MHA)	1.334	0.712	17.78
Seamless Interaction	Full Model (Ours)	2.380	0.753	13.50
	w/ Diffusion (replace FM)	3.219	0.726	7.026

Table 2. Component ablations on Audio2Photoreal and Seamless Interaction.

Contrastive Flow Matching

See all 9 papers

DuoGesture: Motion-Grounded Semantic Conditioning and Biomechanical Beat Priors for Co-Speech Gesture Generation
May 25, 2026Ferdinand Paar, Lanmiao Liu, Aslı Özyürek +2Multimodal Grounding Human Motion Generation
Radboud University, Nijmegen · Max Planck Institute for Psycholinguistics · Utrecht University
DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation
Feb 26, 2026Yichen Peng, Jyun-Ting Song, Siyeol Jung +7Human Motion Generation Co-Speech Gesture Generation
Institute of Science Tokyo · Carnegie Mellon Unversity · UNIST +2
LiveGesture Streamable Co-Speech Gesture Generation Model
Apr 13, 2026Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyoanuntapong +6Human Motion Generation Co-Speech Gesture Generation
University of North Carolina · University of Central Florida

May 25, 2026J/K move · Enter open · S save

DuoGesture: Motion-Grounded Semantic Conditioning and Biomechanical Beat Priors for Co-Speech Gesture Generation

Ferdinand Paar, Lanmiao Liu, Aslı Özyürek, Serge Thill+1

Radboud University, Nijmegen · Max Planck Institute for Psycholinguistics · Utrecht University

Multimodal Grounding Human Motion Generation Co-Speech Gesture Generation

Open paper AskPDF arXiv

Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures. This limits semantic grounding, speech-motion alignment, and kinematic smoothness. We propose DuoGesture, a neuro-inspired and biomechanically informed approach that decomposes co-speech gesture synthesis into semantic and beat streams. The two streams are coordinated by a Semantic Variational Information Bottleneck, a stochastic frame-level gate that learns when semantic gestures should override rhythmic beat motion. The semantic stream is controlled by Motion-Grounded Semantic Conditioning, which replaces purely linguistic word embeddings with motion-language representations to provide motion-aligned semantic priors for long-tailed lexical triggers of gestures. The beat stream is further regularised by an Inertial Beat Prior, an anthropometry-weighted arm-chain module that reduces jitter and improves rhythmic consistency without constraining semantic frames. Objective evaluations and subjective experiments show that DuoGesture outperforms strong baselines, while component ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.

Abstract

Contrastive Flow Matching

Explore similar work

DuoGesture: Motion-Grounded Semantic Conditioning and Biomechanical Beat Priors for Co-Speech Gesture Generation

DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation

LiveGesture Streamable Co-Speech Gesture Generation Model