Human communication involves highly coordinated multimodal signals such as speech and co-speech gestures. Existing approaches for generating both modalities commonly follow a cascaded pipeline, where speech is generated first and subsequently used to drive gesture generation; speech generation is therefore independent of gestures. We introduce Gaspard, a unified speech--gesture generation model based on continuous autoregressive generation with next-step flow-matching predictions, and trained on Seamless Interaction, a large-scale multimodal dataset. The heterogeneous temporal properties of speech and gesture motivate modality-specific prediction heads and decoupled classifier-free guidance over the two generation streams. This guidance rebalances the influence of speech and gesture at inference time, improving expressivity while mitigating dynamics collapse in continuous autoregressive gesture generation. Objective and perceptual evaluations show that Gaspard is a strong gesture generation model against unimodal baselines and that it produces highly synchronous speech and gestures compared to a matched cascaded system.
Figures & tables
Figure 1: Gaspard generates speech and gesture continuous embeddings from text using modality-specific flow-matching heads. Speech and gesture embeddings are fused additively after modality-specific projection. At inference, a short reference prefix is provided to condition the generation, allowing both voice and gesturing style cloning.
Figure 2: Top and middle : PCA views of gesture and speech latent sequences at 12.5 Hz. Gesture and speech embedding sequences exhibit different properties. Gesture latent sequences (top) are smooth and slow-evolving, as demonstrated by repetitive patterns. In contrast, the speech latent sequence is fast-varying and discontinuous. Bottom: Latent velocity of speech and gesture sequences. Velocity is computed as the difference between consecutive steps: vs∣g=∣zt+1s∣g−zts∣g∣ . Speech latent velocity (green) is much higher and more stable than the gesture latent velocity (blue), illustrating the different nature of both signals.
Figure 3: Overview of our decoupled CFG strategy. Next-step speech and gesture latents are sampled conditioned on the context vector ct predicted from the gesture and speech fused history, z<tg and z<ts
Method
FGD ↓
FGDk↓
BC ∼
Div. ∼
Human
–
–
0.63
26.15
DyaSync-S
31.00±0.12
3.40±0.04
0.82
18.46±0.05
GestFlow
8.84±0.35
1.49±0.08
0.91
19.31±0.17
UM-FERI
11.01±0.14
1.02±0.04
0.74
25.33±0.14
UNICAMP
11.93±0.78
2.04±0.23
0.86
16.35±0.41
Gaspard (ours)
6.56±0.16
1.35±0.21
0.67
21.65±0.54
Table 1: Evaluation results. Best results are shown in bold and second-best results are underlined. For BC, performance is ranked by proximity to the ground-truth value. ± indicates 95% confidence intervals.
Method
WER (%) ↓
UTMOS ↑
SS ↑
Human
–
2.88±0.05
–
PocketTTS
4.69±1.15
3.65±0.05
0.70±0.015
Gaspard (ours)
6.38±1.35
2.69±0.05
0.77±0.015
Table 2: Speech generation results on the Genea-95 subset. WER denotes word error rate, UTMOS the predicted naturalness score, and SS the speaker similarity. Best results among generated samples are shown in bold. ± indicates 95% confidence intervals.
Figure 4: Results of the user study across the six conditions, reported as MOS with 95% confidence intervals. Horizontal brackets indicate statistically significant differences ( p<0.05 ). Left : speech–gesture synchrony. Right : gesture human-likeness. We only indicate significant differences w.r.t Gaspard for clarity.
Configuration
FGD ↓
FGDk↓
BC ∼
Div. ∼
WER ↓
UTMOS ↑
SS ↑
RTF ↓
Human
–
–
0.478
24.59
–
2.21±0.16
–
–
No CFG
8.77±2.06
12.32±1.43
0.046±0.023
11.41±1.77
29.51±1.44
1.76±0.02
0.73±0.09
0.78
Standard CFG
4.34±1.18
0.81±0.43
0.761±0.05
22.02±0.83
21.57±1.02
1.91±0.39
0.74±0.01
0.78
Decoupled CFG
3.71±1.71
0.42±0.33
0.657±0.043
22.00±1.08
8.03±0.75
2.62±0.03
0.75±0.01
0.84
Cascaded
3.20±2.60
1.16±0.05
0.74±0.042
22.85±0.73
7.73±0.71
2.68±0.03
0.77±0.01
1.04
Table 3: Comparison on Seamless-416 of no CFG, standard CFG ( γS=1.3 , γG=0.75 ), decoupled CFG ( γsS=γgS=1.3 , γgG=0.825 , γsG=6.5 ), and Cascaded ( γS=1.3 , γgG=0.825 , γsG=6.5 ). ± : 95% CI.
Figure 6: Effect of classifier-free guidance on generated speech. Top: speech-history guidance γsS . Bottom: gesture-history guidance γgS .
Figure 7: Impact of the gesture-history guidance scale γgG on the dynamics of the generated gestures.
Figure 8: Impact of the speech guidance scale γsG on the dynamics of the generated gestures.
Figure 9: User study results (MOS, 95% CI). Brackets: significant differences ( p<0.05 ). Left : voice human-likeness. Middle : speech–gesture synchrony. Right : gesture human-likeness.
Part
Split
Hours
Interactions
Participants
Naturalistic
Train
2625.34
45,126
3,243
Dev
35.22
673
83
Test
44.13
786
109
Improvised
Train
1268.01
16,623
897
Dev
20.98
323
41
Test
15.31
220
35
Table 4: Statistics of the Seamless Interaction dataset by part and split.
Modality
Stage
Module
Frame rate
Latent dim.
Speech
Input waveform
N/A
24 kHz
N/A
Stage 1
WavVAE
100 Hz
24
Stage 2
PatchVAE
12.5 Hz
32
Gesture
Input sequence
Rot6D
25 fps
N/A
Encoding
σ -VAE
12.5 Hz
64
Both
Projection
ϕins / ϕing
12.5 Hz
32/64→1024
Table 5: Frame rates and dimensionality reduction across the speech and gesture pipelines used by Gaspard.
Head
Coefficient
Interpretation
Speech head hθS
γsS
Controls the strength of speech conditioning when predicting the next-step speech embedding.
γgS
Controls the strength of gesture conditioning when predicting the next-step speech embedding.
Gesture head hθG
γsG
Controls the strength of speech conditioning when predicting the next-step gesture embedding.
γgG
Controls the strength of gesture conditioning when predicting the next-step gesture embedding.
Table 6: Interpretation of the modality-specific CFG coefficients.
Figure 10: Simplified overview of the cascaded TTS and speech-driven gesture generation baseline. The TTS module first generates speech conditioned on text, after which the resulting speech embeddings are provided as input to the gesture generation module. Speech synthesis is therefore performed independently of gesture generation, as in most conventional cascaded systems.
Figure 11: We validate the FGDk metric by varying gesturing velocity ( Top ), smoothness ( Middle ) and jerk ( Bottom ). We also include the BC score corresponding to all three perturbations. FGD shows little sensitivity to the perturbations, whereas FGDk captures velocity, smoothness and jerk variations.
Figure 12: Quality–efficiency trade-off with respect to the number of flow-matching sampling steps. Increasing the number of sampling steps consistently improves generation performance at the cost of lower inference speed.
γG
FGD ↓
FGDk↓
BC ∼
Div. ∼
0.75
4.34 ± 1.18
0.81 ± 0.43
0.761 ± 0.05
22.02 ± 0.83
0.85
2.96 ± 0.26
3.14 ± 0.25
0.361 ± 0.02
15.93 ± 0.86
1.00
8.63 ± 0.66
12.53 ± 0.18
0.050 ± 0.01
11.29 ± 0.84
1.50
11.41 ± 0.18
11.72 ± 0.21
0.127 ± 0.02
15.11 ± 0.26
2.50
10.52 ± 0.17
11.00 ± 0.54
0.156 ± 0.02
12.97 ± 0.58
Table 7: Impact of the standard classifier-free guidance scale on gesture generation performance.
γS
WER ↓
UTMOS ↑
SS ↑
1.0
21.04 ± 0.98
1.91 ± 0.39
0.74 ± 0.01
1.1
21.55 ± 1.22
1.91 ± 0.02
0.74 ± 0.01
1.2
21.56 ± 1.21
1.90 ± 0.02
0.74 ± 0.01
1.3
21.57 ± 1.02
1.91 ± 0.02
0.74 ± 0.01
Table 8: Impact of the standard classifier-free speech guidance scale γS on speech generation performance, with γG=0.75 fixed.
Figure 13: Impact of gesture speed on speech generation. Speech is generated while conditioning the model on fixed gesture inputs with varying speeds. Increasing gesture speed results in a higher speech rate and fewer IPUs.
Sample ID
Speaker
Speech rate
Number of IPUs
Spearman ρ
Spearman ρ
1
0093
1.000
−0.800
2
0093
0.200
−0.718
3
0093
0.200
−0.975
4
0093
−0.100
0.700
5
0093
0.600
−0.600
Table 9: Spearman correlations between gesture speed multiplier and generated speech characteristics for each sample.
Figure 14: Example wrist trajectories for three stimuli used in the Seamless-416 user study with different CFG configurations. Left: Human recording. Middle: Gesture sequence generated by setting γgG=1 , γsG=2.5 . Right: Gesture sequence generated by setting γgG=0.825 , γsG=6.5 . Reducing γgG while increasing γsG increases the amplitude and variability of the generated gestures.
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR
Pinxin Liu, Haiyang Liu, Jiahao Luo +3
University of Rochester · University of Tokyo · University of California Santa Cruz +2
Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combines pretrained acoustic features with distinct listening and speaking hidden states from the dialogue backbone and speech synthesizer, enabling continuous non-verbal generation even when the model is verbally silent. To translate dialogue understanding into vivid semantic gestures, we introduce contextual semantic calls and template guidance, separating whether to invoke an available semantic gesture from how to realize it. We also establish a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and investigate reference-based multimodal large language models (MLLMs) as automatic judges. Experiments show that ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models and outperforms cascaded baselines across dialog-aware gesture and expression metrics. Semantic template guidance further significantly improves semantic gesture quality. Strong system-level correlations with human judgments support these reference-based scores as useful preliminary metrics for full-duplex and semantic gesture assessment. Together, these results advance dialogue modeling from spoken responses toward coordinated multimodal interaction.
While the field of co-speech gesture generation has seen significant advances, producing holistic, semantically grounded gestures remains a challenge. Existing approaches rely on external semantic retrieval methods, which limit their generalisation capability due to dependency on predefined linguistic rules. Flow-matching-based methods produce promising results; however, the network is optimised using only semantically congruent samples without exposure to negative examples, leading to learning rhythmic gestures rather than sparse motion, such as iconic and metaphoric gestures. Furthermore, by modelling body parts in isolation, the majority of methods fail to maintain crossmodal consistency. We introduce a Contrastive Flow Matching-based co-speech gesture generation model that uses mismatched audio-text conditions as negatives, training the velocity field to follow the correct motion trajectory while repelling semantically incongruent trajectories. Our model ensures cross-modal coherence by embedding text, audio, and holistic motion into a composite latent space via cosine and contrastive objectives. Extensive experiments and a user study demonstrate that our proposed approach outperforms state-of-the-art methods on two datasets, BEAT2 and SHOW.
Lanmiao Liu, Esam Ghaleb, Aslı Özyürek +1
Max Planck Institute for Psycholinguistics · Donders Institute for Brain Cognition and Behaviour · Utrecht University