Human communication involves highly coordinated multimodal signals such as speech and co-speech gestures. Existing approaches for generating both modalities commonly follow a cascaded pipeline, where speech is generated first and subsequently used to drive gesture generation; speech generation is therefore independent of gestures. We introduce Gaspard, a unified speech--gesture generation model based on continuous autoregressive generation with next-step flow-matching predictions, and trained on Seamless Interaction, a large-scale multimodal dataset. The heterogeneous temporal properties of speech and gesture motivate modality-specific prediction heads and decoupled classifier-free guidance over the two generation streams. This guidance rebalances the influence of speech and gesture at inference time, improving expressivity while mitigating dynamics collapse in continuous autoregressive gesture generation. Objective and perceptual evaluations show that Gaspard is a strong gesture generation model against unimodal baselines and that it produces highly synchronous speech and gestures compared to a matched cascaded system.
Figures & tables
Figure 1: Gaspard generates speech and gesture continuous embeddings from text using modality-specific flow-matching heads. Speech and gesture embeddings are fused additively after modality-specific projection. At inference, a short reference prefix is provided to condition the generation, allowing both voice and gesturing style cloning.
Figure 2: Top and middle : PCA views of gesture and speech latent sequences at 12.5 Hz. Gesture and speech embedding sequences exhibit different properties. Gesture latent sequences (top) are smooth and slow-evolving, as demonstrated by repetitive patterns. In contrast, the speech latent sequence is fast-varying and discontinuous. Bottom: Latent velocity of speech and gesture sequences. Velocity is computed as the difference between consecutive steps: vs∣g=∣zt+1s∣g−zts∣g∣ . Speech latent velocity (green) is much higher and more stable than the gesture latent velocity (blue), illustrating the different nature of both signals.
Figure 3: Overview of our decoupled CFG strategy. Next-step speech and gesture latents are sampled conditioned on the context vector ct predicted from the gesture and speech fused history, z<tg and z<ts
Method
FGD ↓
FGDk↓
BC ∼
Div. ∼
Human
–
–
0.63
26.15
DyaSync-S
31.00±0.12
3.40±0.04
0.82
18.46±0.05
GestFlow
8.84±0.35
1.49±0.08
0.91
19.31±0.17
UM-FERI
11.01±0.14
1.02±0.04
0.74
25.33±0.14
UNICAMP
11.93±0.78
2.04±0.23
0.86
16.35±0.41
Gaspard (ours)
6.56±0.16
1.35±0.21
0.67
21.65±0.54
Table 1: Evaluation results. Best results are shown in bold and second-best results are underlined. For BC, performance is ranked by proximity to the ground-truth value. ± indicates 95% confidence intervals.
Method
WER (%) ↓
UTMOS ↑
SS ↑
Human
–
2.88±0.05
–
PocketTTS
4.69±1.15
3.65±0.05
0.70±0.015
Gaspard (ours)
6.38±1.35
2.69±0.05
0.77±0.015
Table 2: Speech generation results on the Genea-95 subset. WER denotes word error rate, UTMOS the predicted naturalness score, and SS the speaker similarity. Best results among generated samples are shown in bold. ± indicates 95% confidence intervals.
Figure 4: Results of the user study across the six conditions, reported as MOS with 95% confidence intervals. Horizontal brackets indicate statistically significant differences ( p<0.05 ). Left : speech–gesture synchrony. Right : gesture human-likeness. We only indicate significant differences w.r.t Gaspard for clarity.
Configuration
FGD ↓
FGDk↓
BC ∼
Div. ∼
WER ↓
UTMOS ↑
SS ↑
RTF ↓
Human
–
–
0.478
24.59
–
2.21±0.16
–
–
No CFG
8.77±2.06
12.32±1.43
0.046±0.023
11.41±1.77
29.51±1.44
1.76±0.02
0.73±0.09
0.78
Standard CFG
4.34±1.18
0.81±0.43
0.761±0.05
22.02±0.83
21.57±1.02
1.91±0.39
0.74±0.01
0.78
Decoupled CFG
3.71±1.71
0.42±0.33
0.657±0.043
22.00±1.08
8.03±0.75
2.62±0.03
0.75±0.01
0.84
Cascaded
3.20±2.60
1.16±0.05
0.74±0.042
22.85±0.73
7.73±0.71
2.68±0.03
0.77±0.01
1.04
Table 3: Comparison on Seamless-416 of no CFG, standard CFG ( γS=1.3 , γG=0.75 ), decoupled CFG ( γsS=γgS=1.3 , γgG=0.825 , γsG=6.5 ), and Cascaded ( γS=1.3 , γgG=0.825 , γsG=6.5 ). ± : 95% CI.
Figure 6: Effect of classifier-free guidance on generated speech. Top: speech-history guidance γsS . Bottom: gesture-history guidance γgS .
Figure 7: Impact of the gesture-history guidance scale γgG on the dynamics of the generated gestures.
Figure 8: Impact of the speech guidance scale γsG on the dynamics of the generated gestures.
Figure 9: User study results (MOS, 95% CI). Brackets: significant differences ( p<0.05 ). Left : voice human-likeness. Middle : speech–gesture synchrony. Right : gesture human-likeness.
Part
Split
Hours
Interactions
Participants
Naturalistic
Train
2625.34
45,126
3,243
Dev
35.22
673
83
Test
44.13
786
109
Improvised
Train
1268.01
16,623
897
Dev
20.98
323
41
Test
15.31
220
35
Table 4: Statistics of the Seamless Interaction dataset by part and split.
Modality
Stage
Module
Frame rate
Latent dim.
Speech
Input waveform
N/A
24 kHz
N/A
Stage 1
WavVAE
100 Hz
24
Stage 2
PatchVAE
12.5 Hz
32
Gesture
Input sequence
Rot6D
25 fps
N/A
Encoding
σ -VAE
12.5 Hz
64
Both
Projection
ϕins / ϕing
12.5 Hz
32/64→1024
Table 5: Frame rates and dimensionality reduction across the speech and gesture pipelines used by Gaspard.
Head
Coefficient
Interpretation
Speech head hθS
γsS
Controls the strength of speech conditioning when predicting the next-step speech embedding.
γgS
Controls the strength of gesture conditioning when predicting the next-step speech embedding.
Gesture head hθG
γsG
Controls the strength of speech conditioning when predicting the next-step gesture embedding.
γgG
Controls the strength of gesture conditioning when predicting the next-step gesture embedding.
Table 6: Interpretation of the modality-specific CFG coefficients.
Figure 10: Simplified overview of the cascaded TTS and speech-driven gesture generation baseline. The TTS module first generates speech conditioned on text, after which the resulting speech embeddings are provided as input to the gesture generation module. Speech synthesis is therefore performed independently of gesture generation, as in most conventional cascaded systems.
Figure 11: We validate the FGDk metric by varying gesturing velocity ( Top ), smoothness ( Middle ) and jerk ( Bottom ). We also include the BC score corresponding to all three perturbations. FGD shows little sensitivity to the perturbations, whereas FGDk captures velocity, smoothness and jerk variations.
Figure 12: Quality–efficiency trade-off with respect to the number of flow-matching sampling steps. Increasing the number of sampling steps consistently improves generation performance at the cost of lower inference speed.
γG
FGD ↓
FGDk↓
BC ∼
Div. ∼
0.75
4.34 ± 1.18
0.81 ± 0.43
0.761 ± 0.05
22.02 ± 0.83
0.85
2.96 ± 0.26
3.14 ± 0.25
0.361 ± 0.02
15.93 ± 0.86
1.00
8.63 ± 0.66
12.53 ± 0.18
0.050 ± 0.01
11.29 ± 0.84
1.50
11.41 ± 0.18
11.72 ± 0.21
0.127 ± 0.02
15.11 ± 0.26
2.50
10.52 ± 0.17
11.00 ± 0.54
0.156 ± 0.02
12.97 ± 0.58
Table 7: Impact of the standard classifier-free guidance scale on gesture generation performance.
γS
WER ↓
UTMOS ↑
SS ↑
1.0
21.04 ± 0.98
1.91 ± 0.39
0.74 ± 0.01
1.1
21.55 ± 1.22
1.91 ± 0.02
0.74 ± 0.01
1.2
21.56 ± 1.21
1.90 ± 0.02
0.74 ± 0.01
1.3
21.57 ± 1.02
1.91 ± 0.02
0.74 ± 0.01
Table 8: Impact of the standard classifier-free speech guidance scale γS on speech generation performance, with γG=0.75 fixed.
Figure 13: Impact of gesture speed on speech generation. Speech is generated while conditioning the model on fixed gesture inputs with varying speeds. Increasing gesture speed results in a higher speech rate and fewer IPUs.
Sample ID
Speaker
Speech rate
Number of IPUs
Spearman ρ
Spearman ρ
1
0093
1.000
−0.800
2
0093
0.200
−0.718
3
0093
0.200
−0.975
4
0093
−0.100
0.700
5
0093
0.600
−0.600
Table 9: Spearman correlations between gesture speed multiplier and generated speech characteristics for each sample.
Figure 14: Example wrist trajectories for three stimuli used in the Seamless-416 user study with different CFG configurations. Left: Human recording. Middle: Gesture sequence generated by setting γgG=1 , γsG=2.5 . Right: Gesture sequence generated by setting γgG=0.825 , γsG=6.5 . Reducing γgG while increasing γsG increases the amplitude and variability of the generated gestures.
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.