Human communication seamlessly integrates speech and bodily motion, where hand gestures naturally complement vocal prosody to express intent, emotion, and emphasis. While recent text-to-speech (TTS) systems have begun incorporating multimodal cues such as facial expressions or lip movements, the role of hand gestures in shaping prosody remains largely underexplored. We propose a novel multimodal TTS framework, Gesture2Speech, that leverages visual gesture cues to modulate prosody in synthesized speech. Motivated by the observation that confident and expressive speakers coordinate gestures with vocal prosody, we introduce a multimodal Mixture-of-Experts (MoE) architecture that dynamically fuses linguistic content and gesture features within a dedicated style extraction module. The fused representation conditions an LLM-based speech decoder, enabling prosodic modulation that is temporally aligned with hand movements. We further design a gesture-speech alignment loss that explicitly models their temporal correspondence to ensure fine-grained synchrony between gestures and prosodic contours. Evaluations on the PATS dataset show that Gesture2Speech outperforms state-of-the-art baselines in both speech naturalness and gesture-speech synchrony. To the best of our knowledge, this is the first work to utilize hand gesture cues for prosody control in neural speech synthesis. Demo samples are available at https://research.sri-media-analysis.com/aaai26-beeu-gesture2speech/
Figures & tables
Figure 1: Overview of (a) the proposed Gesture2Speech architecture and (b) MoE layer. The system takes text, speech, and video-based gesture features as input and generates expressive speech. Multiple MoE modules enable dynamic routing of features for improved style representation and are aligned with gestural intent via cross-attention.
Figure 2: Comparison of gesture apexes and speech prominence peaks between ground truth and TTS-generated speech. The Cross-Modal Temporal Distance (CMTD) quantifies alignment between gestures and prosodic peaks. Ground truth CMTD: 0.819 seconds, indicating looser temporal alignment, while TTS CMTD: 0.727 seconds suggests improved synchronization with gestures.
Method
Gesture Offset ↓
Mutual Info ↑
WER ↓
CER ↓
UTMOS ↑
WVMOS ↑
AutoPCP ↑
Same Text
Ground Truth
1.0198
0.0362
35.61
25.20
3.34 ± 0.16
3.32 ± 0.23
–
Gesture2Speech (XTTS V2)
1.0386
0.0382
20.27
14.85
3.34 ± 0.11
3.34 ± 0.25
3.08 ± 0.14
Gesture2Speech (GPT-SoVITS)
1.8656
0.0070
34.04
26.00
3.17 ± 0.67
3.51 ± 0.66
3.14 ± 0.48
Gesture2Speech (unimodal MoE)
0.9794
0.0404
22.42
15.20
3.44 ± 0.11
3.45 ± 0.23
3.12 ± 0.10
Gesture2Speech (H-MoE)
1.2008
0.0357
16.93
11.74
3.46 ± 0.12
3.36 ± 0.34
3.12 ± 0.10
Table 1: Objective Evaluations on PATS test set. UTMOS, WVMOS, and AutoPCP are reported with 95% confidence intervals.
Gesture2Speech
Metric
XTTS v2
GPT-SoVITS
Unimodal MoE
H-MoE
Multimodal MoE
Speech Quality ↑
75.79±2.39
70.78±2.87
72.22±2.62
73.55±2.63
81.48±2.25
Prosodic Similarity ↑
72.78±2.44
67.89±3.20
71.59±2.45
71.54±2.91
79.35±2.52
Table 2: Subjective Evaluation on Speech Quality and Prosodic Similarity of Gesture2Speech Variants along with a margin of error corresponding to the 95% confidence interval.
Method
Gesture Offset ↓
Mutual Info ↑
WER ↓
CER ↓
UTMOS ↑
WVMOS ↑
AutoPCP ↑
Same Text
Gesture2Speech (Speech-Unimodal MoE)
0.9663
0.0424
20.78
15.64
3.40 ± 0.11
3.35 ± 0.27
3.16 ± 0.12
Gesture2Speech (Video-Unimodal MoE)
1.0324
0.0191
31.43
25.01
3.41 ± 0.11
3.48 ± 0.26
3.12 ± 0.06
Gesture2Speech (Multimodal MoE)
0.9471
0.0559
17.55
12.14
3.70 ± 0.09
3.65 ± 0.16
3.19 ± 0.06
Different Text
Gesture2Speech (Speech-Unimodal MoE)
2.2088
0.0340
26.87
15.93
3.47 ± 0.11
3.27 ± 0.24
2.71 ± 0.10
Table 3: Ablation Evaluations using different MoE configurations. UTMOS, WVMOS, and AutoPCP are reported with a margin of error corresponding to the 95% confidence intervals.
Method
Gesture Offset ↓
Mutual Info ↑
UTMOS ↑
WVMOS ↑
Cross-Attention
0.8410
0.0223
3.36 ± 0.25
3.42 ± 0.33
Concatenation
1.0295
0.0134
3.04 ± 0.36
3.32 ± 0.46
MoE Fusion
0.7576
0.0606
3.64 ± 0.22
3.67 ± 0.30
Table 4: Ablation with respect to Fusion Strategies.
Figure 3: Speaker-wise t-SNE plots analysis of proposed style transfer network. The t-SNE plot of (a) Speech MoE embeddings, (b) video MoE embeddings and (c) Multimodal MoE embeddings.
Recent Text-To-Speech (TTS) systems have achieved strong naturalness and zero-shot voice cloning performance, but fine-grained control of expressive speech at the word or phoneme level remains challenging. We propose CtrlSpeech, a controllable, expressive TTS framework with coarse-to-fine control. Built on the DiTAR architecture, CtrlSpeech combines global speaker conditioning with phone-aligned pitch, loudness, and duration signals, enabling localized prosodic control while preserving the target speaker's timbre. This design allows users to adjust expressive attributes at a fine temporal granularity, making speech refinement more flexible and controllable. Experimental results show that CtrlSpeech achieves competitive zero-shot TTS performance and improves controllability over expressive attributes, demonstrating its effectiveness for flexible and practical expressive speech synthesis.
Personalized text-to-speech (TTS) aims to clone the target speaker in the synthesized speech, imitating both the voice and speaking style. Current large language model (LLM)-based TTS methods ignore the style-specific prosodic patterns in generated speech, resulting in deficient style learning and thus limiting speaker similarity in synthesized speech. To this end, we investigate the prosody learning conditioned on the synthesized speech, and propose to predict the prosody of the current syllable based on previously predicted speech. Experimental results obtained on three datasets demonstrated the efficacy of the proposed dynamic prosody prediction method in enhancing the prosody learning capability, thereby improving the speaker similarity of the generated speech. Audio samples are available at https://muzw.github.io/dynapros/.
Zhenwei Mou, Liping Chen, Yajun Hu +3
University of Science and Technology of China, Anhui, China · iFLYTEK, Anhui, China
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient.
Wenqian Cui, Xiao-Hui Li, Daxin Tan +2
The Chinese University of Hong Kong · Huawei Technologies