EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation
Authors: Yejin Jeon, Luel Hagos Beyene, Marie Maltais, Hyounghun Kim, David Ifeoluwa Adelani
Organizations: Mila-Québec AI Institute · McGill University · NM-AIST · AIMS Research and Innovation Center · CSE & GSAI, POSTECH · Canada CIFAR AI Chair
Controlling emotional intensity remains challenging in expressive text-to-speech synthesis, particularly when emotion must be inferred without explicit labels. We propose EmoFI-TTS, a face-conditioned framework that derives emotion category and intensity from a single facial image while extracting speaker characteristics from reference audio. Modality-specific representations are fused to condition a pretrained speech synthesizer, with cross-speaker training encouraging separation of affect and vocal identity. An intensity-conditioned refinement module enhances the generated mel-spectrogram, while interpolation of learned intensity embeddings enables continuous control without additional training. On speaker-disjoint MEAD evaluation, EmoFI-TTS achieves on average 64.4% and 75.2% objective and subjective emotion intensity accuracies, while retaining low word error rate and high naturalness.
Figures & tables
Figure 1: System overview. Text, face image, and reference audio each control content, emotion and its intensity, and speaker timbre, respectively. Solid lines carry signal, blue dashed lines carry conditioning, and the orange arrow is the feature-matching loss against the real mel m⋆ , used only in training. At inference, the emotion intensity level is predicted from the face.
Model
WER
UTMOS
MOS
L1
L2
L3
L1-Sbj
L2-Sbj
L3-Sbj
Real (GT)
6.6
2.86
3.98 ± 0.20
69.7
34.6
80.3
88.7
81.2
87.6
EmoSpeech
9.3
2.88
2.96 ± 0.26
42.6
56.0
18.8
63.5
46.3
33.8
FaceTTS
16.3
2.46
2.65 ± 0.28
43.1
44.9
14.9
61.9
44.0
40.1
Ours
6.9
3.14
3.25 ± 0.26
76.4
42.8
74.1
82.2
69.5
73.9
Table 1: Results across objective and subjective ( Sbj ) metrics. MOS scores are reported with 95% confidence intervals.
Figure 2: Emotion visualizations of generated audio.
Figure 3: Continuous intensity adjustment from mild to strong emotion intensity. Dashed lines indicate real audio’s recall.
System
L1
L2
L3
Overall Acc.
Real audio
69.7
34.6
80.3
61.5
Backbone (no refinement)
79.3
42.1
48.9
56.7
+ refine., face-predicted
76.4
42.8
74.1
64.4
+ refine., oracle label
79.3
45.9
76.5
67.2
Table 2: Ablation of the intensity refinement module.
Fusion
Emo. Acc. ↑
Int. Acc ↑
WER ↓
Spk. Transfer ↑
cross-attn.
71.7
55.0
5.7
90.2
frame cross-attn.
61.6
48.3
8.1
50.2
sum
71.2
55.2
5.8
90.8
concat + linear
70.3
56.7
6.6
92.1
Table 3: Fusion ablation. Speaker transfer is reported in percentages. The adopted methodology is indicated in grey.