Organizations: Electronic Information School, Wuhan University, Wuhan, China · Microsoft Corporation, Redmond, US · The Chinese University of Hong Kong, Hong Kong, China
Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2% to 2.7% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.
Figures & tables
Figure 1: Overview of ImageTTS. (a) FLUX.2 is pretrained on images and provides the DiT initialization for speech generation. (b) Log-mel spectrograms are converted into pseudo-RGB inputs; the image VAE and Qwen3 text encoder remain frozen, while the DiT is fine-tuned for speech. (c) Three findings: the image VAE supports high-fidelity speech reconstruction, image-pretrained initialization accelerates convergence and improves WER over random initialization, and the framework supports zero-shot TTS and instruction-guided TTS.
Figure 2: Architecture of ImageTTS-base. The framework adopts a Flow-Matching DiT backbone comprising double-stream blocks and single-stream blocks conditioned on timestep σ . High-fidelity speech is synthesized via Euler integration, a frozen FLUX.2 VAE decoder, and BigVGAN-v2.
Method
LibriTTS (24 kHz)
LibriSpeech (16 kHz)
PESQ ↑
STOI ↑
SIM ↑
DNSMOS ↑
UTMOS ↑
PESQ ↑
STOI ↑
SIM ↑
DNSMOS ↑
UTMOS ↑
Original waveform
4.64
1.00
1.00
3.74
4.01
4.64
1.00
1.00
3.93
4.13
Vocoder only
4.43
1.00
0.98
3.75
3.99
4.36
0.99
0.98
3.89
3.99
Riffusion-o
1.40
0.82
0.72
3.34
1.40
1.42
0.81
0.79
3.46
1.33
Riffusion-n , F=512
2.45
0.92
0.86
3.56
2.41
2.51
0.92
0.89
3.67
2.28
FLUX.2, F=96
2.67
0.94
0.90
3.50
2.66
2.61
0.94
0.91
3.57
2.51
Table 1: Speech reconstruction using different frozen image VAEs. The best results among the evaluated VAE configurations are shown in bold .
Figure 3: Training loss on LibriTTS for image-pretrained and randomly initialized DiTs operating in the frozen image-VAE latent space. Curves show the batch-averaged training loss at each update, smoothed with a 500-update moving-average window in the main panel. The inset enlarges the low-loss region between 0.4 and 0.6 and applies a 10,000-update smoothing window to highlight the long-term trend.
Figure 4: Generation metrics across training stages for image-pretrained and randomly initialized DiTs in the image-VAE latent space. Classifier-free guidance is fixed to 4. Markers denote evaluated checkpoints; connecting lines are visual guides. WER is reported as a percentage.
Image-pretrained
Random initialization
Generation space
WER (%) ↓
SIM ↑
DNSMOS ↑
WER (%) ↓
SIM ↑
DNSMOS ↑
Image-VAE latent
2.7
0.78
3.78
4.2
0.77
3.72
Direct mel
3.1
0.80
3.56
4.2
0.80
3.51
Table 2: Effects of initialization and generation space at 100k training updates. Bold indicates the best result across all models.
Method
Training hours
Seed-TTS test-en
Seed-TTS test-zh
Internal test-en
WER ↓
SIM ↑
MOS ↑
CER ↓
SIM ↑
MOS ↑
WER ↓
SIM ↑
Ground Truth
–
0.88
0.73
4.17±0.19
1.25
0.76
4.25±0.16
–
–
Reconstructed
–
0.90
0.71
–
1.34
0.73
–
–
–
Qwen3-TTS ( Hu et al., 2026 )
> 5M
0.96
0.72
4.06±0.16
1.23
0.77
3.91±0.19
0.72
0.69
FireRedTTS3-Instruct ( Shen et al., 2026 )
> 2.6M
1.55
0.64
4.04±0.18
1.09
0.71
3.67±0.20
0.84
0.59
VoxCPM2 ( Zhou et al., 2026 )
> 2M
1.20
0.68
3.90±0.19
1.04
0.74
4.24±0.17
0.51
0.55
Table 3: Zero-shot TTS results on Seed-TTS test-en, Seed-TTS test-zh, and the internal English test set. MOS is reported as the mean score with a confidence interval (CI). The best and second-best results among models trained on the same source dataset (Emilia) are shown in bold and underlined , respectively; ties receive the same marking. Missing results are denoted by –.
Figure 5: Effects of text and speech CFG scales. Panels (a) and (b) report WER and SIM, respectively, over a grid search of the text and speech CFG scales st and ss . Stars indicate the configurations with the lowest WER and highest SIM, respectively. Panel (c) plots WER as a function of st with ss=1.8 , and SIM as a function of ss with st=2.0 .
Method
Instruction data
Age ↑
Gender ↑
Pitch ↑
Expressiveness ↑
Style-ACC ↑
CLSP-Score ↑
WER ↓
PromptTTS
∼ 0.3K h
0.48
0.59
0.12
0.09
0.32
0.45
6.3
VoxInstruct
2.6K h
0.76
0.99
0.23
0.41
0.60
0.50
8.6
FireRedTTS3-Instruct
330K h
0.56
0.92
0.22
0.47
0.54
0.48
6.7
Qwen3-TTS-VD
N/A
0.66
0.90
0.18
0.42
0.54
0.50
6.1
VoxCPM2
∼ 10K h
0.87
0.97
0.46
0.48
0.69
0.48
5.7
CapTTS-AR
33.6K h
0.55
0.98
0.18
0.36
0.52
0.45
11.9
Table 4: Instruction-guided TTS results on the CapSpeech test set. The best and second-best results among models trained on CapSpeech instruction data are shown in bold and underlined , respectively. N/A indicates that the amount of instruction data is not available.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Input: a reference speech latent zref , text condition y , and the requested total target width Lreq , which covers time columns of both the reference speech ℓref and the continuation target.
1
Initialize the complete target latent with standard Gaussian noise: zσ0∼N(0,I) .
2
Construct the full-size reference conditioning stream: c←0 , c[...,:ℓref]←zref .
3
For k=0,…,NFE−1 :
4
Predict the full conditional velocity: vkfull←vθ(Concat(zk,c),σk∣y) .
5
If text guidance is enabled, vk←vkfull+(st−1)(vkfull−vkno-text).
6
If speech guidance is enabled, vk←vk+(ss−1)(vkfull−vkno-speech).
Table 6: Pronunciation sensitivity to punctuation-induced changes in text conditioning. Each case pairs the original input (O) with its punctuation-modified version (M). Metrics are sentence-level rates in percent; the displayed text and Pinyin cover only the target spans.