Organizations: Electronic Information School, Wuhan University, Wuhan, China · Microsoft Corporation, Redmond, US · The Chinese University of Hong Kong, Hong Kong, China
Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2% to 2.7% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.
Figures & tables
Figure 1: Overview of ImageTTS. (a) FLUX.2 is pretrained on images and provides the DiT initialization for speech generation. (b) Log-mel spectrograms are converted into pseudo-RGB inputs; the image VAE and Qwen3 text encoder remain frozen, while the DiT is fine-tuned for speech. (c) Three findings: the image VAE supports high-fidelity speech reconstruction, image-pretrained initialization accelerates convergence and improves WER over random initialization, and the framework supports zero-shot TTS and instruction-guided TTS.
Figure 2: Architecture of ImageTTS-base. The framework adopts a Flow-Matching DiT backbone comprising double-stream blocks and single-stream blocks conditioned on timestep σ . High-fidelity speech is synthesized via Euler integration, a frozen FLUX.2 VAE decoder, and BigVGAN-v2.
Method
LibriTTS (24 kHz)
LibriSpeech (16 kHz)
PESQ ↑
STOI ↑
SIM ↑
DNSMOS ↑
UTMOS ↑
PESQ ↑
STOI ↑
SIM ↑
DNSMOS ↑
UTMOS ↑
Original waveform
4.64
1.00
1.00
3.74
4.01
4.64
1.00
1.00
3.93
4.13
Vocoder only
4.43
1.00
0.98
3.75
3.99
4.36
0.99
0.98
3.89
3.99
Riffusion-o
1.40
0.82
0.72
3.34
1.40
1.42
0.81
0.79
3.46
1.33
Riffusion-n , F=512
2.45
0.92
0.86
3.56
2.41
2.51
0.92
0.89
3.67
2.28
FLUX.2, F=96
2.67
0.94
0.90
3.50
2.66
2.61
0.94
0.91
3.57
2.51
Table 1: Speech reconstruction using different frozen image VAEs. The best results among the evaluated VAE configurations are shown in bold .
Figure 3: Training loss on LibriTTS for image-pretrained and randomly initialized DiTs operating in the frozen image-VAE latent space. Curves show the batch-averaged training loss at each update, smoothed with a 500-update moving-average window in the main panel. The inset enlarges the low-loss region between 0.4 and 0.6 and applies a 10,000-update smoothing window to highlight the long-term trend.
Figure 4: Generation metrics across training stages for image-pretrained and randomly initialized DiTs in the image-VAE latent space. Classifier-free guidance is fixed to 4. Markers denote evaluated checkpoints; connecting lines are visual guides. WER is reported as a percentage.
Image-pretrained
Random initialization
Generation space
WER (%) ↓
SIM ↑
DNSMOS ↑
WER (%) ↓
SIM ↑
DNSMOS ↑
Image-VAE latent
2.7
0.78
3.78
4.2
0.77
3.72
Direct mel
3.1
0.80
3.56
4.2
0.80
3.51
Table 2: Effects of initialization and generation space at 100k training updates. Bold indicates the best result across all models.
Method
Training hours
Seed-TTS test-en
Seed-TTS test-zh
Internal test-en
WER ↓
SIM ↑
MOS ↑
CER ↓
SIM ↑
MOS ↑
WER ↓
SIM ↑
Ground Truth
–
0.88
0.73
4.17±0.19
1.25
0.76
4.25±0.16
–
–
Reconstructed
–
0.90
0.71
–
1.34
0.73
–
–
–
Qwen3-TTS ( Hu et al., 2026 )
> 5M
0.96
0.72
4.06±0.16
1.23
0.77
3.91±0.19
0.72
0.69
FireRedTTS3-Instruct ( Shen et al., 2026 )
> 2.6M
1.55
0.64
4.04±0.18
1.09
0.71
3.67±0.20
0.84
0.59
VoxCPM2 ( Zhou et al., 2026 )
> 2M
1.20
0.68
3.90±0.19
1.04
0.74
4.24±0.17
0.51
0.55
Table 3: Zero-shot TTS results on Seed-TTS test-en, Seed-TTS test-zh, and the internal English test set. MOS is reported as the mean score with a confidence interval (CI). The best and second-best results among models trained on the same source dataset (Emilia) are shown in bold and underlined , respectively; ties receive the same marking. Missing results are denoted by –.
Figure 5: Effects of text and speech CFG scales. Panels (a) and (b) report WER and SIM, respectively, over a grid search of the text and speech CFG scales st and ss . Stars indicate the configurations with the lowest WER and highest SIM, respectively. Panel (c) plots WER as a function of st with ss=1.8 , and SIM as a function of ss with st=2.0 .
Method
Instruction data
Age ↑
Gender ↑
Pitch ↑
Expressiveness ↑
Style-ACC ↑
CLSP-Score ↑
WER ↓
PromptTTS
∼ 0.3K h
0.48
0.59
0.12
0.09
0.32
0.45
6.3
VoxInstruct
2.6K h
0.76
0.99
0.23
0.41
0.60
0.50
8.6
FireRedTTS3-Instruct
330K h
0.56
0.92
0.22
0.47
0.54
0.48
6.7
Qwen3-TTS-VD
N/A
0.66
0.90
0.18
0.42
0.54
0.50
6.1
VoxCPM2
∼ 10K h
0.87
0.97
0.46
0.48
0.69
0.48
5.7
CapTTS-AR
33.6K h
0.55
0.98
0.18
0.36
0.52
0.45
11.9
Table 4: Instruction-guided TTS results on the CapSpeech test set. The best and second-best results among models trained on CapSpeech instruction data are shown in bold and underlined , respectively. N/A indicates that the amount of instruction data is not available.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Input: a reference speech latent zref , text condition y , and the requested total target width Lreq , which covers time columns of both the reference speech ℓref and the continuation target.
1
Initialize the complete target latent with standard Gaussian noise: zσ0∼N(0,I) .
2
Construct the full-size reference conditioning stream: c←0 , c[...,:ℓref]←zref .
3
For k=0,…,NFE−1 :
4
Predict the full conditional velocity: vkfull←vθ(Concat(zk,c),σk∣y) .
5
If text guidance is enabled, vk←vkfull+(st−1)(vkfull−vkno-text).
6
If speech guidance is enabled, vk←vk+(ss−1)(vkfull−vkno-speech).
Table 6: Pronunciation sensitivity to punctuation-induced changes in text conditioning. Each case pairs the original input (O) with its punctuation-modified version (M). Metrics are sentence-level rates in percent; the displayed text and Pinyin cover only the target spans.
Recent advances in pixel-based text modeling show that representing text as images enables models to exploit visual cues for language understanding. Grounding text in its visual form allows structurally similar characters with different Unicode encodings to produce similar embeddings, benefiting cross-lingual and zero-shot scenarios. Conventional text-based approaches treat each character independently, limiting generalization to unseen characters and requiring embedding expansion during cross-lingual adaptation. We propose Pixel-TTS, the first framework for visually grounded speech synthesis. It renders text as images and projects them through a 2D convolutional layer to generate embeddings. This design eliminates embedding matrix expansion during fine-tuning while improving robustness to unseen characters and orthographic variations. Extensive experiments show Pixel-TTS achieves competitive performance with strong baselines, faster convergence and robust zero-shot generalization.
Adarsh Arigala, Arjun Gangwar, S Umesh +1
SPRING Lab, Indian Institute of Technology, Madras, India · 2MBZUAI, UAE
Large language model (LLM)-based text-to-speech (TTS) systems achieve remarkable naturalness via autoregressive (AR) decoding, but require N sequential steps to generate N speech tokens. We present LLaDA-TTS, which replaces the AR LLM with a masked diffusion model that completes generation in a fixed number of parallel steps, decoupling inference latency from sequence length. Remarkably, using only 50 hours of fine-tuning data, we successfully transfer a pretrained AR checkpoint to the masked diffusion paradigm via bidirectional attention. At 64 steps, LLaDA-TTS achieves 0.98% CER (zh) and 1.96% WER (en) on Seed-TTS-Eval, matching the original CosyVoice 3 baseline performance while delivering a 2x LLM-stage speedup--a notable acceleration achieved despite the absence of KV cache, an optimization the AR baseline heavily relies on. Beyond acceleration, the bidirectional architecture naturally enables zero-shot speech editing--including word-level insertion, deletion, and substitution--without any additional training. Theoretically, we prove that AR-pretrained weights are near-optimal for bidirectional masked prediction under the locality property of acoustic tokens, explaining this rapid convergence. This general method modifies only the attention mask and objective, applying seamlessly to any LLM-based AR TTS system. Code and audio samples will be available at https://deft-piroshki-b652b5.netlify.app/.
We present JoyAI-Voice~2.0, an end-to-end anthropomorphic speech generation model built upon a fully continuous, dual-encoder architecture. Raw speech is encoded into continuous latents and partitioned into patches. Each patch is decomposed by a semantic-acoustic dual encoder into a semantically purified representation and an acoustic representation, which are fused and jointly fed to a causal autoregressive Transformer for planning. The Transformer predicts the conditioning for the next patch, and a local diffusion Transformer renders its full latents for 48,kHz synthesis. The model is trained with a joint flow-matching and stop-prediction objective, followed by supervised fine-tuning and reinforcement learning with DiffusionNFT to improve model performance. It achieves the lowest average word error rate of 2.51% on Seed-TTS, a 14.9% relative reduction over the strongest baseline, and state-of-the-art attribute fidelity on InstructTTSEval in both Chinese and English, leading on 5 of 10 perceptual dimensions with the highest overall score of 0.893 on MDVD-Eval.