Voice imitation aims to transform source speech to match a reference speaker's timbre and speaking style while preserving linguistic content. A straightforward approach is to train on triplets of (source, reference, target), where source and target share the same content but target matches the reference's voice characteristics, yet such data is extremely scarce. Existing approaches either employ carefully designed disentanglement architectures to bypass this data scarcity or leverage external systems to synthesize pseudo-parallel training data. However, the former requires intricate model design, and the latter faces a quality ceiling when synthetic speech is used as training targets. To address these limitations, we propose MimicLM, which takes a novel approach by using synthetic speech as training sources while retaining real recordings as targets. This design enables the model to learn directly from real speech distributions, breaking the synthetic quality ceiling. Building on this data construction approach, we incorporate interleaved text-audio modeling to guide the generation of content-accurate speech and apply post-training with preference alignment to mitigate the inherent distributional mismatch when training on synthetic data. Experiments demonstrate that MimicLM achieves superior voice imitation quality with a simple yet effective architecture, significantly outperforming existing methods in naturalness while maintaining competitive similarity scores across speaker identity, accent, and emotion dimensions.
Figures & tables
Figure 1: Overview of MimicLM architecture for voice imitation. The model processes reference audio ( ref token ) for target timbre/style and source audio ( src token ), then generates the conversion target in two phases: a chunked phase with interleaved text-audio prediction, followed by a continuous phase for remaining speech tokens. Both the audio tokenizer and decoder are frozen during training (snowflake icons). Special control tokens are annotated at bottom right.
Figure 2: Four-stage pipeline for pseudo-parallel data construction. We randomly sample two speakers with their utterances (Stage 1), synthesize cross-speaker audio using TTS (Stage 2), apply role-swapping to ensure real speech serves as training target (Stage 3), and filter low-quality pairs via ASR-based verification (Stage 4).
Model
Naturalness & Speech Quality
Intelligibility
Speaker Similarity
UTMOS ↑
OVRL ↑
SIG ↑
BAK ↑
WER (%) ↓
S-SIM ↑
A-SIM ↑
E-SIM ↑
Timbre-only Voice Conversion
FACodec Ju et al. (2024)
2.13
3.65
4.22
3.87
4.86
0.372
0.571
0.912
MeanVC Ma et al. (2025)
2.84
3.65
4.04
4.14
17.82
0.419
0.450
0.909
OpenVoice v2 Qin et al. (2023)
2.55
4.15
4.47
4.43
4.15
0.424
0.552
0.918
LSCodec Guo et al. (2024b)
2.84
3.94
4.33
4.25
8.76
0.445
0.583
0.911
Table 1: Performance comparison on SeedTTS test-vc-en . We compare our method against two groups of baselines: (1) Timbre-only systems, and (2) Full Voice Imitation systems (our direct baselines: Vevo and SeedVC v2), which transfer both timbre and style. “ ↑ ” indicates higher is better, and “ ↓ ” indicates lower is better. WER comparison is only made among voice imitation systems.
Model
N-MOS
S-MOS
A-MOS
E-MOS
SeedVC v2
3.14 ± 0.11
3.03 ± 0.12
3.82 ± 0.12
3.61 ± 0.16
Vevo
3.85 ± 0.14
4.32 ± 0.13
4.64 ± 0.09
4.23 ± 0.09
Ours (DPO)
4.71 ± 0.08
4.62 ± 0.10
4.53 ± 0.11
3.94 ± 0.13
Table 2: Subjective evaluation results with Mean Opinion Scores (MOS) on a 1–5 scale. Scores are reported as mean ± 95% confidence interval.
Model
Real/Real
Syn/Real
Real/Syn
Vevo
17.99
13.90
20.44
Ours (SFT)
15.80
4.30
18.48
Ours (DPO)
13.81
3.63
15.58
Table 3: Synthetic-to-real gap analysis on MimicLM-Test. Column headers indicate source/reference input types, where source provides linguistic content and reference provides target voice. All values are WER (%).
Configuration
OVRL ↑
SIG ↑
BAK ↑
WER (%) ↓
S-SIM ↑
A-SIM ↑
E-SIM ↑
w/o RS, w/o IT
3.99
4.39
4.25
18.25
0.547
0.678
0.903
w/ RS, w/o IT
4.05
4.41
4.33
20.69
0.555
0.684
0.910
w/o RS, w/ IT
4.03
4.41
4.31
15.34
0.547
0.681
0.896
w/ RS, w/ IT (SFT)
4.11
4.43
4.41
18.64
0.560
0.691
0.913
SFT + DPO
4.12
4.44
4.42
14.73
0.573
0.688
0.905
Table 4: Ablation study on key components using base -scale data. RS = Role-Swapping, IT = Interleaved Text modeling. The baseline uses standard triplets ( Utt_A1 , Utt_B1 , Syn_B1 ) without IT. RS uses role-swapped triplets ( Syn_B1 , Utt_A2 , Utt_A1 ). All configurations are evaluated on SeedTTS test-vc-en . Training hyperparameters follow the SFT stage setup described in Appendix D .
Figure 3: Impact of training data scale on WER and speaker similarity (S-SIM) evaluated on SeedTTS test-vc-en .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Training Data
English
Chinese
WER (%) ↓
S-SIM ↑
CER (%) ↓
S-SIM ↑
English-only (0.84M)
18.64
0.560
-
-
English-only (2.6M)
15.62
0.564
-
-
Chinese-only (0.74M)
-
-
13.16
0.630
Chinese (0.74M) + English (0.84M)
19.49
0.558
12.84
0.632
Chinese (0.74M) + English (2.6M)
15.21
0.566
13.68
0.631
Appendix
Table 5: Performance on Chinese (SeedTTS test-zh ) and English (SeedTTS test-vc-en ) test sets for different training data configurations.
System
Frame Rate
English
Chinese
WER ↓
SIM ↑
WER ↓
SIM ↑
FireRedTTS Guo et al. (2024a)
25
8.53
0.46
1.27
0.65
SparkTTS Wang et al. (2025a)
50
2.50
0.57
1.78
0.66
Llasa Ye et al. (2025) (16 kHz)
50
3.94
0.58
8.02
0.64
F5-TTS Chen et al. (2025)
93.75
3.02
0.63
3.87
0.71
CosyVoice 2 Du et al. (2024)
25
2.89
0.66
1.29
0.76
Appendix
Table 6: Comparison of zero-shot TTS systems on SeedTTS test-en and test-zh . Results are from Wang et al. (2025b) . SIM denotes cosine speaker similarity.
System
Rate (Hz)
Codebooks
Semantic
WER ↓
SIM ↑
UTMOS ↑
EnCodec Défossez et al. (2022)
75
2
×
5.36
0.48
1.54
Mimi Défossez et al. (2024)
12.5
6
✓
4.51
0.52
3.09
BigCodec Xin et al. (2024)
80
1
×
3.25
0.61
3.59
X-Codec 2 Ye et al. (2025)
50
1
✓
2.63
0.62
3.68
Vevo Tokenizer Zhang et al. (2025)
50
1
✓
3.04
0.53
3.50
CosyVoice 2 Du et al. (2024)
25
1
✓
4.10
0.68
3.65
Appendix
Table 7: Comparison of audio tokenizers evaluated via speech reconstruction on SeedTTS test-en . Results are from Wang et al. (2025b) . “Semantic Distill” indicates whether the tokenizer incorporates semantic distillation.
We present a voice conversion (VC) framework that utilizes K-Nearest Neighbors (KNN) retrieval over WavLM representations to align non-parallel source and target speech, constructing synthetic training pairs for supervised learning. The retrieved segments serve as synthetic inputs, while real target audio provides ground-truth outputs, forming a synthetic-to-real training paradigm that naturally supports multilingual data without requiring parallel corpora or explicit alignment. To ensure consistent target-speaker identity, we incorporate a speaker loss derived from a pretrained speaker verification model. Experiments across multiple languages demonstrate that the proposed approach achieves high naturalness and strong speaker similarity, outperforming competitive VC baselines, despite being trained exclusively on English data. Samples can be accessed at: https://palindromic-vc.github.io.
In this paper, we present X-Voice, a 0.4B multilingual zero-shot voice cloning model that clones arbitrary voices and enables everyone to speak 30 languages. X-Voice is trained on a 420K-hour multilingual corpus using the International Phonetic Alphabet (IPA) as a unified representation. To eliminate the reliance on prompt text without complex preprocessing like forced alignment, we design a two-stage training paradigm. In Stage 1, we establish X-Voices1 through standard conditional flow-matching training and use it to synthesize 10K hours of speaker-consistent segments as audio prompts. In Stage 2, we fine-tune on these audio pairs with prompt text masked to derive X-Voices2, which enables zero-shot voice cloning without requiring transcripts of audio prompts. Architecturally, we extend F5-TTS by implementing a dual-level injection of language identifiers and decoupling and scheduling of Classifier-Free Guidance to facilitate multilingual speech synthesis. Subjective and objective evaluation results demonstrate that X-Voice outperforms existing flow-matching based multilingual systems like LEMAS-TTS and achieves zero-shot cross-lingual cloning capabilities comparable to billion-scale models such as Qwen3-TTS. To facilitate research transparency and community advancement, we open-source all related resources.
Rixi Xu, Qingyu Liu, Haitao Li +10
1MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Center for Language and Speech Processing, Johns Hopkins University · 2Shanghai Innovation Institute +4
Text-to-speech (TTS) systems are conventionally optimized for intelligibility and speaker similarity. However, neither objective captures the prosodic detail that distinguishes human performances from synthetic speech, leaving a realism gap that larger models and higher-quality data can narrow but not close. In the absence of an objective that directly targets this gap, realism remains a persistent challenge in TTS. We present Audio8-TTS, a compact 0.6B-parameter model for zero-shot voice cloning that achieves state-of-the-art performance in intelligibility and speaker similarity, with particularly strong results in perceptual naturalness. To align the training signal with human perception rather than signal-level fidelity, we construct a quality ladder from widely used open-source TTS systems and train a realism critic to assess generated speech against matched human recordings of the same sentences. An identity-erasure objective discourages the critic from relying on generator fingerprints. We train the model in three stages: multilingual supervised fine-tuning; group relative policy optimization (GRPO) with word error rate (WER) and speaker similarity (SIM) rewards; and GRPO guided by the realism critic. The final stage improves perceived prosodic naturalness and speaker similarity while preserving WER performance.