Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.
Figures & tables
Figure 1: Decoder transformer block for our AV-DiT architecture. Audio and video streams are fused using cross-attention. Each stream is based on the CrossDiT architecture ( Arkhipkin et al., 2026 ) .
SECS vs. reference
SECS vs. centroid
Model
E
W
R
E
W
R
WER%
CER%
WER 0 %
k6a_5b (ours)
0.766
0.944
0.866
0.770
0.951
0.878
5.76
5.21
86.6
Qwen3-TTS 0.6B
0.678
0.936
0.840
0.711
0.946
0.864
0 0.81
0 0.36
95.7
Qwen3-TTS 1.7B
0.674
0.938
0.839
0.709
0.951
0.864
0 0.74
0 0.17
95.3
IndexTTS2
0.695
0.885
0.854
0.693
0.878
0.867
0 0.96
0 0.22
93.2
XTTS-v2
0.575
0.923
0.816
0.611
0.940
0.841
0 1.14
0 0.32
93.2
Table 1: English VCTK benchmark ( 674 samples per model, 30 speakers, unique texts). The SECS columns are speaker-encoder cosine similarities (higher is better) for three verification networks (E: ECAPA-TDNN, W: WavLM-SV, R: Resemblyzer), reported both against the single reference clip ( vs. reference ) and against the centroid of the reference and three enrollment clips ( vs. centroid ). WER%/CER% are word/character error rates (lower is better) and WER 0 % is the fraction of samples transcribed with zero word errors (higher is better). Best value per column in bold , second best underlined .
Metric
Relative change
FAD vs. real speech ↓
−30.6%
WER ↓
−46.0%
CER ↓
−47.0%
WER 0 (perfect samples) ↑
+113%
CLAP text–audio ↑
+29.4%
UTMOS naturalness ↑
+0.6%
Table 2: No-regression check: relative change of the base T2AV model after reference-aware fine-tuning, evaluated in the reference-free regime on 400 prompts ( 30 speakers, identical seeds/texts). A positive change is an improvement for ↑ -metrics and a negative change is an improvement for ↓ -metrics; every metric moves in the improving direction.
Prompt
Technical
Speech/Aesth.
follow.
quality
quality
Base wins
0 9
0 8
11
Fine-tuned wins
14
23
17
Both good
78
75
78
Both bad
0 9
0 4
0 4
Base win rate
48
43
47
Table 3: SBS human evaluation on 100 held-out text-to-audio prompts, base audio model vs. the same model after reference-aware fine-tuning (both generate without a reference). Numbers are percentages of pair-wise judgements per axis.
TTS systems with autoregressive semantic modeling have demonstrated strong zero-shot voice cloning performance and rich expressive variation, but their sequential decoding incurs substantial latency. Non-autoregressive alternatives offer much faster generation, yet often rely on more restrictive reference conditioning, such as requiring transcripts of the reference speech during inference. We present Tacit-TTS, an efficient transcript-free zero-shot voice cloning system distilled from IndexTTS2. Our model replaces autoregressive text-to-semantic decoding with masked non-autoregressive generation, introduces training-free acoustic length estimation, and accelerates the flow-matching renderer through ReFlow distillation. Across two English and two Mandarin datasets, Tacit-TTS achieves competitive zero-shot quality while generating speech over 10x faster than IndexTTS2 for utterances longer than 5 seconds. Its transcript-free conditioning further supports cross-lingual and non-lexical references. We validate this capability using references from eight other languages, infant babble, and synthetic gibberish, where transcript-dependent systems often degrade or fail due to unreliable ASR transcripts.
Artificially generated speech is increasingly embedded in everyday life. Voice cloning in particular enables applications where identity preservation is important, such as completing a recording, dubbing in a new language, or preserving the voices of individuals with speech loss. However, in our work, we find that despite the term, voice cloning does not faithfully ''clone'' an individual's voice. Instead, we find that widely-used voice cloning models systematically apply style transfer to source voices. As rated by human annotators, cloned voices are perceived as more authoritative, warm, customer-service-like, and human-like compared to their sources. Human annotators also report greater trust in cloned voices than source voices, and a greater willingness to disclose sensitive personal information to them. Our work furthermore shows that voice cloning leads to homogenization of speaker characteristics, as measured by reduced variance in accent, speaking rate, and the audio embedding space. Together, our results highlight a new set of limitations and risks of voice cloning technology and their potential impact on human behavior.
Existing video personalization methods preserve visual likeness but treat video and audio separately. Without access to the visual scene, audio models cannot synchronize sounds with on-screen actions; and because classical voice-cloning models condition only on a reference recording, a text prompt cannot redirect speaking style or acoustic environment. We propose ID-LoRA (Identity-Driven In-Context LoRA), which jointly generates a subject's appearance and voice in a single model, letting a text prompt, a reference image, and a short audio clip govern both modalities together. ID-LoRA adapts the LTX-2 joint audio-video diffusion backbone via parameter-efficient In-Context LoRA and, to our knowledge, is the first method to personalize visual appearance and voice in a single generative pass. Two challenges arise. Reference and generation tokens share the same positional-encoding space, making them hard to distinguish; we address this with negative temporal positions, placing reference tokens in a disjoint RoPE region while preserving their internal temporal structure. Speaker characteristics also tend to be diluted during denoising; we introduce identity guidance, a classifier-free guidance variant that amplifies speaker-specific features by contrasting predictions with and without the reference signal. In human preference studies, ID-LoRA is preferred over Kling 2.6 Pro by 73% of annotators for voice similarity and 65% for speaking style. On cross-environment settings, speaker similarity improves by 24% over Kling, with the gap widening as conditions diverge. A preliminary user study further suggests that joint generation provides a useful inductive bias for physically grounded sound synthesis. ID-LoRA achieves these results with only ~3K training pairs on a single GPU. Code, models, and data will be released.