Closed-Loop Dual-Reference Guidance for Accent Control in Cross-Lingual Voice Cloning

Yuanzhong Zheng; Wei Huang; Ming Ma; Kailun Jin; Youhai Jiang

cs.SDOct 21, 2026

Closed-Loop Dual-Reference Guidance for Accent Control in Cross-Lingual Voice Cloning

Authors: Yuanzhong Zheng, Wei Huang, Ming Ma, Kailun Jin, Youhai Jiang

Organizations: VoxNexus, Singapore

Abstract

Zero-shot cross-lingual voice cloning often carries the reference speaker's accent into the target language. We propose closed-loop dual-reference guidance, which pairs the original reference with a reusable native anchor constructed by voice conversion (VC). The references initialize separate language-model (LM) states that guide a shared acoustic trajectory; each completed patch then updates both states, keeping their generated histories aligned with the emitted speech. Anchor construction runs once per speaker, and online synthesis uses a frozen TTS model without per-utterance VC. On 20 held-out French speakers, the fixed setting $λ^{\!*}=0.33$ reduces residual French LID mass by 73% versus direct cloning, at an ECAPA similarity cost of only $0.042$ ; relative to anchor-only cloning, it recovers $0.155$ . The method achieves the lowest mean WER ( $0.009$ ) and highest mean GOP ( $4.750$ ) among the held-out test arms. In blinded preference tests, native-English listeners favor it over direct cloning on pronunciation naturalness, and over both anchor-only cloning and a TTS $\to$ VC cascade on speaker identity. Online RTF is approximately 26% lower than the cascade's ( $0.59$ vs. $0.80$ ), and the same setting reduces Spanish LID mass by 56% without re-tuning.

Figures & tables

Figure 1: Method overview. (a) Offline native-anchor construction: $r_{\mathrm{FR}}$ and four native donors undergo zero-shot conversion once per speaker; the highest-similarity candidate is kept as $r_{\mathrm{EN}}$ . (b) Closed-loop dual-reference guidance. At autoregressive step $k$ , both reference-conditioned fields are evaluated on the same flow latent $z_{k,t}$ and combined via Eq. (1). After the ODE produces acoustic patch $Z_{k}$ , its encoding advances both LM contexts via Eq. (2). Shared generated history; distinct reference contexts and KV caches. (c) $\lambda=0$ is Arm A (accent leakage); $\lambda=1$ is Arm B (reduced speaker similarity); $\lambda^{\!*}=0.33$ is the selected operating point (Arm C).

	WER $\downarrow$	$P(\mathrm{fr})$ $\downarrow$	$P(\mathrm{en})$ $\uparrow$	Speaker similarity $\uparrow$			GOP $\uparrow$	DNSMOS $\uparrow$
Arm				ECAPA	CAM++-EN	CAM++-CN ∗
A (Direct)	0.016	0.093	0.843	0.601	0.624	0.746	4.663	2.900
B (Anchor)	0.013	0.000	0.993	0.404	0.444	0.702	4.733	2.934
C (Proposed)	0.009	0.025	0.942	0.559	0.586	0.748	4.750	2.887
Cascade	0.047	0.001	0.985	0.468	0.492	0.770	4.378	3.026

Table 1: Main evaluation on held-out test data (20 French speakers

\times

20 texts

\times

3 seeds;

n=1200

per arm,

\lambda^{\!*}=0.33

). Speaker-macro averages; bold values mark the best result in each column.

P(\mathrm{fr})

is residual French LID mass. ∗ CAM++-CN is Seed-VC’s internal speaker encoder (Sec. 6.2).

Figure 2: Accent–timbre trade-off on development (30 speakers). (a) ECAPA; (b) CAM++-EN.

x

\Delta\mathrm{SIM}

vs. A;

y

\Delta\mathrm{GOP}

vs. A. The shaded band is the 95% speaker-clustered bootstrap CI of

\Delta\mathrm{GOP}

. Orange: C (proposed)

\lambda

sweep;

\lambda^{\!*}

is the selected operating point (

0.33

Figure 3: Residual source-language LID mass at

\lambda^{\!*}=0.33

. Dumbbells: A

\to

C reduction; B is the floor. Brackets give speaker-clustered 95% CIs of the paired difference

\Delta=C{-}A

(negative values indicate a reduction under C).

Arm	DiT/patch	VC NFE	Lat. (s)	RTF	Peak VRAM (MiB)
(a) Direct cloning and cascade comparison
A	18	—	1.27	0.474	8239
C (velocity)	27	—	1.52	0.587	8235
Cascade	18	50	$\approx$ 2.10	$\approx$ 0.80	8563 / 1474 †
(b) Matched implementation comparison
C (velocity)	27	—	1.62	0.589	8406

Table 2: H100 80GB inference. (a) vs. direct cloning and cascade. (b) Matched velocity vs. conditioning (

\mu

-mix). TTS: 9-word text, five timed runs after warmup. DiT: batched branch evaluations per patch over nine nonzero Euler updates. † sequential cascade stages.

Comp.	Dim.	Pref. base	Pref. C	Tie	$p$
C vs. A	Accent / nat.	6.7 (4)	28.3 (17)	65.0 (39)	$0.0057$
C vs. B	Speaker sim.	5.0 (3)	65.0 (39)	30.0 (18)	$<0.0001$
C vs. Casc.	Accent / nat.	53.3 (32)	21.7 (13)	25.0 (15)	$0.026$
C vs. Casc.	Speaker sim.	8.3 (5)	63.3 (38)	28.3 (17)	$<0.0001$
C vs. A †	Speaker sim.	26.7 (16)	21.7 (13)	51.7 (31)	$0.723$

Table 3: Preferences on development speakers (7 evaluators

\times

60 sessions). % (sessions), majority vote. Bold: significantly preferred system.

p

: speaker-clustered sign-flip; pre-planned rows remain significant under Holm–Bonferroni. † Follow-up speaker-similarity comparison.

Config.	ECAPA $\uparrow$	CAM++-EN $\uparrow$	GOP $\uparrow$	WER $\downarrow$	$P(\mathrm{fr})$ $\downarrow$
A ( $\lambda=0$ )	0.517	0.589	4.602	0.020	0.153
B (anchor only)	0.308	0.378	4.770	0.009	0.001
C (velocity)	0.467	0.542	4.791	0.004	0.044
C (conditioning)	0.465	0.546	4.828	0.000	0.034
concat	0.372	0.459	4.743	0.003	0.003
open-KV	0.458	0.522	3.524	0.139	0.045

Table 4: Paired ablations on development (30 speakers,

\lambda=0.33

n=300

Voice Cloning

Voice cloning uses speech samples to reproduce a speaker’s vocal identity in generated speech across different content, languages, and styles.

See all 42 papers

Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning
Sep 24, 2026Yoomee Cho, Jisun LeeTTS Synthesis Multilingual TTS Synthesis
ESTsoft, Seoul, Republic of Korea
X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning
May 7, 2026Rixi Xu, Qingyu Liu, Haitao Li +10TTS Synthesis Zero-Shot TTS
1MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Center for Language and Speech Processing, Johns Hopkins University · 2Shanghai Innovation Institute +4
One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speech
Apr 28, 2026Amanuel Gizachew Abebe, Yasmin MoslemTTS Synthesis Text-to-Speech
Shaggar Institute of Technology · Shaggar City, Ethiopia · Trinity College Dublin +1

Sep 24, 2026J/K move · Enter open · S save

Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning

Yoomee Cho, Jisun Lee

ESTsoft, Seoul, Republic of Korea

TTS Synthesis Multilingual TTS Synthesis Zero-Shot TTS Text-to-Speech

Open paper AskPDF arXiv

In cross-lingual zero-shot text-to-speech, the accent of the reference leaks into the target speech. We propose accent analogy guidance (AAG), a training-free sampler term that subtracts an accent direction estimated from the model's own predictions for one synthetic voice rendered in both languages, so the voice cancels and only the accent remains. By a blind LLM accent judge on real dubbing data, reweighting classifier-free guidance between reference and text, and its variants, stay near one identity-accent trade-off curve; we score a method by its speaker similarity above that curve at equal accent (

Δ

SIM). Across four open TTS models AAG lies above the curve: on OmniVoice

Δ

SIM is +0.11 to +0.27 on three test sets (accent 3.51 to 4.28 on a 1-5 scale at speaker similarity 0.29, where reweighting keeps 0.02); MaskGCT and CosyVoice 2 also lie above their curves, and on F5-TTS it is more native than any reweighting setting. An LLM-free language-ID measure and a twelve-listener panel agree. A premise test and the reach of a model's own curve indicate in advance whether and roughly how much AAG can gain, predicting the one model where it gains nothing (X-Voice).

Abstract

Voice Cloning

Explore similar work

Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning

X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning

One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speech