HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos

Pavel Andreev; Andrei Gusev; Adithya Narayan; Yi Ren; Ivan Somov; Zijian Li; Sergey Demyanov; Viktor Goriachko

cs.SDOct 21, 2026

HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos

Authors: Pavel Andreev, Andrei Gusev, Adithya Narayan, Yi Ren, Ivan Somov, Zijian Li, Sergey Demyanov, Viktor Goriachko,

Organizations: HeyGen Research

Abstract

Most voice-cloning systems are built for audio alone: they copy the reference recording's delivery and acoustics, so a voice cloned on a phone sounds like a phone recording whether the speaker is pictured in a studio, on a street, or in a cathedral. Speech for a talking video should sound like the pictured person speaking in the pictured place. We present HeyGen Voice, a scene-aware voice-cloning model that separates who is speaking from how and where: it keeps the speaker's voice from an everyday recording while the target image sets the delivery and acoustics. Prior image-conditioned TTS does half of this job: it either adds scene acoustics without keeping a specific speaker or infers a voice from a face. HeyGen Voice is a multimodal autoregressive transformer conditioned on the image, speaker and scene captions, and the voice recording, trained in five stages that move from multilingual TTS pretraining through simulated and real cross-scene adaptation to reinforcement and preference optimization. Two clone policies trade scene adaptation against fidelity to the original recording, and an audio anchor holds long scripts in one voice and one acoustic environment. Listeners prefer HeyGen Voice over leading commercial systems across a third-party arena, production votes, and talking-video studies: it ranks first (Elo 1201) on the public Artificial Analysis controlled-voice TTS leaderboard (October 2026); in a blind test, users cloning their own voice chose it over ElevenLabs v4 and Cartesia Sonic 3.6 in 62% and 54% of votes; and volunteers rated talking videos of their own digital twin about 10 to 30 points higher (out of 100) when voiced by HeyGen Voice than by leading commercial alternatives. Try HeyGen Voice and hear samples at https://heygen.com/heygen-voice.

Figures & tables

Panel of Table 1 — Table 1: Capabilities by method category.

Panel of Figure 1 — Figure 1: HeyGen Voice architecture. Multimodal inputs are encoded by frozen encoders and pooled into conditioning tokens; the clone policy determines whether the continuation state comes from an anchor designed for the pictured scene or from the reference recording itself. The conditioning prefix, script tokens, and acoustic codes form one sequence processed by an autoregressive transformer decoder. The model predicts acoustic codes, supervised by cross-entropy during supervised training, by intelligibility and speaker-similarity rewards during GRPO, and by preference pairs during DPO; a flow-matching decoder conditioned on the transformer’s hidden states and the reference audio produces the waveform.

Panel of Figure 2 — Figure 2: Example speaker and background captions for one recording, a speaker talking inside a moving car. Each caption is a short descriptive paragraph; the background caption names the sound sources, the space, and what stays distant, and the speaker caption covers voice, tone, pace, expressiveness, and articulation.

Source	Role
Target image	Person and scene context
Speaker caption	Delivery and vocal attributes
Background caption	Environment and acoustics
Reference tokens	Speaker voice: timbre, accent, and speaking cadence
Consistency embedding	Shared voice and scene acoustics
Local acoustic context	Local acoustic continuity

Table 2: Conditioning prefix of the acoustic model. Each source contributes a small, fixed number of pooled tokens; script and continuation lengths vary.

Panel of Figure 3 — Figure 3: Training curriculum. Talking-video data drives the adaptation stages. The GRPO checkpoint is the release base model and the base for the expressiveness adapter, which is an optional inference control.

Panel of Figure 4 — Figure 4: One voice reference, multiple target scenes. An everyday phone recording supplies the speaker reference. For each target image and the captions written from it, HeyGen Voice generates speech that retains that voice and adapts delivery and acoustics to the requested setting; the designed audio anchors subsequent utterances within the scene. Images depict a fictional speaker; video rendering is a separate downstream stage.

Clone policy	Behavior
voice-clone	Clones the voice: timbre, accent, and speaking cadence. Delivery and acoustics follow the image, captions, and acoustic controls.
full-clone	Clones the voice and the acoustic conditions of the reference. Highest similarity to the original recording.

Table 3: Clone policies, named for what they clone from the reference.

Evaluation	Benchmark	Main evidence
Audio-based	Cross-scene (Section 5.1 )	Speaker similarity and audio–audio and audio–image alignment on 331 cases
	Multilingual (Section 5.2 )	Ten-language CER with corrected scripts and transcription review
	Listening tests (Section 5.3 )	Paired human judgments of naturalness, accent, clarity, narrator choice, and reference similarity in seven system pairs on 245 cases (42,875 ratings)
	Production voice cloning (Section 5.4 )	Blind pairwise and three-way picks by platform users cloning their own voice, vs. ElevenLabs v4 and Cartesia Sonic 3.6
	Public arena (Section 5.5 )	Artificial Analysis controlled-voice leaderboard: blind pairwise listener votes on eight shared cloned voices, Elo ranking
Video-based	Volunteer self-evaluation (Section 5.6 )	43 participants rate talking videos of themselves, vs. Cartesia Sonic 3.6, ElevenLabs v4, and Seedance 2.5

Table 4: Evaluation overview. Automatic audio–image similarity is a diagnostic of the generated speech; human judgments of talking videos are collected in the video-based studies.

System / clone policy	Speaker similarity $\uparrow$		ImageBind $\uparrow$
	To target	To reference	Audio–audio	Audio–image
HeyGen Voice ( voice-clone )	0.590	0.739	0.637	0.282
HeyGen Voice ( full-clone )	0.581	0.757	0.608	0.278
Cartesia Sonic 3.6	0.582	0.745	0.599	0.261
ElevenLabs v4	0.556	0.712	0.607	0.269
Inworld Realtime TTS-2	0.464	0.567	0.544	0.231

Table 5: Cross-scene benchmark on 331 matched cases. Speaker similarity is measured against the real target-scene recording and against the original voice reference. Best value per column in bold, second best underlined.

Language	HeyGen full-clone	Qwen3-TTS 1.7B	Cartesia Sonic 3.6	ElevenLabs v4	Inworld TTS-2
English	0.0000	0.0000	0.0089	0.0000	0.0000
Chinese	0.0000	0.0286	0.0526	0.0000	0.0492
Spanish	0.0000	0.0000	0.0000	0.0000	0.0109
French	0.0000	0.0000	0.1389	0.0000	0.0000
German	0.0000	0.0114	0.0827	0.0000	0.0000
Italian	0.0099	0.0270	0.0393	0.0238	0.0099

Table 6: Agreement-only CER (%, lower is better) on the corrected multilingual benchmark; 100 utterances per language and system, macro average over languages. Vendor versions follow the benchmark labels.

Figure 5: Target images of the listening benchmark. Each speaker’s image is adapted to the same podcast-studio template, which guides HeyGen Voice toward a professional narrator’s delivery in a treated room. The four faces shown are AI-generated examples adapted in the same way.

Target language	Intra-lingual	Cross-lingual	Total
English (US)	22	25	47
Spanish (Mexico)	25	24	49
German (Germany)	25	25	50
French (France)	25	24	49
Chinese (China)	45	5	50
Total	142	103	245

Table 7: Listening benchmark composition. Counts are unique reference–script cases, reused in every system comparison.

Aspect	Question
Reference similarity	Which candidate sounds more similar to the reference? (five positions, $-2$ to $+2$ )
Narrator choice	Which voice would you choose for an entire educational lesson narrated by a presenter?
Naturalness	Which voice sounds more human and natural, ignoring the spoken content?
Native accent	Which voice sounds more native in pronunciation, rhythm, and stress?
Clarity	Which recording is clearer and higher quality (muffling, noise, distortion, glitches)?

Table 8: Listening rubric. Descriptions paraphrase the English study questions; localized equivalents are used in the other four locales.

Matchup (A / B)	Reference similarity	Narrator choice	Naturalness	Native accent	Clarity
voice-clone / ElevenLabs v4	43.1 * $\pm$ 3.3	61.5 * $\pm$ 3.4	50.2 $\pm$ 3.6	55.7 $\pm$ 4.1	78.4 * $\pm$ 2.9
voice-clone / Cartesia Sonic 3.6	22.7 * $\pm$ 2.4	77.9 * $\pm$ 2.8	69.6 * $\pm$ 3.4	77.8 * $\pm$ 3.2	86.6 * $\pm$ 2.5
voice-clone / Inworld TTS-2	44.9 $\pm$ 3.5	68.1 * $\pm$ 3.3	63.9 * $\pm$ 3.4	70.9 * $\pm$ 3.5	63.5 * $\pm$ 3.5
full-clone / ElevenLabs v4	63.0 * $\pm$ 3.6	47.3 $\pm$ 3.7	43.9 * $\pm$ 3.7	45.8 $\pm$ 4.5	65.5 * $\pm$ 4.0
full-clone / Cartesia Sonic 3.6	53.8 $\pm$ 3.2	60.8 * $\pm$ 3.9	46.0 $\pm$ 3.9	67.9 * $\pm$ 4.5	75.0 * $\pm$ 3.4
voice-clone / full-clone	27.9 * $\pm$ 3.3	73.6 * $\pm$ 3.3	58.6 * $\pm$ 4.2	62.2 * $\pm$ 5.0	77.3 * $\pm$ 3.5

Table 9: Share of decided votes (%) for the first-named system (A) in each pair, with 95% whole-case bootstrap intervals, on 245 cases and 1,225 ratings per aspect and pair. Ties are excluded from the denominator. ∗ : Holm-corrected

p<0.05

over the 35 pooled tests.

System	Reference similarity	Narrator choice	Naturalness	Native accent	Clarity
HeyGen voice-clone	1000 (anchor)	1000 (anchor)	1000 (anchor)	1000 (anchor)	1000 (anchor)
HeyGen full-clone	1144 $\pm$ 17	904 $\pm$ 14	951 $\pm$ 12	964 $\pm$ 11	907 $\pm$ 14
ElevenLabs v4	1060 $\pm$ 17	935 $\pm$ 15	995 $\pm$ 13	985 $\pm$ 12	860 $\pm$ 16
Cartesia Sonic 3.6	1141 $\pm$ 15	836 $\pm$ 19	934 $\pm$ 14	878 $\pm$ 18	811 $\pm$ 15
Inworld TTS-2	1032 $\pm$ 22	896 $\pm$ 21	931 $\pm$ 18	909 $\pm$ 18	939 $\pm$ 17

Table 10: Bradley–Terry scores on the Elo scale, fitted separately per aspect to the seven pairs over all 245 cases with voice-clone fixed at 1000, with 95% whole-case bootstrap intervals. Inworld TTS-2 is placed through its single comparison with voice-clone .

Figure 6: Narrator choice against reference similarity on the Elo scale, pooled over 245 cases, with 95% whole-case bootstrap intervals; the dashed line joins the non-dominated systems. The benchmark pairs user-like references with a studio scene, so the two questions trade off: full-clone keeps the reference and gives up narrator quality, and voice-clone sounds like a professional narrator and gives up reference similarity.

Figure 7: Listening profiles, pooled and for English, Spanish, and German. Each spoke is scaled to its own displayed range, printed at the spoke; numerical keys give the fitted Elo-scale scores with voice-clone at 1000. Polygon areas are not comparable across panels.

Figure 8: Listening profiles for French and Chinese, and for the 142 intra-lingual and 103 cross-lingual cases. Subset fits use the same model as Table 10 ; their bootstrap intervals are wider (typically

\pm

35 points per language and

\pm

20 per condition).

Comparison	Votes	Chance	ElevenLabs v4	Cartesia Sonic 3.6	HeyGen Voice
Pairwise	8,332	50.0	37.8 $\pm$ 1.1	–	62.2 $\pm$ 1.1
Pairwise	7,286	50.0	–	45.9 $\pm$ 1.2	54.1 $\pm$ 1.2
Three-way	6,153	33.3	23.0 $\pm$ 1.1	35.3 $\pm$ 1.2	41.7 $\pm$ 1.3

Table 11: Blind side-by-side picks in the HeyGen voice-cloning flow (snapshot of October 2, 2026). Each vote is one user choosing the clone of their own voice; HeyGen Voice runs the full-clone policy. Entries are pick shares in percent with Wilson 95% intervals over picks; the largest share in each row is bold. Chance is the share expected from uniform random picking, and every HeyGen Voice share exceeds it (exact binomial test,

p<0.001

Figure 9: Top ten of the Artificial Analysis controlled-voice text-to-speech leaderboard, snapshot of October 9, 2026. Elo scores with the leaderboard’s 95% intervals; the right column is the leaderboard’s rank range. All systems clone the same eight reference voices; HeyGen Voice runs voice-clone .

System	Mean score	Centered mean	Mean rank	Top-1 ballots
HeyGen Voice ( voice-clone )	76.1 $\pm$ 5.3	+12.7 $\pm$ 3.5	2.19	18
HeyGen Voice ( full-clone )	72.6 $\pm$ 5.8	+9.3 $\pm$ 3.8	2.41	13
Cartesia Sonic 3.6	63.1 $\pm$ 7.2	-0.2 $\pm$ 4.7	2.99	12
ElevenLabs v4	59.2 $\pm$ 7.6	-4.2 $\pm$ 5.0	3.26	9
Seedance 2.5	45.7 $\pm$ 6.9	-17.6 $\pm$ 4.3	4.16	2

Table 12: Volunteer self-evaluation: 43 participants each rated one talking video of themselves per system on a 1–100 scale. Mean scores carry 95% participant-bootstrap intervals. The centered score subtracts the participant’s mean over the five systems, so it removes differences in how participants use the scale; its intervals use a common 88% level following the Goldstein–Healy approach [ 29 ] , providing an approximate visual guide to unadjusted paired comparisons. Formal significance is assessed using the paired tests in Table 13 . Mean rank averages each participant’s ranking of the five systems (1 = best, ties share the average rank); top-1 ballots count the participants who gave the system their highest score, crediting every system tied at the top.

Figure 10: Centered scores in the volunteer self-evaluation with 88% Goldstein–Healy intervals for approximate visual comparison. Formal paired tests are reported in Table 13 . Zero is the participant’s mean over the five systems.

HeyGen Voice setting	Competitor	Mean difference	Wins / ties / losses	$p$ (Holm)
voice-clone	Cartesia Sonic 3.6	+13.0 $\pm$ 9.0	27 / 0 / 16	0.016 *
voice-clone	ElevenLabs v4	+16.9 $\pm$ 7.4	29 / 3 / 11	$<$ 0.001 *
voice-clone	Seedance 2.5	+30.3 $\pm$ 7.6	39 / 1 / 3	$<$ 0.001 *
full-clone	Cartesia Sonic 3.6	+9.5 $\pm$ 6.9	24 / 5 / 14	0.016 *
full-clone	ElevenLabs v4	+13.4 $\pm$ 9.7	29 / 2 / 12	0.016 *
full-clone	Seedance 2.5	+26.9 $\pm$ 8.1	35 / 2 / 6	$<$ 0.001 *

Table 13: Paired comparisons of each HeyGen Voice setting against each competitor in the volunteer self-evaluation. The mean difference is in scale points over the 43 participants with a 95% participant-bootstrap interval; wins, ties, and losses count participants;

p

is an exact two-sided Wilcoxon signed-rank test, Holm-corrected over the six comparisons ( ∗ :

p<0.05

Panel of Figure 11 — Figure 11: Sixteen of the 50 scenes in the diverse-acoustic-conditions study, grouped by the kind of acoustics they call for. Each scene is an AI-generated image of a speaker in a place; the reference voice for the speaker was designed from the image, and the same image guides HeyGen Voice and Avatar V.

Competitor	Aspect	Votes (win / same / loss)	Win share	Scenes (won / tied / lost)	$p$ (Holm)
Seedance 2.5	Overall preference	105 / 62 / 83	55.9 $\pm$ 9.2	27 / 6 / 17	0.954
	Acoustic fit	73 / 111 / 66	52.5 $\pm$ 10.7	23 / 7 / 20	1.000
	Ambient sound fit	126 / 57 / 67	65.3 $\pm$ 9.2	31 / 5 / 14	0.017 *
ElevenLabs v4	Overall preference	112 / 29 / 109	50.7 $\pm$ 8.0	22 / 6 / 22	1.000
	Acoustic fit	104 / 61 / 85	55.0 $\pm$ 8.4	26 / 4 / 20	0.954
	Ambient sound fit	136 / 55 / 59	69.7 $\pm$ 8.6	31 / 4 / 15	0.002 *

Table 14: Diverse acoustic conditions. Votes are counted from the HeyGen Voice side (win = HeyGen Voice preferred); win share is the HeyGen Voice share of decided votes with a 95% scene-bootstrap interval; scene outcomes follow the majority of a scene’s five reviews;

p

is an exact two-sided sign-flip test on each scene’s net vote, Holm-corrected over the nine tests ( ∗ :

p<0.05

). Reviewers are fixed, so intervals and tests generalize over scenes.

Voice Cloning

Voice cloning uses speech samples to reproduce a speaker’s vocal identity in generated speech across different content, languages, and styles.

See all 42 papers

Aug 16, 2026

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.

Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko +4

Kandinsky Lab, Moscow, Russia

Voice Cloning Audio-Video Generation Audio Diffusion Models

Ask PDF

May 15, 2026

Voice "Cloning" is Style Transfer

Artificially generated speech is increasingly embedded in everyday life. Voice cloning in particular enables applications where identity preservation is important, such as completing a recording, dubbing in a new language, or preserving the voices of individuals with speech loss. However, in our work, we find that despite the term, voice cloning does not faithfully ''clone'' an individual's voice. Instead, we find that widely-used voice cloning models systematically apply style transfer to source voices. As rated by human annotators, cloned voices are perceived as more authoritative, warm, customer-service-like, and human-like compared to their sources. Human annotators also report greater trust in cloned voices than source voices, and a greater willingness to disclose sensitive personal information to them. Our work furthermore shows that voice cloning leads to homogenization of speaker characteristics, as measured by reduced variance in accent, speaking rate, and the audio embedding space. Together, our results highlight a new set of limitations and risks of voice cloning technology and their potential impact on human behavior.

Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3

Cornell University · TogetherAI · Stanford University

Controllable Speech Generation Trust in AI Audio Generation Voice Cloning

Ask PDF

May 7, 2026

X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning

In this paper, we present X-Voice, a 0.4B multilingual zero-shot voice cloning model that clones arbitrary voices and enables everyone to speak 30 languages. X-Voice is trained on a 420K-hour multilingual corpus using the International Phonetic Alphabet (IPA) as a unified representation. To eliminate the reliance on prompt text without complex preprocessing like forced alignment, we design a two-stage training paradigm. In Stage 1, we establish X-Voice

_{\text{s1}}

through standard conditional flow-matching training and use it to synthesize 10K hours of speaker-consistent segments as audio prompts. In Stage 2, we fine-tune on these audio pairs with prompt text masked to derive X-Voice

_{\text{s2}}

, which enables zero-shot voice cloning without requiring transcripts of audio prompts. Architecturally, we extend F5-TTS by implementing a dual-level injection of language identifiers and decoupling and scheduling of Classifier-Free Guidance to facilitate multilingual speech synthesis. Subjective and objective evaluation results demonstrate that X-Voice outperforms existing flow-matching based multilingual systems like LEMAS-TTS and achieves zero-shot cross-lingual cloning capabilities comparable to billion-scale models such as Qwen3-TTS. To facilitate research transparency and community advancement, we open-source all related resources.

Rixi Xu, Qingyu Liu, Haitao Li +10

1MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Center for Language and Speech Processing, Johns Hopkins University · 2Shanghai Innovation Institute +4

TTS Synthesis Zero-Shot TTS Voice Cloning Multilingual TTS Synthesis

Ask PDF

Abstract

Voice Cloning

Explore similar work

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

Voice "Cloning" is Style Transfer

X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning