HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos

Pavel Andreev; Andrei Gusev; Adithya Narayan; Yi Ren; Ivan Somov; Zijian Li; Sergey Demyanov; Viktor Goriachko

cs.SDOct 21, 2026

HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos

Authors: Pavel Andreev, Andrei Gusev, Adithya Narayan, Yi Ren, Ivan Somov, Zijian Li, Sergey Demyanov, Viktor Goriachko,

Organizations: HeyGen Research

Abstract

Most voice-cloning systems are built for audio alone: they copy the reference recording's delivery and acoustics, so a voice cloned on a phone sounds like a phone recording whether the speaker is pictured in a studio, on a street, or in a cathedral. Speech for a talking video should sound like the pictured person speaking in the pictured place. We present HeyGen Voice, a scene-aware voice-cloning model that separates who is speaking from how and where: it keeps the speaker's voice from an everyday recording while the target image sets the delivery and acoustics. Prior image-conditioned TTS does half of this job: it either adds scene acoustics without keeping a specific speaker or infers a voice from a face. HeyGen Voice is a multimodal autoregressive transformer conditioned on the image, speaker and scene captions, and the voice recording, trained in five stages that move from multilingual TTS pretraining through simulated and real cross-scene adaptation to reinforcement and preference optimization. Two clone policies trade scene adaptation against fidelity to the original recording, and an audio anchor holds long scripts in one voice and one acoustic environment. Listeners prefer HeyGen Voice over leading commercial systems across a third-party arena, production votes, and talking-video studies: it ranks first (Elo 1201) on the public Artificial Analysis controlled-voice TTS leaderboard (October 2026); in a blind test, users cloning their own voice chose it over ElevenLabs v4 and Cartesia Sonic 3.6 in 62% and 54% of votes; and volunteers rated talking videos of their own digital twin about 10 to 30 points higher (out of 100) when voiced by HeyGen Voice than by leading commercial alternatives. Try HeyGen Voice and hear samples at https://heygen.com/heygen-voice.

Figures & tables

Panel of Table 1 — Table 1: Capabilities by method category.

Panel of Figure 1 — Figure 1: HeyGen Voice architecture. Multimodal inputs are encoded by frozen encoders and pooled into conditioning tokens; the clone policy determines whether the continuation state comes from an anchor designed for the pictured scene or from the reference recording itself. The conditioning prefix, script tokens, and acoustic codes form one sequence processed by an autoregressive transformer decoder. The model predicts acoustic codes, supervised by cross-entropy during supervised training, by intelligibility and speaker-similarity rewards during GRPO, and by preference pairs during DPO; a flow-matching decoder conditioned on the transformer’s hidden states and the reference audio produces the waveform.

Panel of Figure 2 — Figure 2: Example speaker and background captions for one recording, a speaker talking inside a moving car. Each caption is a short descriptive paragraph; the background caption names the sound sources, the space, and what stays distant, and the speaker caption covers voice, tone, pace, expressiveness, and articulation.

Source	Role
Target image	Person and scene context
Speaker caption	Delivery and vocal attributes
Background caption	Environment and acoustics
Reference tokens	Speaker voice: timbre, accent, and speaking cadence
Consistency embedding	Shared voice and scene acoustics
Local acoustic context	Local acoustic continuity

Table 2: Conditioning prefix of the acoustic model. Each source contributes a small, fixed number of pooled tokens; script and continuation lengths vary.

Panel of Figure 3 — Figure 3: Training curriculum. Talking-video data drives the adaptation stages. The GRPO checkpoint is the release base model and the base for the expressiveness adapter, which is an optional inference control.

Panel of Figure 4 — Figure 4: One voice reference, multiple target scenes. An everyday phone recording supplies the speaker reference. For each target image and the captions written from it, HeyGen Voice generates speech that retains that voice and adapts delivery and acoustics to the requested setting; the designed audio anchors subsequent utterances within the scene. Images depict a fictional speaker; video rendering is a separate downstream stage.

Clone policy	Behavior
voice-clone	Clones the voice: timbre, accent, and speaking cadence. Delivery and acoustics follow the image, captions, and acoustic controls.
full-clone	Clones the voice and the acoustic conditions of the reference. Highest similarity to the original recording.

Table 3: Clone policies, named for what they clone from the reference.

Evaluation	Benchmark	Main evidence
Audio-based	Cross-scene (Section 5.1 )	Speaker similarity and audio–audio and audio–image alignment on 331 cases
	Multilingual (Section 5.2 )	Ten-language CER with corrected scripts and transcription review
	Listening tests (Section 5.3 )	Paired human judgments of naturalness, accent, clarity, narrator choice, and reference similarity in seven system pairs on 245 cases (42,875 ratings)
	Production voice cloning (Section 5.4 )	Blind pairwise and three-way picks by platform users cloning their own voice, vs. ElevenLabs v4 and Cartesia Sonic 3.6
	Public arena (Section 5.5 )	Artificial Analysis controlled-voice leaderboard: blind pairwise listener votes on eight shared cloned voices, Elo ranking
Video-based	Volunteer self-evaluation (Section 5.6 )	43 participants rate talking videos of themselves, vs. Cartesia Sonic 3.6, ElevenLabs v4, and Seedance 2.5

Table 4: Evaluation overview. Automatic audio–image similarity is a diagnostic of the generated speech; human judgments of talking videos are collected in the video-based studies.

System / clone policy	Speaker similarity $\uparrow$		ImageBind $\uparrow$
	To target	To reference	Audio–audio	Audio–image
HeyGen Voice ( voice-clone )	0.590	0.739	0.637	0.282
HeyGen Voice ( full-clone )	0.581	0.757	0.608	0.278
Cartesia Sonic 3.6	0.582	0.745	0.599	0.261
ElevenLabs v4	0.556	0.712	0.607	0.269
Inworld Realtime TTS-2	0.464	0.567	0.544	0.231

Table 5: Cross-scene benchmark on 331 matched cases. Speaker similarity is measured against the real target-scene recording and against the original voice reference. Best value per column in bold, second best underlined.

Language	HeyGen full-clone	Qwen3-TTS 1.7B	Cartesia Sonic 3.6	ElevenLabs v4	Inworld TTS-2
English	0.0000	0.0000	0.0089	0.0000	0.0000
Chinese	0.0000	0.0286	0.0526	0.0000	0.0492
Spanish	0.0000	0.0000	0.0000	0.0000	0.0109
French	0.0000	0.0000	0.1389	0.0000	0.0000
German	0.0000	0.0114	0.0827	0.0000	0.0000
Italian	0.0099	0.0270	0.0393	0.0238	0.0099

Table 6: Agreement-only CER (%, lower is better) on the corrected multilingual benchmark; 100 utterances per language and system, macro average over languages. Vendor versions follow the benchmark labels.

Figure 5: Target images of the listening benchmark. Each speaker’s image is adapted to the same podcast-studio template, which guides HeyGen Voice toward a professional narrator’s delivery in a treated room. The four faces shown are AI-generated examples adapted in the same way.

Target language	Intra-lingual	Cross-lingual	Total
English (US)	22	25	47
Spanish (Mexico)	25	24	49
German (Germany)	25	25	50
French (France)	25	24	49
Chinese (China)	45	5	50
Total	142	103	245

Table 7: Listening benchmark composition. Counts are unique reference–script cases, reused in every system comparison.

Aspect	Question
Reference similarity	Which candidate sounds more similar to the reference? (five positions, $-2$ to $+2$ )
Narrator choice	Which voice would you choose for an entire educational lesson narrated by a presenter?
Naturalness	Which voice sounds more human and natural, ignoring the spoken content?
Native accent	Which voice sounds more native in pronunciation, rhythm, and stress?
Clarity	Which recording is clearer and higher quality (muffling, noise, distortion, glitches)?

Table 8: Listening rubric. Descriptions paraphrase the English study questions; localized equivalents are used in the other four locales.

Matchup (A / B)	Reference similarity	Narrator choice	Naturalness	Native accent	Clarity
voice-clone / ElevenLabs v4	43.1 * $\pm$ 3.3	61.5 * $\pm$ 3.4	50.2 $\pm$ 3.6	55.7 $\pm$ 4.1	78.4 * $\pm$ 2.9
voice-clone / Cartesia Sonic 3.6	22.7 * $\pm$ 2.4	77.9 * $\pm$ 2.8	69.6 * $\pm$ 3.4	77.8 * $\pm$ 3.2	86.6 * $\pm$ 2.5
voice-clone / Inworld TTS-2	44.9 $\pm$ 3.5	68.1 * $\pm$ 3.3	63.9 * $\pm$ 3.4	70.9 * $\pm$ 3.5	63.5 * $\pm$ 3.5
full-clone / ElevenLabs v4	63.0 * $\pm$ 3.6	47.3 $\pm$ 3.7	43.9 * $\pm$ 3.7	45.8 $\pm$ 4.5	65.5 * $\pm$ 4.0
full-clone / Cartesia Sonic 3.6	53.8 $\pm$ 3.2	60.8 * $\pm$ 3.9	46.0 $\pm$ 3.9	67.9 * $\pm$ 4.5	75.0 * $\pm$ 3.4
voice-clone / full-clone	27.9 * $\pm$ 3.3	73.6 * $\pm$ 3.3	58.6 * $\pm$ 4.2	62.2 * $\pm$ 5.0	77.3 * $\pm$ 3.5

Table 9: Share of decided votes (%) for the first-named system (A) in each pair, with 95% whole-case bootstrap intervals, on 245 cases and 1,225 ratings per aspect and pair. Ties are excluded from the denominator. ∗ : Holm-corrected

p<0.05

over the 35 pooled tests.

System	Reference similarity	Narrator choice	Naturalness	Native accent	Clarity
HeyGen voice-clone	1000 (anchor)	1000 (anchor)	1000 (anchor)	1000 (anchor)	1000 (anchor)
HeyGen full-clone	1144 $\pm$ 17	904 $\pm$ 14	951 $\pm$ 12	964 $\pm$ 11	907 $\pm$ 14
ElevenLabs v4	1060 $\pm$ 17	935 $\pm$ 15	995 $\pm$ 13	985 $\pm$ 12	860 $\pm$ 16
Cartesia Sonic 3.6	1141 $\pm$ 15	836 $\pm$ 19	934 $\pm$ 14	878 $\pm$ 18	811 $\pm$ 15
Inworld TTS-2	1032 $\pm$ 22	896 $\pm$ 21	931 $\pm$ 18	909 $\pm$ 18	939 $\pm$ 17

Table 10: Bradley–Terry scores on the Elo scale, fitted separately per aspect to the seven pairs over all 245 cases with voice-clone fixed at 1000, with 95% whole-case bootstrap intervals. Inworld TTS-2 is placed through its single comparison with voice-clone .

Figure 6: Narrator choice against reference similarity on the Elo scale, pooled over 245 cases, with 95% whole-case bootstrap intervals; the dashed line joins the non-dominated systems. The benchmark pairs user-like references with a studio scene, so the two questions trade off: full-clone keeps the reference and gives up narrator quality, and voice-clone sounds like a professional narrator and gives up reference similarity.

Figure 7: Listening profiles, pooled and for English, Spanish, and German. Each spoke is scaled to its own displayed range, printed at the spoke; numerical keys give the fitted Elo-scale scores with voice-clone at 1000. Polygon areas are not comparable across panels.

Figure 8: Listening profiles for French and Chinese, and for the 142 intra-lingual and 103 cross-lingual cases. Subset fits use the same model as Table 10 ; their bootstrap intervals are wider (typically

\pm

35 points per language and

\pm

20 per condition).

Comparison	Votes	Chance	ElevenLabs v4	Cartesia Sonic 3.6	HeyGen Voice
Pairwise	8,332	50.0	37.8 $\pm$ 1.1	–	62.2 $\pm$ 1.1
Pairwise	7,286	50.0	–	45.9 $\pm$ 1.2	54.1 $\pm$ 1.2
Three-way	6,153	33.3	23.0 $\pm$ 1.1	35.3 $\pm$ 1.2	41.7 $\pm$ 1.3

Table 11: Blind side-by-side picks in the HeyGen voice-cloning flow (snapshot of October 2, 2026). Each vote is one user choosing the clone of their own voice; HeyGen Voice runs the full-clone policy. Entries are pick shares in percent with Wilson 95% intervals over picks; the largest share in each row is bold. Chance is the share expected from uniform random picking, and every HeyGen Voice share exceeds it (exact binomial test,

p<0.001

Figure 9: Top ten of the Artificial Analysis controlled-voice text-to-speech leaderboard, snapshot of October 9, 2026. Elo scores with the leaderboard’s 95% intervals; the right column is the leaderboard’s rank range. All systems clone the same eight reference voices; HeyGen Voice runs voice-clone .

System	Mean score	Centered mean	Mean rank	Top-1 ballots
HeyGen Voice ( voice-clone )	76.1 $\pm$ 5.3	+12.7 $\pm$ 3.5	2.19	18
HeyGen Voice ( full-clone )	72.6 $\pm$ 5.8	+9.3 $\pm$ 3.8	2.41	13
Cartesia Sonic 3.6	63.1 $\pm$ 7.2	-0.2 $\pm$ 4.7	2.99	12
ElevenLabs v4	59.2 $\pm$ 7.6	-4.2 $\pm$ 5.0	3.26	9
Seedance 2.5	45.7 $\pm$ 6.9	-17.6 $\pm$ 4.3	4.16	2

Table 12: Volunteer self-evaluation: 43 participants each rated one talking video of themselves per system on a 1–100 scale. Mean scores carry 95% participant-bootstrap intervals. The centered score subtracts the participant’s mean over the five systems, so it removes differences in how participants use the scale; its intervals use a common 88% level following the Goldstein–Healy approach [ 29 ] , providing an approximate visual guide to unadjusted paired comparisons. Formal significance is assessed using the paired tests in Table 13 . Mean rank averages each participant’s ranking of the five systems (1 = best, ties share the average rank); top-1 ballots count the participants who gave the system their highest score, crediting every system tied at the top.

Figure 10: Centered scores in the volunteer self-evaluation with 88% Goldstein–Healy intervals for approximate visual comparison. Formal paired tests are reported in Table 13 . Zero is the participant’s mean over the five systems.

HeyGen Voice setting	Competitor	Mean difference	Wins / ties / losses	$p$ (Holm)
voice-clone	Cartesia Sonic 3.6	+13.0 $\pm$ 9.0	27 / 0 / 16	0.016 *
voice-clone	ElevenLabs v4	+16.9 $\pm$ 7.4	29 / 3 / 11	$<$ 0.001 *
voice-clone	Seedance 2.5	+30.3 $\pm$ 7.6	39 / 1 / 3	$<$ 0.001 *
full-clone	Cartesia Sonic 3.6	+9.5 $\pm$ 6.9	24 / 5 / 14	0.016 *
full-clone	ElevenLabs v4	+13.4 $\pm$ 9.7	29 / 2 / 12	0.016 *
full-clone	Seedance 2.5	+26.9 $\pm$ 8.1	35 / 2 / 6	$<$ 0.001 *

Table 13: Paired comparisons of each HeyGen Voice setting against each competitor in the volunteer self-evaluation. The mean difference is in scale points over the 43 participants with a 95% participant-bootstrap interval; wins, ties, and losses count participants;

p

is an exact two-sided Wilcoxon signed-rank test, Holm-corrected over the six comparisons ( ∗ :

p<0.05

Panel of Figure 11 — Figure 11: Sixteen of the 50 scenes in the diverse-acoustic-conditions study, grouped by the kind of acoustics they call for. Each scene is an AI-generated image of a speaker in a place; the reference voice for the speaker was designed from the image, and the same image guides HeyGen Voice and Avatar V.

Competitor	Aspect	Votes (win / same / loss)	Win share	Scenes (won / tied / lost)	$p$ (Holm)
Seedance 2.5	Overall preference	105 / 62 / 83	55.9 $\pm$ 9.2	27 / 6 / 17	0.954
	Acoustic fit	73 / 111 / 66	52.5 $\pm$ 10.7	23 / 7 / 20	1.000
	Ambient sound fit	126 / 57 / 67	65.3 $\pm$ 9.2	31 / 5 / 14	0.017 *
ElevenLabs v4	Overall preference	112 / 29 / 109	50.7 $\pm$ 8.0	22 / 6 / 22	1.000
	Acoustic fit	104 / 61 / 85	55.0 $\pm$ 8.4	26 / 4 / 20	0.954
	Ambient sound fit	136 / 55 / 59	69.7 $\pm$ 8.6	31 / 4 / 15	0.002 *

Table 14: Diverse acoustic conditions. Votes are counted from the HeyGen Voice side (win = HeyGen Voice preferred); win share is the HeyGen Voice share of decided votes with a 95% scene-bootstrap interval; scene outcomes follow the majority of a scene’s five reviews;

p

is an exact two-sided sign-flip test on each scene’s net vote, Holm-corrected over the nine tests ( ∗ :

p<0.05

). Reviewers are fixed, so intervals and tests generalize over scenes.

Voice Cloning

Voice cloning uses speech samples to reproduce a speaker’s vocal identity in generated speech across different content, languages, and styles.

See all 42 papers

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
Aug 16, 2026Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko +4Voice Cloning Audio-Video Generation
Kandinsky Lab, Moscow, Russia
Voice "Cloning" is Style Transfer
May 15, 2026Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3Controllable Speech Generation Trust in AI
Cornell University · TogetherAI · Stanford University
X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning
May 7, 2026Rixi Xu, Qingyu Liu, Haitao Li +10TTS Synthesis Zero-Shot TTS
1MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Center for Language and Speech Processing, Johns Hopkins University · 2Shanghai Innovation Institute +4

Aug 16, 2026J/K move · Enter open · S save

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko, Alexey Letunovskiy+3

Kandinsky Lab, Moscow, Russia

Voice Cloning Audio-Video Generation Audio Diffusion Models

Open paper AskPDF arXiv

Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.

Abstract

Voice Cloning

Explore similar work

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

Voice "Cloning" is Style Transfer

X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning