cs.CVOct 22, 2026

AudioLift: Audio-Driven Single Image to Full-Head Gaussian Avatar via Multi-view Video Generation

Authors: Zhuoyuan Jiang, Weijie Lyu, Xiaoyuan Wang, Tiancheng Shen, Ming-Hsuan Yang

Organizations: Yale University · University of California, Merced · Carnegie Mellon University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Existing 3D talking avatar methods mainly rely on parametric face models including 3DMM or FLAME, whose compact motion representations limit their ability to synthesize extreme expressions and to model accessories such as headwear. We introduce AudioLift, a two-stage framework that uses generated multi-view videos as the intermediate representation for audio-driven avatar generation without parametric face models or per-instance optimization. Our framework combines a synchronized multi-view video generation stage with an inconsistency-robust Gaussian reconstruction stage to recover coherent full-head geometry from a single portrait image. Extensive experiments demonstrate that AudioLift produces 3D talking avatars capable of capturing more extreme facial expressions than existing methods, with improved geometric completeness and strong multi-view consistency across the full 360∘360^\circ viewing range.

Figures & tables

Figure 1: AudioLift generates a temporally coherent sequence of 3D Gaussian talking avatars from a single portrait and audio. An audio-conditioned video model jointly generates synchronized views, which a robust Gaussian reconstructor lifts into 3D. The resulting avatars support rendering from arbitrary viewpoints at any time while preserving identity and accessories, capturing expressive motion, and generalizing to diverse in-the-wild portraits.
Figure 2: Overview of AudioLift . (a) Five synchronized videos are generated from a single portrait and speech. The clean reference view conditions the denoiser, and the side and frontal views are interleaved into one token sequence where shared self-attention couples the views. (b) Stage II lifts generated frames at each timestamp to a 3D Gaussian avatar with a fine-tuned feed-forward reconstructor. The resulting Gaussian sequence can be rendered from any viewpoint at any time.
Figure 3: Stage-II training components. (a) View-aware corruption keeps the most frontal view clean and perturbs the other four with grid distortion and camera jitter, while supervision stays uncorrupted. (b) Rear-view distillation supervises the student’s rear renders against a frozen teacher in three rear camera bins. (c) Active-volume regularization reduces opacity outside the head volume and moves outlying Gaussian centers inward.
 (a) Frontal comparison(b) Novel-view comparison
MethodPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowFID ↓\downarrowCSIM-ref ↑\uparrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
Real3D-Portrait ( Ye et al., 2024 )17.500.7000.24683.640.65010.820.5570.491
SDTalk ( Jia et al., 2026 )17.630.7100.23277.050.64510.990.5710.473
NeRFFaceSpeech ( Kim et al., 2024 )11.100.5050.466108.080.3988.510.4260.622
Ours18.620.6780.21644.530.68912.270.5820.445
Table 1: End-to-end comparison on RenderMe-360. Best results are bolded.
MethodPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowFID ↓\downarrowCSIM-ref ↑\uparrow
Real3D-Portrait ( Ye et al., 2024 )13.690.6500.34435.310.671
SDTalk ( Jia et al., 2026 )13.870.6510.33840.530.635
NeRFFaceSpeech ( Kim et al., 2024 )10.580.5610.48293.720.391
Ours13.980.6530.33529.730.666
Table 2: Frontal comparison on HDTF. Best results are bolded. Second-best results are underlined.
Figure 4: Same-frame multi-view comparison on RenderMe-360. Rows are ground truth, AudioLift , Real3D-Portrait ( Ye et al., 2024 ) , SDTalk ( Jia et al., 2026 ) , and NeRFFaceSpeech ( Kim et al., 2024 ) ; columns move from the frontal camera through oblique and profile views to the rear. The results show that AudioLift produces facial expressions closer to the ground truth. Meanwhile, AudioLift also achieves better novel-view synthesis results, and faithfully maintains the head accessories from the reference image.
Figure 5: Qualitative comparison under challenging headwear and facial expressions. The first two rows emphasize preservation of fine headwear, while the remaining rows compare expressive mouth shapes. Green boxes mark target expressions reproduced by our method; red boxes highlight missing details, boundary artifacts, or expression mismatches in the baselines.
Figure 6: Ablation study results visualization. (a) With the additionally front-view videos as input, the reconstruction results demonstrate obviously fewer artifacts. (b) Training with view-aware corruption helps model learn to resolve minor view inconsistency, active volume regularization removes the outlying Gaussian floaters, while rear distillation improves the visual results of back of the head.
Reconstruction InputsPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
4-view videos16.190.71900.2843
5-view videos20.180.77870.1803
Table 3: Quantitative ablations of (a) Stage I: number of multi-view videos and (b) Stage II: effectiveness of view-aware corruption. Best results are bolded.
Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

MethodMouth-opening contrast ↑\uparrow
Ours0.166
NeRFFaceSpeech0.122
Real3D-Portrait0.084
SDTalk0.078
Appendix
Table 4: Mouth-opening contrast on all 16 in-the-wild clips, measured as the mean mouth opening at open vowels minus the mean opening at the bilabials /m/ , /b/ , /p/ . Phoneme times are selected automatically within the curated set. Implementation details are in Appendix B.11 .
Figure 7: Van Gogh self-portrait, English speech. Our method opens the mouth on “now”, closes the lips on the /p/ of “penetrate”, and rounds them on “wild”; the three baselines keep almost the same mouth shape throughout.
Figure 8: Anime portrait, English speech. Five phonemes give five distinct mouth shapes in our result: spread lips on “Taylor”, closed lips on “supported”, rounded lips on “through”, open on “college”, and wide open on “cotton”. Real3D-Portrait also shifts the colour of the drawing.
Figure 9: Photographic portrait, English speech. Only our method purses the lips for the /w/ of “without”, closes them for the /p/ of “positively”, and opens the jaw on “understand”.
Figure 10: Heavy makeup and earrings, sung English. Our method reproduces all four target mouth shapes, while the baselines stay close to a single shape.
Figure 11: Wide-brimmed hat and face paint, English speech. Our method presses the upper teeth to the lower lip for /v/ and closes the lips for /b/ ; NeRFFaceSpeech blurs the hat.
Figure 12: Renaissance painting, Swahili speech. NeRFFaceSpeech shows a grid pattern across the face.
Figure 13: Child portrait, English speech.
Figure 14: Photographic portrait, sung Japanese.
Figure 15: Portrait with drop earrings and a raised hand, English speech.
Figure 16: Temporal consistency over long windows. (a) Van Gogh self-portrait, English speech. (b) Wide-brimmed hat and face paint, English speech. (c) Anime portrait, English speech.
Figure 17: More than twenty seconds of driving audio. (a) 25 s of sung Japanese. (b) 22.6 s of English speech.
Figure 18: Van Gogh self-portrait, English speech, five azimuths.
Figure 19: Renaissance painting, Swahili speech, five azimuths.
Figure 20: Wide-brimmed hat and face paint, English speech, five azimuths.
Figure 21: Anime portrait, English speech, five azimuths.
ClipReference imageDriving audioLength
009FFHQ #47591, photograph by Danny Bollinger; CC BY 2.0LibriSpeech 1089-134686-0002 , English; CC BY 4.06.6
011FFHQ #53017, photograph by Maverick L.; CC BY 2.0GTSinger IT-Bass-1 , Di Provenza il mar, il suol , Italian; CC BY-NC-SA 4.09.7
014FFHQ #00555, photograph by Joseph Liao; CC BY-NC 2.0LibriSpeech 2094-142345-0000 , English; CC BY 4.011.0 / 22.6
020FFHQ #58471, photograph by Angelo & Mantia Photography; CC BY 2.0LibriSpeech 3729-6852-0001 , English; CC BY 4.012.2
022FFHQ #42543, photograph by April Killingsworth; CC BY 2.0GTSinger EN-Alto-1 , someday or oneday , English; CC BY-NC-SA 4.010.6
025AI-generated illustration (2022) by Benlisquare, Commons; public domainLibriSpeech 1995-1826-0002 , English; CC BY 4.04.6
Appendix
Table 5: Sources and licenses of the in-the-wild pairs. Length is the duration of the driving audio in seconds. The second length of clips 014 and 090 belongs to their long-sequence version. Commons stands for Wikimedia Commons.
Dataset and useSelected source interval
NeRSemble (training)0.00–3.24 seconds of each retained SEN recording.
RenderMe-360 (evaluation)5.00–8.24 seconds of each session, shared by all 42 clips.
HDTF (evaluation)A subject-specific 3.24-second window within an official target-speaker interval, starting at a speech segment when possible; for example, 13.74–16.98 seconds for AnnMarieBuerkle.
Appendix
Table 6: Selection of the 81-frame windows. Start times refer to the source recording, not to the generated video.
 Frame 0Frames 1–80
ModelPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
Released FaceLift17.920.74730.262717.250.72910.2831
Fine-tuned w/o corruption17.730.75150.246617.090.73280.2672
Fine-tuned w/ corruption18.000.75450.241217.280.73390.2635
Full 12-camera model18.030.75430.244217.310.73360.2666
Appendix
Table 7: Generalization to four held-out NeRSemble cameras using the 12-camera training variant. All checkpoints receive the same generated five-view inputs. Frame 0 and frames 1–80 are evaluated separately. Bold indicates the best result among the first three rows; the full 12-camera model is listed separately for reference.
Figure 22: Reference-conditioning robustness for degraded frontal, off-frontal, and standard frontal references. Both strategies produce plausible outputs for the standard input, with frame-wise conditioning retaining sharper details; its robustness advantage becomes more pronounced for the two challenging references.
TensorShape
Canvas video latentB×16×21×152×104B\times 16\times 21\times 152\times 104
Frontal video latentB×16×21×76×52B\times 16\times 21\times 76\times 52
Canvas tokens, with contextB×22×3952×1536B\times 22\times 3952\times 1536
Frontal tokens, with contextB×22×988×1536B\times 22\times 988\times 1536
Flattened transformer inputB×108680×1536B\times 108680\times 1536
Appendix
Table 8: Stage-I tensor shapes for an 81-frame clip, with batch size BB . Token shapes include one additional context position: a clean reference latent for the canvas and a zero latent for the frontal stream, each with the corresponding stream’s spatial dimensions.
Figure 23: Stage-I audio-driven multi-view diffusion. A clean reference conditions five jointly denoised views through frame-wise token packing and shared self-attention.
Figure 24: Stage-II Gaussian lifting and training curriculum.
PhaseUpdatesNominal batchWarmupLearning-rate schedule
Four-view warm start9,5998500Constant 10−410^{-4} after warmup
Five-view adaptation4,2008400Constant 5×10−45\times 10^{-4} after warmup
Continuation1,600850Constant 10−410^{-4} after warmup
Cooldown1,00080Linear 10−4→010^{-4}\to 0 over 2,000 updates
Appendix
Table 9: Stage-I training schedule starting from released StableAvatar weights. Rows are applied sequentially, and update counts include warmup.
PhaseUpdatesBatchPeak LRHorizonλrear\lambda_{\mathrm{rear}}λAV\lambda_{\mathrm{AV}}
Camera/background adaptation3,240168×10−58\times 10^{-5}3,24000
View-aware corruption3,240168×10−58\times 10^{-5}3,24000
Active-volume refinement501210−510^{-5}86400.1
Rear-prior distillation6481610−510^{-5}3,2400.20
Joint rear/active-volume refinement1501610−510^{-5}3,2400.20.1
Appendix
Table 10: Stage-II training schedule, totaling 7,328 selected updates. Horizon denotes the total cosine-scheduler length. Each phase has 20 warmup updates. View-aware corruption remains enabled after its introduction.
More in this topic

3D Avatar Generation

3D avatar generation creates realistic, controllable digital humans from images, video, text, or other inputs.

See all 90 papers

Explore similar work

CardsList

  1. MegaAvatar: Controllable Talking Avatar Generation

    Sep 30, 2026Junyao Gao, Sibo Liu, Weidong Zhang +2Talking Head Generation3D Avatar Generation

    Tencent · Tongji University

  2. LiftAvatar: Kinematic-Space Completion for Expression-Controlled 3D Gaussian Avatar Animation

    Mar 2, 2026Hualiang Wei, Shunran Jia, Jialun Liu +1Facial Expression SynthesisHuman Avatar Animation

    College of Computer Science and Technology, Jilin University, No. 2699 Qianjin Street, Changchun, 130012, Jilin, China · Impressed Inc DBA SocialBook, 950 Tower Ln, Foster City, 94404, California, USA · Institute of Artificial Intelligence of China Telecom, Zhonghui Building, Dongcheng District, 100007, Beijing, China

  3. 3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars

    Feb 11, 2026Zhongju Wang, Zhenhong Sun, Beier Wang +4Lip Synchronization3D Avatar Generation

    University of New South Wales · Australian National University · Vertex Lab +1