Retrieval-Augmented Gaussian Avatars: Improving Expression Generalization

Matan Levy; Gavriel Habib; Issar Tzachor; Dvir Samuel; Rami Ben-Ari; Nir Darshan; Or Litany; Dani Lischinski

cs.CVMar 9, 2026

Retrieval-Augmented Gaussian Avatars: Improving Expression Generalization

Authors: Matan Levy, Gavriel Habib, Issar Tzachor, Dvir Samuel, Rami Ben-Ari, Nir Darshan, Or Litany, Dani Lischinski

Organizations: The Hebrew University of Jerusalem, Israel · OriginAI, Israel · Bar-Ilan University, Israel · NVIDIA ·

Abstract

Template-free animatable head avatars can achieve high visual fidelity by learning expression-dependent facial deformation directly from a subject's capture, avoiding parametric face templates and hand-designed blendshape spaces. However, since learned deformation is supervised only by the expressions observed for a single identity, these models suffer from limited expression coverage and often struggle when driven by motions that deviate from the training distribution. We introduce RAF (Retrieval-Augmented Faces), a simple training-time augmentation designed for template-free head avatars that learn deformation from data. RAF constructs a large unlabeled expression bank and, during training, replaces a subset of the subject's expression features with nearest-neighbor expressions retrieved from this bank while still reconstructing the subject's original frames. This exposes the deformation field to a broader range of expression conditions, encouraging stronger identity-expression decoupling and improving robustness to expression distribution shift without requiring paired cross-identity data, additional annotations, or architectural changes. We further analyze how retrieval augmentation increases expression diversity and validate retrieval quality with a user study showing that retrieved neighbors are perceptually closer in expression and pose. Experiments on the NeRSemble benchmark demonstrate that RAF consistently improves expression fidelity over the baseline, in both self-driving and cross-driving scenarios.

Figures & tables

Panel of Figure 1 — Figure 1: Cross-identity qualitative comparison. The avatars are taken from the NeRSemble dataset [ 15 ] . Compared to the baselines, RAF produces expressions that more closely resemble those of the driving subject.

Subject	MMD $\downarrow$		KL $\downarrow$		B2T Dist $\downarrow$
Subject	Vanilla	+RAF	Vanilla	+RAF	Vanilla	+RAF
388	0.254	0.216	38.948	25.365	1.398	0.796
422	0.186	0.128	50.575	31.425	1.001	0.560
443	0.131	0.103	33.078	19.658	0.763	0.533
445	0.152	0.089	56.310	32.658	0.872	0.499
475	0.213	0.167	35.662	21.417	1.023	0.583

Table 1: Coverage analysis of expression features. Applying RAF substantially improves coverage of the test expression distribution.

Figure 2: PCA visualization of the expression bank. We project expression features into 2D with PCA. For three query expressions (stars), we highlight their

10

nearest neighbors computed in the original BFM expression-feature space. Despite the projection, each query’s neighbors form a cluster of similar expressions, even though they belong to different identities. This indicates that the expression bank contains meaningful cross-identity matches, which enables reliable nearest-neighbor retrieval for expression substitution during training.

Figure 3: Human preference of most similar match to a query face, reported separately for head pose and facial expression. Nearest neighbors (NN) retrieved from a large/small expression bank, built from the same set of identities but with fewer frames.

Figure 4: An overview of our method. During training, we randomly replace 50% of each subject’s expression features with those of their nearest neighbors originating from different identities. This retrieval-based substitution encourages the model to generalize expressions across identities and disentangle expression from appearance. The avatar network (MLP) is subsequently trained to reconstruct the original frame conditioned on the substituted expression features, optimized under a reconstruction objective.

Method	Self-Driving					Cross-Driving
Method	$\uparrow$ PSNR	$\uparrow$ SSIM	$\downarrow$ AED	$\downarrow$ APD	$\uparrow$ Emotion Sim.	$\downarrow$ AED	$\downarrow$ APD	$\uparrow$ Emotion Sim.
Original	18.038	0.7881	0.24227	0.13409	0.82016	0.27037	0.12289	0.78746
Random Noise	18.004	0.7890	0.24998	0.13023	0.81604	0.26920	0.12105	0.78770
Ours (RAF)	18.002	0.7917	0.23094	0.12932	0.85595	0.26164	0.12687	0.80801

Table 2: Quantitative comparison of our method with the original and random-noise baselines on a test set from the NeRSemble dataset [ 15 ] . Best results are in bold. RAF outperforms both baselines across most metrics in both self- and cross-driving settings, despite being designed primarily for the cross-driving task. In particular, RAF achieves a substantial improvement in AED under cross-driving, indicating a more accurate reproduction of the driver’s expressions.

Panel of Figure 5 — Figure 5: Self-driving qualitative comparison. Compared to the baselines, RAF produces unseen expressions that more closely resemble those of the subject.

Method	Self-Driving					Cross-Driving
Method	$\uparrow$ PSNR	$\uparrow$ SSIM	$\downarrow$ AED	$\downarrow$ APD	$\uparrow$ Emotion Sim.	$\downarrow$ AED	$\downarrow$ APD	$\uparrow$ Emotion Sim.
Half bank #1	18.070	0.7926	0.23513	0.13226	0.85169	0.27079	0.12495	0.80305
Half bank #2	18.137	0.7941	0.24693	0.13225	0.85616	0.26828	0.12295	0.80733
RAF (top 5)	-	-	0.23927	0.17058	0.88228	0.26687	0.15089	0.83073
RAF	18.002	0.7917	0.23094	0.12932	0.85595	0.26164	0.12687	0.80801

Table 3: Ablation study on a test set from the NeRSemble dataset [ 15 ] . The expression bank for nearest-neighbor retrieval was limited to approximately half of the identities.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Screenshot of the Amazon Mechanical Turk (AMT) user study interface. Each HIT presents two pairs of face images (Left Pair and Right Pair), each consisting of a Reference image and a Candidate. Workers answer two forced-choice questions—one on head pose similarity and one on facial expression similarity—while being instructed to ignore identity.

Human Avatar Animation

Human avatar animation builds digital people that can be reconstructed, driven by motion or speech, and rendered consistently across poses and views.

See all 91 papers

Self-Learning Expression Deformations for Data-Efficient Gaussian Avatars
Jun 4, 2026Jiahao Yang, Xiaohang Yang, Qing Wang +33D Gaussian Splatting Self-Supervised Learning
Queen Mary University of London
RegHead: Non-Humanoid Head Blendshapes via Feed-Forward Registration
Jul 13, 2026Jiahao Luo, Hao Zhang, Jianqi Chen +93D Avatar Generation Non-Rigid Shape Correspondence
1UCSC · 2UIUC · 3KAUST +1
FFAvatar: Few-Shot, Feed-Forward, and Generalizable Avatar Reconstruction
May 14, 2026Thuan Hoang Nguyen, Jiahao Luo, Yinyu Nie +33D Gaussian Splatting 3D Reconstruction
Snap Inc. · MBZUAI · University of California, Santa Cruz

Jun 4, 2026J/K move · Enter open · S save

Self-Learning Expression Deformations for Data-Efficient Gaussian Avatars

Jiahao Yang, Xiaohang Yang, Qing Wang, Yilan Dong+2

Queen Mary University of London

3D Gaussian Splatting Self-Supervised Learning Dynamic Scene Reconstruction 3D Avatar Generation Human Avatar Animation 3D Face Reconstruction

Open paper AskPDF arXiv

Modeling dynamic facial expressions using 3D Gaussian representations remains challenging due to their unstructured nature. Conventional Gaussian avatar pipelines require extensive multiview and sequential expression data, limiting scalability and accessibility. In this work, we introduce Self-Adaptive Gaussian Expression (SAGE), a framework for self-learning expression-induced Gaussian deformations that enables high-fidelity, animatable avatars from minimal input data. Our method jointly optimizes 2D Gaussian surfels and a Signed Distance Field (SDF) to enforce compact, surface-aligned Gaussian distributions, while a self-supervised expression learning phase replaces long training sequences with geometric and appearance consistency constraints. This design allows flexible deployment across multiple reconstruction regimes: in the multiview setting, only a single frame (timestep) is required instead of thousands; in the monocular setting, only head rotations are needed without expression sequences; and in the one-shot setting, no pretraining or priors are necessary. Experiments demonstrate that our approach achieves reconstruction and animation quality comparable to state-of-the-art methods, while reducing data requirements by several orders of magnitude. Our results highlight the potential of self-supervised Gaussian deformation learning as a step toward accessible, data-efficient avatar creation.

Abstract

Human Avatar Animation

Explore similar work

Self-Learning Expression Deformations for Data-Efficient Gaussian Avatars

RegHead: Non-Humanoid Head Blendshapes via Feed-Forward Registration

FFAvatar: Few-Shot, Feed-Forward, and Generalizable Avatar Reconstruction