High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions

Yikang He; Jichao Zhang; Wei Wang; Nicu Sebe; Yao Zhao

cs.CVMar 16, 2026

High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions

Authors: Yikang He, Jichao Zhang, Wei Wang, Nicu Sebe, Yao Zhao

Organizations: Institute of Information Science, Beijing Jiaotong University, Beijing, China. · School of Computer Science, Ocean University of China, Qingdao, China. · Department of Information Engineering and Computer Science (DISI), University of Trento, Italy.

Abstract

Facial expression editing methods can be mainly categorized into two types based on their architectures: 2D-based and 3D-based methods. The former lacks 3D face modeling capabilities, making it difficult to edit 3D factors effectively. The latter has demonstrated superior performance in generating high-quality and view-consistent renderings using single-view 2D face images. Although these methods have successfully used animatable models to control facial expressions, they still have limitations in achieving precise control over fine-grained expressions. To address this issue, in this paper, we propose a novel approach by simultaneously refining both the latent code of a pretrained 3D-Aware GAN model for texture editing and the expression code of the driven 3DMM model for mesh editing. Specifically, we introduce a Dual Mappers module, comprising Texture Mapper and Emotion Mapper, to learn the transformations of the given latent code for textures and the expression code for meshes, respectively. To optimize the Dual Mappers, we propose a Text-Guided Optimization method, leveraging a CLIP-based objective function with expression text prompts as targets, while integrating a SubSpace Projection mechanism to project the text embedding to the expression subspace such that we can have more precise control over fine-grained expressions. Extensive experiments and comparative analyses demonstrate the effectiveness and superiority of our proposed method.

Figures & tables

Fig. 1: Our model can generate high-quality, view-consistent face images while enabling fine-grained expression editing. The first row displays the generated images of different facial expression edits (with the reference image located in the lower-right corner), the second row presents different viewpoints, and the third row showcases the generated 3D head meshes.

Fig. 2: Overview of the proposed architecture which consists of two main modules: (a) Dual Mappers and (b) Text-Guided Optimization . In (a), Dual Mappers refine the random latent code $\boldsymbol{w}$ and expression code $\boldsymbol{\alpha_{R}}$ from a reference image $\boldsymbol{I_{R}}$ using a Cross Attention module and MLP layers, with updates stabilized by skip connections and a $L_{2}$ Loss . The refined codes $\boldsymbol{w^{\prime}}$ and $\boldsymbol{\alpha_{R}^{\prime}}$ are passed to a frozen synthesis network to generate the edited image. (The red arrow represents replacing $\boldsymbol{w}$ , $\boldsymbol{\alpha_{I}}$ with $\boldsymbol{w^{\prime}}$ , $\boldsymbol{\alpha_{R}^{\prime}}$ .) In (b), Text-Guided Optimization aligns the edited image with a text description ( e.g. , “A person who is raising brow”). A Projection module maps the text embedding $\boldsymbol{e_{T}}$ to the image embedding space, enhancing compatibility. Cosine Similarity Loss aligns the embeddings, while Identity Loss preserves facial identity.

Fig. 3: The Synthesis Network utilizes Tri-Plane and Volume Rendering to achieve 3D-aware head avatar generation. The generator

\boldsymbol{G_{uv}}

produces a Tri-Plane from the latent code

\boldsymbol{w}

, while FLAME accepts

\boldsymbol{\alpha}

\boldsymbol{\theta}

\boldsymbol{\beta}

as input and provides a coarse head mesh aligned via rasterization. An MLP predicts color

\boldsymbol{c}

and density

\boldsymbol{\sigma}

, which are used for Volume Rendering . Finally, a Super-Resolution module enhances the output, creating realistic, editable 3D avatars.

Fig. 4: Fine-grained expression editing comparison with state-of-the-art animatable 3D image synthesis methods. To ensure a strict and direct comparison, methods supporting reference-guided editing within the same domain (Next3D, Diffusion-rig, and Ours) are evaluated using the exact same identity. For unconditional generative models (DiscoFaceGAN, AniFaceGAN) and models trained on entirely disparate datasets (Morphable Diffusion on FaceScape [ 74 ] , Gaussian Head Avatar on NeRSemble [ 75 ] ), forcing a specific target identity would require error-prone inversion or cross-domain adaptation. To avoid disadvantaging these baselines with reconstruction artifacts, we display high-quality native identities sampled directly from their respective distributions. Fig. 5 illustrates a zoomed comparison of the red-box region.

Fig. 5: In contrast to Next3D and Diffusion-rig, our approach excels in capturing fine-grained expression facial details, such as the curvature of the mouth corners during smiling and the positioning of the eyebrows during frowning, bringing generated facial features closer to the reference image and target emotion text descriptions.

Method	AU Acc $\uparrow$							CLIP Score $\uparrow$
Method	Smile	Angry	Sad	CE	RB	FB	Avg	Smile	Angry	Sad	CE	RB	FB	Avg
DiscoFaceGAN [ 44 ]	0.87	0.48	0.33	N/A	0.35	0.23	0.45	25.10	23.89	23.02	23.82	24.04	23.53	23.90
AniFaceGAN [ 23 ]	0.57	0.43	0.38	N/A	0.60	0.46	0.49	24.01	25.94	24.64	23.98	24.50	25.06	24.69
Morphable Diffusion [ 27 ]	0.46	0.39	0.52	N/A	0.48	0.48	0.49	24.28	25.11	24.29	25.22	24.75	25.26	24.82
Gaussian Head Avatar [ 28 ]	0.34	0.44	0.54	N/A	0.50	0.52	0.47	23.83	26.26	25.28	24.63	24.26	24.84	24.85
Next3D [ 24 ]	0.65	0.41	0.42	N/A	0.49	0.38	0.47	24.56	24.61	23.35	25.93	24.63	24.61	24.62

TABLE I: Quantitative results on the expression editing results using two metrics: Action Units Accuracy (AU Acc) and CLIP Score. The used expressions are Smile, Angry, Sad, Close Eyes (CE), Raise Brow (RB), and Frown Brow (FB). Since OpenFace [ 81 , 82 ] cannot detect the AU corresponding to “Close Eyes”, we do not provide the AU accuracy for this expression.

Fig. 6: Comparison with the recent 2D text-guided face editing methods. We perform three expression edits on the same individual. The final three columns demonstrate view-consistent editing capabilities from our method, a feature lacking in both 2D methods, DeltaEdit and StyleCLIP, which struggle to control views.

	StyleCLIP [ 15 ]	DeltaEdit [ 87 ]	DiscoFaceGAN [ 44 ]	AniFaceGAN [ 23 ]	Next3D [ 24 ]	Diffusion-rig [ 26 ]	Ours
FID $\downarrow$	35.53	45.78	58.95	51.00	32.95	31.26	30.71
KID $\downarrow$	0.021	0.024	0.040	0.035	0.019	0.024	0.018

TABLE II: Quantitative results of image quality using FID and KID.

Method	CLIP Score $\uparrow$
Method	Smile	Angry	Sad	CE	RB	FB	Avg
DeltaEdit [ 87 ]	25.81	23.69	23.26	23.41	23.51	22.68	23.73
StyleCLIP [ 15 ]	25.15	26.48	25.69	23.59	23.67	23.27	24.64
Ours	24.70	26.27	23.96	26.16	24.66	25.06	25.14

TABLE III: Quantitative results on the fine-grained expression editing results using CLIP Score. The used expressions are Smile, Angry, Sad, Close Eyes (CE), Raise Brow (RB), and Frown Brow (FB).

Method	AU Acc $\uparrow$							CLIP Score $\uparrow$							FID&KID $\downarrow$
Method	Smile	Angry	Sad	CE	RB	FB	Avg	Smile	Angry	Sad	CE	RB	FB	Avg	FID	KID
w/o $\mathcal{L}_{\mathrm{ID}}$	0.99	0.48	0.81	N/A	0.92	0.21	0.68	27.30	28.80	26.07	27.94	26.96	29.29	27.73	43.42	0.027
w/o Projection	0.75	0.46	0.49	N/A	0.68	0.67	0.61	24.89	25.77	23.98	26.18	24.85	25.62	25.22	30.38	0.018
w/o Cross-Attention	0.83	0.56	0.48	N/A	0.68	0.62	0.63	25.34	26.43	23.94	26.28	24.76	25.56	25.38	30.93	0.018
w/o EMOCA	0.70	0.48	0.52	N/A	0.64	0.47	0.56	24.79	26.44	24.44	26.49	24.80	25.52	25.41	33.47	0.019
Full Model	0.90	0.49	0.56	N/A	0.73	0.56	0.65	25.12	26.38	24.67	26.10	24.72	24.96	25.33	30.13	0.017

TABLE IV: Ablation study on fine-grained expression editing quality and image quality.

Fig. 7: The qualitative ablation experiments examine the main components of our method, including the removal of identity loss ( w/o $\mathcal{L}_{\mathrm{ID}}$ ), Subspace Projection ( w/o Projection ), Cross-Attention ( w/o Cross-Attention ) and EMOCA ( w/o EMOCA ). We present the results of expression editing on the “Close Eyes” and “Angry” expressions, for each ablation.

Metric	DiscoFaceGAN [ 44 ]	AniFaceGAN [ 23 ]	Morphable Diffusion [ 27 ]	Gaussian Head Avatar [ 28 ]	Next3D [ 24 ]	Diffusion-rig [ 26 ]	Ours
SSIM $\uparrow$	0.8457	0.9996	0.9911	0.9998	0.9998	0.9998	0.9996

TABLE V: Quantitative results for background preservation.

Fig. 8: Illustration of new expression editing results using the proposed method, including “Mouth Stretch”, “Mouth Right” “Dimpler”, “Lip Funneler”, “Mouth Left” and “Grin”. The edited images demonstrate the model’s ability to generate diverse fine-grained expressions while maintaining high visual quality.

Fig. 9: Expression editing results for the same individual. Each row corresponds to a different subject, and each column represents a target expression, including “Smile”, “Angry”, “Sad”, “Raise Brow”, “Close Eyes”, and “Frown Brow”. The results demonstrate the ability of the proposed method to modify expressions while preserving the identity and overall image quality.

Fig. 10: Comparison results of fine-grained expression editing on real images: real images, inverted images, Next3D outputs, and results from the proposed method across six expressions (“Smile”, “Angry”, “Sad”, “Raise Brow”, “Close Eyes”, and “Frown Brow”).

Fig. 11: Comparison results under different training setups: training without target expression data versus training initialized with target expression codes, evaluated across six expressions (“Smile”, “Angry”, “Sad”, “Raise Brow”, “Close Eyes”, and “Frown Brow”).

Fig. 12: The view consistency across different viewpoints is evaluated by calculating

\mathcal{L}_{\mathrm{ID}}

between images generated by rotating the head model by 1-30 degrees from the frontal view. Our method demonstrates a smoother trend compared to Diffusion-rig and Morphable Diffusion (evaluated via uniform sampling from

-30^{\circ}

30^{\circ}

due to its 16-image generation limit).

Fig. 13: The images generated from different viewpoints demonstrate that our method maintains superior view consistency, particularly in the generation of hair, compared to Diffusion-rig.

Methods	Inference Time (s)
DiscoFaceGAN [ 44 ]	0.62
AnifaceGAN [ 23 ]	0.95
Next3D [ 24 ]	0.52
Gaussian Head Avatar [ 28 ]	0.42
Diffusion-rig [ 26 ]	1.58
Morphable Diffusion [ 27 ]	1.26

TABLE VI: Comparison of inference time (in seconds) per run across different methods. Our method achieves competitive efficiency compared to GAN-based methods while significantly outperforming diffusion-based baselines.

Fig. 14: Sensitivity Analysis of the Identity Regularization Weight (

\lambda_{\mathrm{ID}}

Fig. 15: Qualitative results of compositional expression editing demonstrate our model’s ability to seamlessly combine multiple distinct facial actions, such as “Close Eyes” alongside “Frown Brow” or “Raise Brow”, applied to various input images.

3D Avatar Generation

3D avatar generation creates realistic, controllable digital humans from images, video, text, or other inputs.

See all 90 papers

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions
Jul 2, 2026Haoran Wang, Mohit Mendiratta, Christian Theobalt +1Text-to-3D Generation Facial Expression Synthesis
Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security
RAGMesh with FaME-G2E: Long-Form Text-Driven 3D Face Generation and Editing
Aug 10, 2026Hao Li, Ju Dai, Feng Zhou +63D Mesh Generation 3D Shape Editing
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Pengcheng Laboratory, Shenzhen, China · North China University of Technology, Beijing, China +3
PixelSmile: Toward Fine-Grained Facial Expression Editing
Mar 26, 2026Jiabin Hua, Hengyuan Xu, Aojie Li +4Image Editing Diffusion-Based Image Editing
Fudan University · StepFun

Jul 2, 2026J/K move · Enter open · S save

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

Haoran Wang, Mohit Mendiratta, Christian Theobalt, Adam Kortylewski

Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security

Text-to-3D Generation Facial Expression Synthesis

Open paper AskPDF arXiv

Precise control of 3D facial expressions from text is crucial for virtual avatars, animation, and human-computer interaction, yet existing text-to-3D methods jointly generate identity, expression, and texture, making fine-grained expression control difficult. We instead formulate text-driven expression synthesis as a regression problem in the disentangled parameter space of a 3D Morphable Model (3DMM). This setting, however, requires paired data linking detailed language to precise expression parameters, which are missing from existing resources. To fill this gap, we introduce Txt2Emote, a benchmark of diverse 3D facial expressions with fine-grained textual annotations obtained from GPT-4o and a high-fidelity face tracker, providing both explicit descriptions detailing facial features and implicit descriptions referencing the situational context behind the expression. Leveraging this dataset, we present EmoteGPT, a text-to-3D expression framework based on a Multimodal Large Language Model (MLLM) with a dedicated <Expr> token to semantically ground expression representations, which are then decoded into 3DMM parameters. We further improve EmoteGPT by augmenting training with large-scale image-to-3DMM data, enabling it to surpass state-of-the-art text-to-3D face synthesis methods on emotion recognition metrics and in perceived expressiveness. Integrated into avatar pipelines, our method enables photorealistic and stylized 3D avatars, as well as expressive 3D-consistent 2D face synthesis from textual input.

Abstract

3D Avatar Generation

Explore similar work

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

RAGMesh with FaME-G2E: Long-Form Text-Driven 3D Face Generation and Editing

PixelSmile: Toward Fine-Grained Facial Expression Editing