cs.CVMar 4, 2026
SaveDual Diffusion Models for Multi-modal Guided 3D Avatar Generation
Organizations: Beihang University · KAUST
Abstract
Generating high-fidelity 3D avatars from text or image prompts is highly sought after in virtual reality and human-computer interaction. However, existing text-driven methods often rely on iterative Score Distillation Sampling (SDS) or CLIP optimization, which struggle with fine-grained semantic control and suffer from excessively slow inference. Meanwhile, image-driven approaches are severely bottlenecked by the scarcity and high acquisition cost of high-quality 3D facial scans, limiting model generalization. To address these challenges, we first construct a novel, large-scale dataset comprising over 100,000 pairs across four modalities: fine-grained textual descriptions, in-the-wild face images, high-quality light-normalized texture UV maps, and 3D geometric shapes. Leveraging this comprehensive dataset, we propose PromptAvatar, a framework featuring dual diffusion models. Specifically, it integrates a Texture Diffusion Model (TDM) that supports flexible multi-condition guidance from text and/or image prompts, alongside a Geometry Diffusion Model (GDM) guided by text prompts. By learning the direct mapping from multi-modal prompts to 3D representations, PromptAvatar eliminates the need for time-consuming iterative optimization, successfully generating high-fidelity, shading-free 3D avatars in under 10 seconds. Extensive quantitative and qualitative experiments demonstrate that our method significantly outperforms existing state-of-the-art approaches in generation quality, fine-grained detail alignment, and computational efficiency.
Figures & tables
Figure 1: PromptAvatar generates realistic and animatable 3D avatars from a single text prompt, image, or both, compatible with 3D rendering engines like Blender. The top-left corner uses a text prompt to create an accurate texture UV-map and mesh. The bottom-left corner combines text with an FLUX-dev-1.0-generated [ 1 ] image to guide high-quality texture UV-map creation. When an image prompt is used, facial geometry is extracted via a pre-trained 3D face reconstruction network [ 2 , 3 ] . On the right, image prompts enable detailed texture effects like crow’s feet and beards.
Figure 2: Our dataset creation pipeline consists of three main modules: De-lighting and Re-lighting face image generation, Incomplete UV-map correction and completion, and Identity coefficients estimation and facial attribute description.
Figure 3: Architecture of PromptAvatar. The framework comprises a Texture Diffusion Model (TDM), which targets high-quality normalized texture generation, and a Geometry Diffusion Model (GDM) for geometric identity coefficients. Both models are guided by multi-modal prompts embedded via CLIP. For image prompts, incomplete textures are encoded into the latent space to provide localized guidance.
| Instruction |
|---|
| Provide a detailed description of this portrait’s appearance features within 200 tokens, ignoring hair and expression. Please answer in the following format, for example: |
| {{ : A middle-aged male about 50s, Asian yellow human}, |
| { : Yellow skin with highland red undertones, large pores, thick and long eyebrows, chin and cheeks covered with beard, deep wrinkles around the forehead and eyes, wrinkles between the eyes, freckles distributed on the cheeks, slight eye bags, red lips, deep red nose, dark circles, deep gray beard, black eyebrows, melanin deposits}, |
| { : rectangular face, overall slender, forehead slightly broad and protruding, willow leaf eyebrows, large almond eyes, relatively narrow eye spacing, high nasal bridge, rounded nose tip, slightly high cheekbones, cherry lips, round but slightly upturned narrow chin, overall carrying a hint of a calm demeanor.}} |
| Note that this is very important to my work. Your description should be as detailed and fair as possible. |
Table 1: The instruction template used for Qwen2.5-VL-32B-Instruct to generate structured facial attribute descriptions.
Figure 4: The network architecture of GDM.
| Similarity | left-front | left-right | front-right |
|---|---|---|---|
| Ours | 0.754 | 0.801 | 0.774 |
| FFHQ-UV | 0.634 | 0.544 | 0.632 |
Table 2: Quantitative evaluation of Identity Similarity on multi-view images.
| Dataset | Ours | FFHQ-UV |
|---|---|---|
| BS Error | 5.098 | 7.094 |
Table 3: Quantitative comparison of Brightness Symmetry (BS) Error. Lower values indicate better light-normalization.
Figure 5: Visual comparison with DreamFace and Describe3D. Our results demonstrate superior alignment with fine-grained text prompts (highlighted in red). For instance, in the first row, observe the distribution of facial hair, eyebrow shape, and chin, and in the second row, note the facial shape, eye bags, and other features.
| Metric | DreamFusion | Describe3D | DreamFace | Ours |
|---|---|---|---|---|
| CLIP score | 20.16 | 19.81 | 20.56 | 21.14 |
| Infer time(s) | 2400 | 15 | 300 | 10 |
Table 4: Quantitative comparison with other text-to-avatar methods.
Figure 6: Demonstration of fine-grained facial attribute editing via text prompts. Starting from an initial generated UV-map (left), our Texture Diffusion Model (TDM) accurately manipulates specific semantic features—such as gender, facial hair, age, and ethnicity—by simply modifying the corresponding keywords in the prompts (highlighted in red). The results show that TDM achieves precise localized editing while preserving the unedited features.
Figure 7: Visual comparison of the image-to-avatar methods. Each group consists of the original image, the re-lit PromptAvatar, and FFHQ-UV. It can be observed that our method is able to preserve more facial details and skin tone of the original image. Please zoom in for better visibility of the details.
| Similarity | FFHQ-UV | FlameTex | Ours | w/emb | w/both |
| FFHQ | 0.354 | 0.1589 | 0.370 | 0.284 | 0.338 |
| CMask-HQ | 0.324 | 0.114 | 0.334 | 0.242 | 0.304 |
| Infer times(s) | 80 | 72 | 6 | / | / |
Table 5: Quantitative experiments of Image-to-Avatar and Ablation studies. CMask-HQ is CelebAMask-HQ.
Figure 8: Ablation study on VAE fine-tuning. Left: Original in-the-wild image. Middle: Texture UV-map generated using the frozen pre-trained SD-2.1 VAE, which results in overly smoothed high-frequency details. Right: Texture generated using our fine-tuned VAE. The fine-tuned VAE successfully adapts to the UV-map data distribution, effectively recovering realistic facial features such as crow’s feet and wrinkles (highlighted in the red boxes).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Part I: Coarse-grained Prompts |
|---|
| 55-year-old Caucasian male, rosy cheeks, noticeably thin face with slightly sunken cheeks. |
| African child, round face, small eyes, thick lips, short curly hair. |
| Elderly Caucasian male, elegant appearance, big eyes, long pointed nose with visible wrinkles, wide lips. |
| 30-year-old Middle Eastern male, prominent nose, thick eyebrows, medium beard, calm and regal facial expression. |
| 15-year-old female, cute appearance, round face, bright eyes, straight black hair. |
| 25-year-old female, captivating smile, elegant facial features, wavy blonde hair. |
Table 6: Extended list of text prompts (Part I). These coarse-grained descriptions focus on general demographic attributes and basic facial features, explicitly omitting non-facial body descriptions.
| Part II: Fine-grained Prompts |
|---|
| A Caucasian male in his 30s with medium-tone skin and neutral undertones has some visible pores, very light stubble on the chin and upper lip, a few faint smile lines around the mouth, minimal creases under the eyes, and slightly noticeable frown lines on the forehead. |
| A mixed man in his 40s with a personable appearance has fair skin and rosy cheeks. His complexion is clear with subtle lines under his eyes and faint laugh lines around his mouth. |
| A young adult Caucasian male in his 30s with a professional appearance has fair skin with slight pink undertones, small pores, and a clean-shaven look. |
| A 30-year-old light-skinned adult male with olive skin tone and faint pink undertones has small pores and slight nasolabial fold visibility. |
| A Caucasian male in his 30s has light skin with subtle pink undertones and visible pores. He has a small number of faint freckles on the cheeks and minimal signs of wrinkles, mainly in the nasolabial fold area. |
| A middle-aged Asian male in his 50s with a professional demeanor. His skin features relatively small pores, faint fine lines, and wrinkles mainly on the forehead and around the eyes, with some visible around the mouth. |
Table 7: Extended list of text prompts (Part II). These fine-grained descriptions incorporate diverse and challenging facial details such as pores, wrinkles, acne, and pigmentation.
Figure 9: Visual examples of the multi-modal facial dataset. Each row displays a unique identity from our collection. From left to right: multi-view source images, unwrapped high-resolution texture maps, facial renderings under novel lighting, and the corresponding 3D geometry.
Figure 10: More visual comparison of the image-to-avatar methods. The [red dashed boxes] indicate that our method exhibits better details than FFHQ-UV, while the [blue dashed boxes] indicate flaws in FFHQ-UV. The bottom-right corner of each image in the first column is the environment map used for rendering. The FlameTex and Deep3D results use textures directly estimated by each method for visualization.
More in this topic
3D Avatar Generation
3D avatar generation creates realistic, controllable digital humans from images, video, text, or other inputs.
See all 90 papers