High-resolution facial datasets with consistent UV-mapped geometry, albedo, and physically based displacement are scarce, yet such assets are essential for photorealistic digital-human rendering. Existing 3D face generation methods are often slow, single-modality, or require large specialized datasets. Meanwhile, large image diffusion models encode strong facial priors but do not natively produce lighting-independent textures, pore-scale displacement, or UV-aligned geometry. We present FaceRiff, a unified, data- and parameter-efficient framework for generating complete 3D face assets (mesh, albedo, and displacement) directly from text in a shared UV space. FaceRiff adapts a pretrained SDXL backbone with task-specific LoRA adapters and uses UV normal maps as geometric conditioning: per-instance normals guide albedo and displacement synthesis, while a dataset-mean normal provides a canonical spatial anchor for mesh generation. Using only 2,049 scans from approximately 350 identities and training on a single GPU, FaceRiff updates less than 3% of the backbone parameters per task and produces render-ready 3D face assets without per-instance optimization. Compared with FaceG2E and DreamFace, FaceRiff produces greater per-prompt diversity while preserving prompt alignment and fixed-topology UV consistency, and additionally generates lighting-independent albedo suitable for PBR relighting and pore-scale displacement absent from prior text-driven baselines. A blind study with 45 industry professionals shows that FaceRiff achieves substantially higher perceived realism than DreamFace, approaching that of real captures. Overall, FaceRiff demonstrates that pretrained diffusion priors can be adapted to unified 3D face asset synthesis under limited data.
Figures & tables
Figure 1. FaceRiff: text-driven 3D face asset generation. From a text prompt, FaceRiff synthesizes mesh, lighting-independent albedo, and pore-scale displacement in a shared UV space in a feed-forward manner, combining them into the final rendered full head. Teaser figure showing a text prompt alongside the generated 3D face assets—mesh, albedo, displacement—and the final rendered head, produced feed-forward by FaceRiff.
Figure 2. Guide texture used for validating mesh alignment.
Figure 3. FaceRiff pipeline overview. (a–c) Task-specific LoRA adapters synthesize displacement, albedo, and mesh UV maps from a VLM caption and normal conditioning. Displacement and albedo use per-subject normals N , while mesh generation uses the dataset-mean normal Nˉ as a canonical scaffold. (d) The ground-truth UV map is encoded as z0 , noised to zt , and denoised by SDXL to predict ϵθ(zt,t,ct,cf) . A frozen Normal ControlNet provides normal conditioning cf , while task-specific LoRA weights adapt SDXL for each modality.
Figure 4. FaceRiff text-to-mesh prompt control. Four seeds per prompt. Rows (a,b) differ by a single-word gender edit; rows (c,d) use demographic prompts. Identity varies across seeds yet samples stay prompt-faithful; expression is prompted jointly with identity, not edited independently. FaceRiff mesh generation results for four prompts. Each row shows multiple generated mesh samples from the same text prompt, demonstrating prompt control, gender editing, and seed-level diversity.
Figure 5. FaceG2E text-to-mesh results. Using the first two prompts from Fig. 4 , we show three random seeds per prompt. FaceG2E produces near-identical meshes across seeds, indicating limited seed-level identity diversity under the same prompt. FaceG2E mesh generation results for the first two prompts from Fig. 4. Each row shows three random seeds, illustrating limited variation across seeds.
Scale
Mean ↓
p95 ↓
DistToMean ↑
Pixel_Std ↑
0.0
12.72
26.57
7.59
0.068
0.5
11.63
24.00
7.58
0.090
1.0
11.47
24.93
8.10
0.103
2.0
12.19
24.64
9.81
0.123
Table 1. ControlNet scale sweep ( 100 prompts, mm). Mean error is lowest at scale 1.0; diversity rises with conditioning strength. Default in bold .
Figure 6. Text-to-albedo with GT normal conditioning. Each row: ground truth and three FaceRiff samples from the same prompt on the same GT mesh. Attributes stay consistent while pore-level detail varies across samples. Text-to-albedo generation showing two rows of results. Each row contains a ground-truth facial texture and three FaceRiff-generated samples from one prompt, with the corresponding prompts listed underneath.
Config
FID ↓
LPIPS ↓
CLIP ↑
DINO pair↓
DINO FD↓
w/o normal
85.01
0.338
0.290
0.252
621.79
w/ normal
80.71
0.316
0.295
0.236
613.64
Table 2. Normal conditioning ablation for text-to-albedo on UV albedo maps ( 100 validation prompts, valid face region). Default in bold .
Figure 7. Luminance distribution comparison. DreamFace (b) shows an extended high-luminance tail from baked-in lighting; FaceRiff (c) more closely matches the GT distribution (a). See Fig. 10 for full qualitative comparisons.
Figure 8. Text-to-displacement with normal conditioning. Top: GT displacement, base mesh without displacement, and FaceRiff-generated displacement. Bottom: beard-region close-up showing that FaceRiff synthesizes underlying skin detail rather than displacing facial hair. Two-row displacement result. Top row: ground-truth displacement, mesh without displacement, and FaceRiff-generated displacement. Bottom row: beard-region close-up showing FaceRiff synthesizes skin detail beneath facial hair rather than displacing the hair.
Figure 9. End-to-end 3D face generation. Four full-head renders combining FaceRiff-generated mesh, albedo, and displacement, each produced from text alone. Mesh is synthesized first, and its UV normal map conditions albedo and displacement, preserving cross-modality correspondence without post-hoc registration. Prompt summaries are shown below; full prompts are provided in the supplemental material. Results span diverse demographics and fine-grained attributes (ethnicity, age, skin tone, and skin detail).
Figure 10. Fair comparison with DreamFace (no-normal variant). For each prompt: GT (left), FaceRiff (middle), DreamFace (right). Across seeds, FaceRiff produces lighting-independent albedo with sharper pore detail and greater inter-sample diversity; DreamFace shows baked-in shading, smoother texture, and mode collapse across seeds. Comparison of FaceRiff and DreamFace results for text-to-albedo generation, showing differences in lighting independence, pore-level detail, and diversity across four prompts.
Figure 11. Additional text-to-displacement results. Full-face comparisons (GT displacement, base mesh, FaceRiff) across diverse subjects, ages, and expressions. All results use normal conditioning with GT UV normals. Multiple rows of text-to-displacement results showing ground-truth displacement, base mesh, and FaceRiff-generated displacement across diverse subjects with varying ages and expressions.
Diffusion models have achieved strong results in high-fidelity image synthesis, but their iterative sampling process makes large-scale generation computationally expensive. This limitation is especially relevant when generating synthetic face datasets for face recognition, where a large number of subjects with many samples in different poses, expressions, ages, etc., are required. In this work, we show that identity-conditioned face synthesis can be performed at a substantially lower computational cost by a latent Consistency Model with few iterations, without compromising image quality. For training, we distill knowledge from the foundation Diffusion Model Arc2Face (teacher) by adapting its original text-to-image pipeline to an embedding-to-face setting, replacing textual prompts with ArcFace identity embeddings. Our distilled model (student) generates identity-conditioned face images with an average inference time of 0.4819 seconds per image, compared with 2.102 seconds for Arc2Face, resulting in a 4.36× speed-up. Quantitative results, based on FID scores, show that the distilled model remains competitive with Arc2Face across all evaluation protocols. On 100k generated images, it achieves near-parity on CelebA (13.921 vs. 12.928) and outperforms the teacher on WebFace42M (9.317 vs. 9.802). Further evaluations on Synth-500 and AgeDB show a moderate performance gap for the former but comparable results for the latter. These results indicate that Arc2Face can be accelerated through task-specific latent consistency distillation while preserving high image quality for large-scale synthetic face generation. Our proposal is publicly available at https://github.com/UFPR-IPASP-PR/FaceRec-IdentityConsistency.
Tiago Kienen Chaves, Bernardo Biesseck, David Menotti
Department of Informatics, Federal University of Paraná, Curitiba, Brazil · Federal Institute of Mato Grosso (IFMT), Pontes e Lacerda, MT, Brazil
Digitizing humans and synthesizing photorealistic avatars with explicit 3D pose and camera controls are central to VR, telepresence, and entertainment. Existing skinning-based workflows require laborious manual rigging or template-based fittings, while neural volumetric methods rely on canonical templates and re-optimization for each unseen pose. We present PoseCraft, a diffusion framework built around tokenized 3D interface: instead of relying only on rasterized geometry as 2D control images, we encode sparse 3D landmarks and camera extrinsics as discrete conditioning tokens and inject them into diffusion via cross-attention. Our approach preserves 3D semantics by avoiding 2D re-projection ambiguity under large pose and viewpoint changes, and produces photorealistic imagery that faithfully captures identity and appearance. To train and evaluate at scale, we also implement GenHumanRF, a data generation workflow that renders diverse supervision from volumetric reconstructions. Our experiments show that PoseCraft achieves significant perceptual quality improvement over diffusion-centric methods, and attains better or comparable metrics to latest volumetric rendering SOTA while better preserving fabric and hair details.
Acquiring labeled datasets for 3D human mesh estimation is challenging due to depth ambiguities and the inherent difficulty of annotating 3D geometry from monocular images. Existing datasets are either real, with manually annotated 3D geometry and limited scale, or synthetic, rendered from 3D engines that provide precise labels but suffer from limited photorealism, low diversity, and high production costs. In this work, we explore a third path: generated data. We introduce PoseDreamer, a novel pipeline that leverages diffusion models to generate large-scale synthetic datasets with 3D mesh annotations. Our approach combines controllable image generation with Direct Preference Optimization for control alignment, curriculum-based hard sample mining, and multi-stage quality filtering. Together, these components naturally maintain correspondence between 3D labels and generated images, while prioritizing challenging samples to maximize dataset utility. Using PoseDreamer, we generate more than 500,000 high-quality synthetic samples, achieving a 76% improvement in image-quality metrics compared to rendering-based datasets. Models trained on PoseDreamer achieve performance comparable to or superior to those trained on real-world and traditional synthetic datasets. In addition, combining PoseDreamer with synthetic datasets results in better performance than combining real-world and synthetic datasets, demonstrating the complementary nature of our dataset. We will release the full dataset and generation code.
The Podium Institute for Sports Medicine and Technology University of Oxford Oxford, UK · Visual Geometry Group University of Oxford Oxford, UK · Ukrainian Catholic University Lviv, Ukraine