GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos
Authors: Yiqian Wu, Rawal Khirodkar, Egor Zakharov, Timur Bagautdinov, Lei Xiao, Zhaoen Su, Shunsuke Saito, Xiaogang Jin, +1 more
Organizations: State Key Laboratory of CAD&CG, Zhejiang University, China · Codec Avatars Lab, Meta, USA · State Key Lab of CAD&CG, Zhejiang University, China
We present GenLCA, a diffusion-based generative model for generating and editing photorealistic full-body avatars from text and image inputs. The generated avatars are faithful to the inputs, while supporting high-fidelity facial and full-body animations. The core idea is a novel paradigm that enables training a full-body 3D diffusion model from partially observable 2D data, allowing the training dataset to scale to millions of real-world videos. This scalability contributes to the superior photorealism and generalizability of GenLCA. Specifically, we scale up the dataset by repurposing a pretrained feed-forward avatar reconstruction model as an animatable 3D tokenizer, which encodes unstructured video frames into structured 3D tokens. However, most real-world videos only provide partial observations of body parts, resulting in excessive blurring or transparency artifacts in the 3D tokens. To address this, we propose a novel visibility-aware diffusion training strategy that replaces invalid regions with learnable tokens and computes losses only over valid regions. We then train a flow-based diffusion model on the token dataset, inherently maintaining the photorealism and animatability provided by the pretrained avatar reconstruction model. Our approach effectively enables the use of large-scale real-world video data to train a diffusion model natively in 3D. We demonstrate the efficacy of our method through diverse and high-fidelity generation and editing results, outperforming existing solutions by a large margin. The project page is available at https://onethousandwu.com/GenLCA-Page.
Figures & tables
Figure 1. GenLCA is a diffusion-based generative model for generating and editing full-body 3D Gaussian avatars from text and image inputs. (A) Generation. GenLCA generates avatars that are visually realistic and consistent with both the identity in the input face image and the semantic descriptions in the input text, while supporting high-fidelity facial and full-body animations. We present zoomed-in and animated face results. (B) Editing. By leveraging text, RGB images, or scribbles as control signals, GenLCA enables seamless multi-modal editing of the generated avatars. (C) Diverse 3D avatars generated by GenLCA from text inputs.
Dataset
# total IDs
# synthetic data
# captured data
# in-the-wild data
StructLDM ( Hu et al., 2024 )
1.8K
0.8K ∗
0.5K ∗
0.5K ∗
HumanLiff ( Hu et al., 2025 )
1.1K
1K ∗
0.1K ∗
0
Rodin ( Wang et al., 2023 )
100K
100K
0
0
RodinHD ( Zhang et al., 2024a )
46K
46K
0
0
SimAvatar ( Li et al., 2025b )
20K
20K
0
0
TeRA ( Wang et al., 2025b )
70K
70K
0
0
Table 1. Comparison of training datasets for 3D human diffusion models. Our dataset contains the largest number of identities and the most realistic data among existing methods. “ ∗ ” indicates that the corresponding model is trained on each sub-dataset individually, rather than on a mixture of them.
Figure 2. The architecture of the reconstruction model. The transformer takes image tokens and query point embeddings as inputs, and outputs GS tokens. The GS tokens are decoded to get dynamic GS attributes. The resulting Gaussian splats are rendered using LBS to obtain the final renderings.
Figure 3. (A) Training pipeline of GenLCA. During training, the high dimensional GS tokens are first encoded into compact GS latent by the compressor encoder. For conditional inputs, we use CLIP ( Radford et al., 2021 ) to extract text embeddings and DINOv2 ( Oquab et al., 2024 ) to extract scribble and body part embeddings. To prevent the training process from being affected by corrupted information, we replace invalid regions (as indicated by the visibility mask) with learnable placeholder features and employ a masked loss. (B) Detailed architecture of the compressor. The compressor’s encoder and decoder consist of MLPs for downsampling or upsampling, combined with self-attention blocks for feature fusion. Positional encoding is applied within each block. (C) Detailed architecture of the GenLCA block. We adapt the MMDiT block as the basic block of GenLCA. Each GenLCA block takes the query points, time step, latent features, and conditional features as inputs. Separate branches are used to process latent and conditional features, and positional encoding is added to the latent features.
Figure 4. (A) The tokens accurately reconstruct the visible regions of the images. After filtering out all invalid tokens and retaining only the valid ones, the rendered results still achieve high-quality reconstruction. (B) We present the rendered tokens in canonical space (1st row), where blurry regions are highlighted with blue boxes and transparent regions with yellow boxes. The visibility mask (2nd row) separates the valid and invalid regions.
Figure 5. 3D avatars generated by GenLCA from texts. All results are generated with CFG scale = 5.0, 50 sampling steps, and animated with random poses.
Figure 6. We compare our GenLCA with SOTA methods, including SDS based approaches: TADA ( Liao et al., 2024 ) , HumanGaussian ( Liu et al., 2024 ) , and DreamWaltz-G ( Huang et al., 2025b ) , and text conditioned 3D human diffusion models, TeRA ( Wang et al., 2025b ) and SIGMAN ( Yang et al., 2025 ) . We use the same text prompt as input. In addition to full-body renderings, we also provide zoomed-in views for comparison of facial regions.
Method
Semantic Align
Quality
FID ↓
Inference Time ↓
BLIP-VQA ↑
Text CLIP Score ↑
CLIB-FIQA ↑
HyperIQA ↑
2D Diffusion
THuman 2.0
HuGe100K
TADA ( Liao et al., 2024 )
0.50
0.71
0.48
55.02
188.19
N/A
N/A
2.5h
HumanGaussian ( Liu et al., 2024 )
0.62
0.73
0.39
33.61
239.33
N/A
N/A
1.2h
DreamWaltz-G ( Huang et al., 2025b )
0.58
0.75
0.50
59.33
175.23
N/A
N/A
3.0h
TeRA ( Wang et al., 2025b )
0.42
0.67
0.44
44.01
151.80
N/A
N/A
12s
SIGMAN ( Yang et al., 2025 )
0.29
0.58
0.42
56.11
280.06
121.40
160.48
3s
Table 2. Quantitative comparison results with SOTA methods. ■ and ■ denote the 1st and 2nd places.
Method
User study ↑
Semantic align.
Consistency
Visual quality
Geometric quality
TADA ( Liao et al., 2024 )
2.89
3.18
2.29
2.15
HumanGaussian ( Liu et al., 2024 )
3.59
3.37
2.79
2.77
DreamWaltz-G ( Huang et al., 2025b )
3.68
3.93
3.41
3.37
TeRA ( Wang et al., 2025b )
2.63
3.74
3.30
3.28
SIGMAN ( Yang et al., 2025 )
1.65
1.86
1.40
1.52
Table 3. User studies. ■ and ■ denote the 1st and 2nd places.
Figure 7. We conduct ablation studies by individually removing the visibility-aware training, learnable placeholder, and in-the-wild training data components to demonstrate the effectiveness of each.
We present Generative Relightable Avatars (GRA), a person-specific method for photorealistic free-view rendering and environment-map relighting of full-body humans. We postulate that modeling fine-grained appearance details is inherently a one-to-many problem that can benefit from a generative formulation. In contrast to fully regressive relightable avatar methods, GRA follows a hybrid approach that combines controllable, physics-grounded relighting with probabilistic refinement. Starting from a tracked animated mesh, we optimize material parameters in UV-space and render a coarse relit appearance under a target HDR environment map. Next, we refine the textures with a feed-forward model to capture pose-dependent texture dynamics and illumination effects beyond simplified reflectance assumptions. Finally, a fine-tuned video-to-video diffusion model transforms the physically grounded renderings into temporally coherent, high-detail videos while preserving 3D control, with an error-recycling strategy for generating long videos. Experimental evaluations demonstrate our method's improved perceptual quality over prior relightable avatar baselines. Project Page: https://vcai.mpi-inf.mpg.de/projects/GRA/
Kunwar Maheep Singh, Christian Theobalt, Rishabh Dabral
Max Planck Institute for Informatics, Saarland Informatics Campus
We introduce FiCA, a Feed-forward, instant Gaussian Codec Avatar generation pipeline that creates lifelike avatars from a single portrait image. Generating a photorealistic and drivable avatar from just a single image is significantly challenging due to the limited visual information available to accurately infer the 3D appearance and geometry of human heads. To address this, we develop a novel system that combines human-centric vision foundation models with a diffusion model. This system is designed to fully exploit partial visual observations to generate lifelike human avatars. Our proposed diffusion model learns a generative mapping from these partial observations to complete and authentic 3D mesh reconstruction. Additionally, we introduce a feed-forward mesh refinement network that enhances the fidelity and identity preservation of the generated avatars, eliminating the need for person-specific test-time optimization. By leveraging a universal prior model that decodes a generated mesh into a set of 3D Gaussians, we generate a photorealistic 3D Gaussian avatar, capable of being driven with novel expressions in real-time. Our experiments demonstrate that the avatars generated by our feed-forward approach faithfully represent diverse identities and surpass the visual quality of avatars produced by recent competing methods.
Kim Youwang, Zhengyu Yang, Liuhao Ge +8
Codec Avatars Lab, Meta · Dept. of Electrical Engineering, POSTECH
Controllable 3D human avatars have found widespread applications in 3D games, the metaverse, and AR/VR scenarios. The conventional approach to creating such a 3D avatar requires a lengthy, intricate pipeline encompassing appearance modeling, motion planning, rigging, and physical simulation. In this paper, we introduce UNICA (UNIfied neural Controllable Avatar), a skeleton-free generative model that unifies all avatar control components into a single neural framework. Given keyboard inputs akin to video game controls, UNICA generates the next frame of a 3D avatar's geometry through an action-conditioned diffusion model operating on 2D position maps. A point transformer then maps the resulting geometry to 3D Gaussian Splatting for high-fidelity free-view rendering. Our approach naturally captures hair and loose clothing dynamics without manually designed physical simulation, and supports extra-long autoregressive generation. To the best of our knowledge, UNICA is the first model to unify the workflow of "motion planning, rigging, physical simulation, and rendering". Code is released at https://github.com/zjh21/UNICA.
Jiahe Zhu, Xinyao Wang, Yiyu Zhuang +4
Nanjing University, Nanjing, China State Key Laboratory of Novel Software Technology, China