Single-view 3D human reconstruction has achieved remarkable progress through the adoption of multi-view diffusion models, yet the recovered 3D humans often exhibit unnatural poses. This phenomenon becomes pronounced when reconstructing 3D humans with dynamic or challenging poses, which we attribute to the limited scale of available 3D human datasets with diverse poses. To address this limitation, we introduce DrPose, Direct Reward fine-tuning algorithm on Poses, which enables post-training of a multi-view diffusion model on diverse poses without requiring expensive 3D human assets. DrPose trains a model using only human poses paired with single-view images, employing a direct reward fine-tuning to maximize PoseScore, which is our proposed differentiable reward that quantifies consistency between a generated multi-view latent image and a ground-truth human pose. This optimization is conducted on DrPose15K, a novel dataset that was constructed from an existing human motion dataset and a pose-conditioned video generative model. Constructed from abundant human pose sequence data, DrPose15K exhibits a broader pose distribution compared to existing 3D human datasets. We validate our approach through evaluation on conventional benchmark datasets, in-the-wild images, and a newly constructed benchmark, with a particular focus on assessing performance on challenging human poses. Our results demonstrate consistent qualitative and quantitative improvements across all benchmarks. Project page: https://seunguk-do.github.io/drpose.
Figures & tables
Figure 1: We propose DrPose ( D irect R eward Fine-tuning on Pose s), a method to post-train a multi-view diffusion model for enhanced posture of reconstructed 3D humans in dynamic and acrobatic scenarios.
Figure 2: Illustration of DrPose presented in Algo 1 . A Denoising multi-view U-Net ϵω is trained to minimize Ltotal=Lreward+wKL⋅LKL . Multi-view latent images x0 are generated from xT∼N(0,I) , and Lreward is computed from x0 and the ground-truth 3D human pose θ . For efficiency, only a subset of denoising steps is sampled for training. Concurrently, LKL is computed as the KL divergence between ϵω and the frozen reference U-Net ϵw0 at intermediate denoising steps. For clarity, only 3 of the 6 multi-view images are shown.
Figure 3
Figure 5: Overview of our 3D human reconstruction pipeline. In this pipeline, the multi-view normal and RGB images are generated from the input image using an image-to-multi-view (I2MV) diffusion model. These images are then converted into a 3D representation using explicit human carving ( Li et al., 2024b ) . In this work, we propose post-training the I2MV diffusion model to achieve better alignment with accurate poses in dynamic and acrobatic scenarios. For clarity, only 3 of the 6 multi-view images are displayed for normal maps and RGB images.
Figure 6: Representative visualizations for the MixamoRP benchmark. The 18 meshes shown were randomly sampled from the complete dataset containing 60 meshes.
THuman2.1-test
CustomHumans-test
MixamoRP
Method
CD ↓
NC ↑
f-Score ↑
CD ↓
NC ↑
f-Score ↑
CD ↓
NC ↑
f-Score ↑
ECON
101.6465
0.6311
8.5244
126.1430
0.6205
6.4700
166.5384
0.5705
5.2173
SiTH
63.3041
0.6790
14.9221
71.9378
0.6713
12.7957
158.2729
0.5685
6.5176
H3D
75.8328
0.5959
12.2189
94.0864
0.5872
10.5563
149.2832
0.5400
7.2219
Era3D*
55.4071
0.6976
16.2928
63.1260
0.6914
14.1348
150.0118
0.5916
7.3487
PSHuman
52.9643
0.7194
18.6308
52.2187
0.7272
18.5624
137.2814
0.5876
8.2065
Table 1: Quantitative comparisons of geometry quality on single-view human reconstruction benchmarks. Our proposed benchmark MixamoRP is described in Section 4.2.2 . Era3D* represents the original Era3D model fine-tuned on CustomHumans and THuman2.1 training splits using conventional DDPM loss. Ours (Era3D) denotes the Era3D model post-trained with our proposed DrPose on DrPose15K.
THuman2.1-test
CustomHumans-test
MixamoRP
Method
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
SiTH
17.7473
0.8829
0.1371
18.7462
0.8599
0.1533
15.6096
0.8118
0.2182
H3D
17.3290
0.8656
0.1561
17.6244
0.8328
0.1882
15.2037
0.7915
0.2440
Era3D*
17.7662
0.8826
0.1410
18.6322
0.8581
0.1584
17.5337
0.8623
0.1519
PSHuman
18.3880
0.8922
0.1286
18.9082
0.8612
0.1538
17.5931
0.8638
0.1499
Ours (Era3D)
20.3563
0.9050
0.1134
18.9286
0.8644
0.1508
17.5064
0.8662
0.1474
Table 2: Quantitative evaluation of 3D human reconstruction quality. Six RGB views evenly distributed in azimuth are rendered to compute appearance metrics. Our proposed benchmark MixamoRP is described in Section 4.2.2 . Era3D* represents the original Era3D model fine-tuned on CustomHumans and THuman2.1 training splits using conventional DDPM loss. Ours (Era3D) denotes the Era3D model post-trained with our proposed DrPose on DrPose15K.
Figure 7: Qualitative evaluation on the CustomHumans dataset. Era3D* denotes Era3D fine-tuned on CustomHumans and THuman2.1 datasets.
Figure 8: Qualitative evaluation on the internet-source images. Era3D* denotes Era3D fine-tuned on CustomHumans and THuman2.1 datasets.
Figure 9: Qualitative results of DrPose on in-the-wild internet images.
Benchmark
PSNR
SSIM
LPIPS
THuman2.1-test
22.4807
0.9337
0.0580
CustomHumans-test
24.4081
0.9536
0.0430
Table 3: Quantitative evaluation of gskel in PoseScore
Figure 10: Visualization of gskel in PoseScore. gskel converts the multi-view latent images encoded from the normal and RGB images using the base model’s VAE.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Our pipeline is sensitive to the quality of the segmented masks, producing artifacts.
ECON
SiTH
Era3D
PSHuman
Ours (Era3D)
Ours
183.27 sec.
117.35 sec.
15.24 sec.
42.70 sec.
15.24 sec.
42.70 sec.
Appendix
Table 4: Reconstruction latency per sample for each model.
Figure 12: Architecture of the Denoising U-Net for multi-view diffusion in our pipeline inspired by Li et al. (2024b) (illustrated in Fig 5 ). The denoising U-Net follows the architecture of PSHuman ( Li et al., 2024b ) . The input image is conditioned into the denoising process through two parallel pathways: (1) A VAE encoder E encodes the input image, which is then concatenated with the latent image xt . (2) A CLIP image encoder τ encodes the input image, and the generated tokens are fed into the cross-attention layers of the denoising U-Net.
Figure 15: Additional qualitative evaluation on the internet-source images. Era3D* denotes Era3D fine-tuned on CustomHumans and THuman2.1 datasets.
Figure 16: Additional qualitative evaluation on the internet-source images. Era3D* denotes Era3D fine-tuned on CustomHumans and THuman2.1 datasets.
Single-image 3D human reconstruction often suffers from over-smoothed textures and geometric inconsistencies. While diffusion models improve generative quality, their reliance on multi-view synthesis prior to 3D reconstruction is computationally expensive and prone to view inconsistency. We propose DiGS-Avatar, which reformulates this task as an efficient, diffusion-based UV-latent completion task, ensuring 3D consistency by design. To capture accurate spatial structure, we introduce a teacher-student framework where a multi-view teacher provides geometrically aligned pseudo-ground-truth latents to supervise a single-view diffusion student. Treating this inferred latent as a robust structural skeleton, our method injects high-level semantic features to accurately recover fine textural details without disrupting spatial integrity. The refined representation is then decoded into 3D Gaussian primitives. Extensive experiments demonstrate that DiGS-Avatar achieves state-of-the-art or highly competitive visual fidelity and zero-shot generalization, while reconstructing a fully animatable 3D avatar in just 0.71 seconds. Code is available at https://github.com/KLMAV-CUC/DiGS-Avatar.
Jiakun Li, Li Fang, Hao Zhu +4
Key Laboratory of Media Audio and Video (Communication University of China), Ministry of Education, Beijing 100024, China · School of Intelligence Science and Technology, Nanjing University, Nanjing 210023, China
Recovering 3D human poses for multiple individuals from different camera views is a fundamental bottleneck for analyzing interacting behaviors. Existing self-supervised approaches leverage synthetic catalogues of 3D poses; however, this leads to poor generalization in real-world scenarios due to distribution shifts. We therefore introduce DisPOSE, a self-supervised framework that approximates the inherently discrete multi-view person-assignment problem as a generative diffusion process over the space of polystochastic tensors. By employing differentiable Sinkhorn projections during denoising, our model learns to guide solutions toward valid and feasible assignments based on 2D image priors. The complete 3D skeletons of localized individuals are then regressed using a Hypergraph-Convolutional Decoder that explicitly models relational structures and articulated joints across multiple views. The proposed approach outperforms current state-of-the-art self-supervised methods on standard datasets and demonstrates strong performance on a newly proposed benchmark featuring highly occluded scenes from surgical operating rooms. Our diffusion-based localization demonstrates high label efficiency, retaining 99% of its performance with only 10% of the pseudo-labels. Notably, disentangling the assignment and root regression components while maintaining differentiability makes DisPOSE nearly agnostic to different camera arrangements.
Tony Danjun Wang, Tolga Birdal, Nassir Navab +1
School of Computation, Information, and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning, Germany · Department of Computing, Imperial College London, United Kingdom
Detailed and photorealistic 3D human modeling is essential for various applications and has seen tremendous progress. However, full-body reconstruction from a monocular RGB image remains challenging due to the ill-posed nature of the problem and sophisticated clothing topology with self-occlusions. In this paper, we propose PSHuman, a novel framework that explicitly reconstructs human meshes utilizing priors from the multiview diffusion model. It is found that directly applying multiview diffusion on single-view human images leads to severe geometric distortions, especially on generated faces. To address it, we propose a cross-scale diffusion that models the joint probability distribution of global full-body shape and local facial characteristics, enabling detailed and identity-preserved novel-view generation without any geometric distortion. Moreover, to enhance cross-view body shape consistency of varied human poses, we condition the generative model on parametric models like SMPL-X, which provide body priors and prevent unnatural views inconsistent with human anatomy. Leveraging the generated multi-view normal and color images, we present SMPLX-initialized explicit human carving to recover realistic textured human meshes efficiently. Extensive experimental results and quantitative evaluations on CAPE and THuman2.1 datasets demonstrate PSHumans superiority in geometry details, texture fidelity, and generalization capability.
Peng Li, Wangguandong Zheng, Yuan Liu +9
HKUST · Southeast University · Tsinghua University +1