SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation

Shuang Liang; Jing He; Lejun Liao; Chuanmeizhi Wang; Guo Zhang; Yingcong Chen; Yuan Yuan

cs.CVSep 29, 2025

SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation

Authors: Shuang Liang, Jing He, Lejun Liao, Chuanmeizhi Wang, Guo Zhang, Yingcong Chen, Yuan Yuan

Organizations: Rama Alpaca Technology · The University of Hong Kong · HKUST (GZ) · Boston College ·

Abstract

Pre-trained diffusion models provide rich latent features across U-Net levels and are emerging as powerful vision backbones. While prior works such as Marigold and Lotus repurpose diffusion priors for dense geometric perception tasks such as depth and surface normal estimation, their potential for cross-domain human pose estimation remains largely unexplored. Through a systematic analysis of latent features from different upsampling levels of the Stable Diffusion U-Net, we identify the levels that deliver the strongest robustness and cross-domain generalization for pose estimation. Building on these findings, we propose SDPose, which (i) extracts U-Net features from the selected upsampling blocks, (ii) fuses them with a lightweight feature aggregation module to form a robust representation, and (iii) jointly optimizes keypoint heatmap supervision with an auxiliary latent reconstruction loss to regularize training and preserve the pre-trained generative prior. To evaluate cross-domain generalization and robustness, we construct COCO-OOD, a COCO-based benchmark with four subsets: three style-transferred splits to assess domain shift and one corruption split (noise, weather, digital artifacts, and blur) to test robustness. With data-efficient fine-tuning, SDPose achieves performance comparable to Sapiens on COCO, and establishes the best zero-shot performance on cross-domain datasets.

Figures & tables

Figure 1: t-SNE visualization of features from Sapiens ViT, Stable Diffusion U-Net blocks, Sapiens Pose, and SDPose across four visual domains. Each point corresponds to an image sample; colors represent person instances and marker shapes denote artistic styles.

Figure 2: Training pipeline of SDPose. The input image $x$ is encoded by a pre-trained SD-VAE into $Z_{\mathrm{RGB}}$ . A shared latent diffusion U-Net $f_{\theta}$ is conditioned by a task/class embedding $C$ (text prompt disabled). With $C_{\mathrm{RGB}}$ , the model reconstructs $Z_{\mathrm{RGB}}$ via an $\ell_{2}$ latent reconstruction loss. With $C_{\mathrm{Pose}}$ , we extract selected multi-level U-Net features, perform feature aggregation, and predict keypoint heatmaps supervised by a heatmap MSE loss.

Figure 3: Qualitative results on real-world photographs. The yellow boxes highlight regions where baselines fail to predict accurate poses.

Figure 4: Illustration of the COCO-OOD dataset.

Figure 5: Visualization of pose-guided image generation results. The lower images illustrate results from the baseline, which combines a pre-trained ControlNet with the DWPose estimator. In comparison, the upper images show results obtained using our SDPose as the pose annotator. Yellow boxes highlight baseline failures. Prompts, random seeds, and other settings are kept identical for fairness.

Figure 6: Qualitative comparison for pose-controlled video generation in the wild. The first row shows the source image and frames from the driving video. The second row shows output video frames generated from the pose sequence estimated by the baseline model DWPose, while the third row shows the results guided by our SDPose. Red boxes highlight failures in the generated video, and yellow boxes highlight errors in pose estimation.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visual illustration of the style-transferred subsets in the COCO-OOD benchmark. We create an OOD split of the COCO validation set by stylizing natural images into three distinct artistic domains: Monet (via StyTR2), Ukiyo-e (via CycleGAN), and Color-sketch (via Nano Banana), enabling comprehensive evaluation under diverse style shifts.

Figure 8: Hierarchical features are fused through a CSP PAFPN block. Deep features are upsampled via PixelShuffle to align with shallow features for final heatmap prediction.


Model	Input Size	AP	AR
SimpleBaseline [ 45 ]	256 $\times$ 192	73.5	79.0
HRNet [ 42 ]	384 $\times$ 288	76.3	81.2
HRFormer [ 53 ]	256 $\times$ 192	77.2	82.0
ViTPose-S [ 48 ]	256 $\times$ 192	73.8	79.2
ViTPose-B [ 48 ]	256 $\times$ 192	75.8	81.1

Appendix

Table 3: Quantitative comparison on the COCO-2017 validation set. All results are self-reported by the corresponding works.

Table 5: Quantitative comparison on the COCO-WholeBody validation set.

Table 6: Quantitative Comparison on the COCO-OOD-Monet Wholebody validation set.

Table 7: Quantitative Comparison on the COCO-OOD-Ukiyoe Wholebody validation set.

Table 8: Quantitative Comparison on the COCO-OOD-Corruption Wholebody validation set.

Figure 9: Comparison on Stylized Paintings: Sapiens WholeBody vs. SDPose WholeBody. All erroneous predictions are highlighted with yellow boxes. SDPose yields fewer false positives and notably better facial keypoint localization.

Figure 10: Failure Cases of SDPose. SDPose may produce inaccurate keypoints on anthropomorphic or non-human characters, especially around ears, facial landmarks, and body parts with substantially different proportions from humans.

Split	$n$	OKS	$P_{o}$	$\kappa$
COCO-Monet	50	0.8975	0.960	0.733
COCO-Ukiyo-e	50	0.8899	0.967	0.749
All	100	0.8937	0.964	0.741

Appendix

Table 12: Human baseline on COCO-OOD, averaged over all 17 keypoints and over three independent annotators.

P_{o}

and

\kappa

are the raw agreement and Fleiss’

\kappa

of the binary placed / not placed decision across the three annotators.

Human Pose Estimation

Human pose estimation recovers people’s body keypoints, 3D skeletons, or meshes from images, video, and other sensor data.

See all 79 papers

Unsupervised Domain Adaptation for Sim-to-Real Object Pose Estimation with Contrastive Alignment and Pseudo-Label Refinement
Jun 19, 2026Nidhal Eddine Chenni, Arunkumar Rathinam, Djamila AouadaSynthetic-to-Real Domain Adaptation Unsupervised Domain Adaptation
SnT, University of Luxembourg
UniPose9D: Universal Category-Agnostic Object Pose Estimation
Jul 10, 2026Yang You, Yi Du, Cole Harrison +1Object Pose Estimation
Stanford University · Amazon
Enhancing Domain Generalization in 3D Human Pose Estimation through Controllable Generative Augmentation
May 12, 2026Xinhao Hu, Yiyi Zhang, Liqing Zhang +1Synthetic Data Augmentation Domain Generalization
School of Computer Science, Shanghai Jiao Tong University

Jun 19, 2026J/K move · Enter open · S save

Unsupervised Domain Adaptation for Sim-to-Real Object Pose Estimation with Contrastive Alignment and Pseudo-Label Refinement

Nidhal Eddine Chenni, Arunkumar Rathinam, Djamila Aouada

SnT, University of Luxembourg

Synthetic-to-Real Domain Adaptation Unsupervised Domain Adaptation Object Pose Estimation

Open paper AskPDF arXiv

Unsupervised domain adaptation (UDA) enables robust transfer of knowledge from simulated to real environments while exploiting a subset of unlabeled target data to improve real-world performance. Existing UDA methods for Object pose estimation often rely on global feature matching, multi-stage larger frameworks, or image translation pipelines, which tend to overlook the pose-specific information embedded in feature representations. To bridge this limitation, we introduce CAPLR that targets the adaptation of pose-sensitive features in localized regions, ensuring that domain alignment preserves the geometric cues essential for accurate pose estimation. CAPLR achieves UDA with three key components: (1) Efficient Cross-Domain Pairing strategy leveraging intermediate features to identify pose similar image pairs across domains without supervision; (2) Contrastive Alignment to perform feature alignment at localised regions in both intermediate and task-specific representations; and (3) Consistency-Based Pseudo-Label Refinement to improve reliability by encouraging stable target predictions. Extensive experiments demonstrate that CAPLR achieves state-of-the-art performance across multiple well-known object pose estimation benchmarks featuring diverse and challenging scenarios.

Abstract

Human Pose Estimation

Explore similar work

Unsupervised Domain Adaptation for Sim-to-Real Object Pose Estimation with Contrastive Alignment and Pseudo-Label Refinement

UniPose9D: Universal Category-Agnostic Object Pose Estimation

Enhancing Domain Generalization in 3D Human Pose Estimation through Controllable Generative Augmentation