Organizations: Rama Alpaca Technology · The University of Hong Kong · HKUST (GZ) · Boston College · Research Institute of Tsinghua University in Shenzhen · HKUST
Pre-trained diffusion models provide rich latent features across U-Net levels and are emerging as powerful vision backbones. While prior works such as Marigold and Lotus repurpose diffusion priors for dense geometric perception tasks such as depth and surface normal estimation, their potential for cross-domain human pose estimation remains largely unexplored. Through a systematic analysis of latent features from different upsampling levels of the Stable Diffusion U-Net, we identify the levels that deliver the strongest robustness and cross-domain generalization for pose estimation. Building on these findings, we propose SDPose, which (i) extracts U-Net features from the selected upsampling blocks, (ii) fuses them with a lightweight feature aggregation module to form a robust representation, and (iii) jointly optimizes keypoint heatmap supervision with an auxiliary latent reconstruction loss to regularize training and preserve the pre-trained generative prior. To evaluate cross-domain generalization and robustness, we construct COCO-OOD, a COCO-based benchmark with four subsets: three style-transferred splits to assess domain shift and one corruption split (noise, weather, digital artifacts, and blur) to test robustness. With data-efficient fine-tuning, SDPose achieves performance comparable to Sapiens on COCO, and establishes the best zero-shot performance on cross-domain datasets.
Figures & tables
Figure 1: t-SNE visualization of features from Sapiens ViT, Stable Diffusion U-Net blocks, Sapiens Pose, and SDPose across four visual domains. Each point corresponds to an image sample; colors represent person instances and marker shapes denote artistic styles.
Figure 2: Training pipeline of SDPose. The input image x is encoded by a pre-trained SD-VAE into ZRGB . A shared latent diffusion U-Net fθ is conditioned by a task/class embedding C (text prompt disabled). With CRGB , the model reconstructs ZRGB via an ℓ2 latent reconstruction loss. With CPose , we extract selected multi-level U-Net features, perform feature aggregation, and predict keypoint heatmaps supervised by a heatmap MSE loss.
Figure 3: Qualitative results on real-world photographs. The yellow boxes highlight regions where baselines fail to predict accurate poses.
Figure 4: Illustration of the COCO-OOD dataset.
Figure 5: Visualization of pose-guided image generation results. The lower images illustrate results from the baseline, which combines a pre-trained ControlNet with the DWPose estimator. In comparison, the upper images show results obtained using our SDPose as the pose annotator. Yellow boxes highlight baseline failures. Prompts, random seeds, and other settings are kept identical for fairness.
Figure 6: Qualitative comparison for pose-controlled video generation in the wild. The first row shows the source image and frames from the driving video. The second row shows output video frames generated from the pose sequence estimated by the baseline model DWPose, while the third row shows the results guided by our SDPose. Red boxes highlight failures in the generated video, and yellow boxes highlight errors in pose estimation.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Visual illustration of the style-transferred subsets in the COCO-OOD benchmark. We create an OOD split of the COCO validation set by stylizing natural images into three distinct artistic domains: Monet (via StyTR2), Ukiyo-e (via CycleGAN), and Color-sketch (via Nano Banana), enabling comprehensive evaluation under diverse style shifts.
Figure 8: Hierarchical features are fused through a CSP PAFPN block. Deep features are upsampled via PixelShuffle to align with shallow features for final heatmap prediction.
Model
Input Size
AP
AR
SimpleBaseline [ 45 ]
256 × 192
73.5
79.0
HRNet [ 42 ]
384 × 288
76.3
81.2
HRFormer [ 53 ]
256 × 192
77.2
82.0
ViTPose-S [ 48 ]
256 × 192
73.8
79.2
ViTPose-B [ 48 ]
256 × 192
75.8
81.1
Appendix
Table 3: Quantitative comparison on the COCO-2017 validation set. All results are self-reported by the corresponding works.
Table 5: Quantitative comparison on the COCO-WholeBody validation set.
Table 6: Quantitative Comparison on the COCO-OOD-Monet Wholebody validation set.
Table 7: Quantitative Comparison on the COCO-OOD-Ukiyoe Wholebody validation set.
Table 8: Quantitative Comparison on the COCO-OOD-Corruption Wholebody validation set.
Figure 9: Comparison on Stylized Paintings: Sapiens WholeBody vs. SDPose WholeBody. All erroneous predictions are highlighted with yellow boxes. SDPose yields fewer false positives and notably better facial keypoint localization.
Figure 10: Failure Cases of SDPose. SDPose may produce inaccurate keypoints on anthropomorphic or non-human characters, especially around ears, facial landmarks, and body parts with substantially different proportions from humans.
Split
n
OKS
Po
κ
COCO-Monet
50
0.8975
0.960
0.733
COCO-Ukiyo-e
50
0.8899
0.967
0.749
All
100
0.8937
0.964
0.741
Appendix
Table 12: Human baseline on COCO-OOD, averaged over all 17 keypoints and over three independent annotators. Po and κ are the raw agreement and Fleiss’ κ of the binary placed / not placed decision across the three annotators.
Unsupervised domain adaptation (UDA) enables robust transfer of knowledge from simulated to real environments while exploiting a subset of unlabeled target data to improve real-world performance. Existing UDA methods for Object pose estimation often rely on global feature matching, multi-stage larger frameworks, or image translation pipelines, which tend to overlook the pose-specific information embedded in feature representations. To bridge this limitation, we introduce CAPLR that targets the adaptation of pose-sensitive features in localized regions, ensuring that domain alignment preserves the geometric cues essential for accurate pose estimation. CAPLR achieves UDA with three key components: (1) Efficient Cross-Domain Pairing strategy leveraging intermediate features to identify pose similar image pairs across domains without supervision; (2) Contrastive Alignment to perform feature alignment at localised regions in both intermediate and task-specific representations; and (3) Consistency-Based Pseudo-Label Refinement to improve reliability by encouraging stable target predictions. Extensive experiments demonstrate that CAPLR achieves state-of-the-art performance across multiple well-known object pose estimation benchmarks featuring diverse and challenging scenarios.
Object pose estimation is a fundamental problem in 3D vision. Although recent state-of-the-art approaches achieve strong performance, generalization to novel categories and unseen scenes remains challenging. We propose UniPose9D, a unified model for category-agnostic 9D object pose estimation: given an instance mask/ROI and either an RGB-D observation or an RGB image with predicted depth, the model estimates rotation, translation, and metric size without category labels, CAD models, mean-shape priors, or reference views. Specifically, UniPose9D samples point pairs from the observed object geometry and uses DINOv2 and PointNet features to predict NOCS coordinates for each pair. To improve accuracy, we introduce a point-pair-based RANSAC N-hop Kabsch-Umeyama algorithm with an adaptive threshold. We further employ flow matching to address symmetric ambiguities and construct a large-scale training set by curating and aligning pose annotations from existing public datasets. Experiments across eight datasets show that a single unified model achieves competitive performance on standard benchmarks while generalizing to unseen objects, unseen categories, and in-the-wild scenarios. Our code and model are available at https://github.com/qq456cvb/UniPose9D.
Pedestrian motion, due to its causal nature, is strongly influenced by domain gaps arising from discrepancies between training and testing data distributions. Focusing on 3D human pose estimation, this work presents a controllable human pose generation framework that synthesizes diverse video data by systematically varying poses, backgrounds, and camera viewpoints. This generative augmentation enriches training datasets, enhances model generalization, and alleviates the limitations of existing methods in handling domain discrepancies. By leveraging both indoor/real-world and outdoor/virtual datasets, we perform cross-domain data fusion and controllable video generation to construct enriched training data, tailored to realistic deployment settings. Extensive experiments show that the augmented datasets significantly improve model performance on unseen scenarios and datasets, validating the effectiveness of the proposed approach.
Xinhao Hu, Yiyi Zhang, Liqing Zhang +1
School of Computer Science, Shanghai Jiao Tong University