A Self-Supervised Pose Estimation Approach for Construction Machines
Authors: Alaa Alshubbak, Daniel Görges
Organizations: Institute of Electromobility, University of Kaiserslautern-Landau, Kaiserslautern, Germany · German Jordanian University, Amman, Jordan.
Pose estimation is a computer vision task used to estimate a skeleton of dynamic systems to predict future movements. Most of the research in this direction is based on a supervised learning approach which requires a massive amount of labeled datasets. In this paper, a self-supervised three-stage model based on a contrastive learning approach is introduced for estimating a skeleton of dynamic construction machines; such as excavators without using any labeled images for the first stage. The whole model structure is divided into three stages: the pre-train stage using the SimCLR contrastive approach, and two fine-tuning stages for the transfer learning and the downstream task. The model can leverage the features and learn from a huge unlabeled dataset called ACID to two small datasets generated from NVIDIA Isaac and MATLAB Simscape simulators as well as transfer the knowledge to a smaller dataset with a ratio of 3.5% from the original ACID dataset. The results show that the proposed approach can improve the accuracy of pose estimation for heavy construction machines in real images by 11% and 13% in comparison to the normal self-supervised approach with two backbones ResNet-50 and HRNet-W32, respectively.
Table 1: Definition of machine keypoints in literature and this work.
Figure 1: General diagram of a self-supervised model trained over unlabeled data.
Figure 2: Diagram of the approach with three-stage self-supervised paradigm.
Figure 3: Image samples from ACID manually annotated for pose estimation.
Model
Datasets
Status
AP
AP 50
AP 75
AP M
AP L
AR
AR 50
AR 75
AR M
AR L
Epochs
ResNet-50
ACID p
Sup
0.4455
1.0
0.3307
-1.0
0.4455
0.4867
1.0
0.4667
-1.0
0.4867
120
ImageNet
Self-Sup
0.37962
0.80528
0.17327
-1.0
0.37962
0.42
0.86667
0.2
-1.0
0.42
590
HRNet-W32
ACID p
Sup
0.4228
1.0
0.3187
-1.0
0.4228
0.520
1.0
0.5333
-1.0
0.520
120
ImageNet
Self-Sup
0.3832
0.7958
0.2911
-1.0
0.3832
0.4200
0.8667
0.3333
-1.0
0.4200
450
Table 2: The first scenario: supervised learning versus normal self-supervised learning approach for learning features for a pose estimation downstream task. There is no intermediate stage in this approach.
Model
Data
Status
AP
AP 50
AP 75
AP M
AP L
AR
AR 50
AR 75
AR M
AR L
Epochs
ResNet-50
6 keys
Self-Sup
0.4913
0.9261
0.3964
-1.0
0.4913
0.5400
0.9333
0.5333
-1.0
0.5400
100
Nvidia
Self-Sup
0.405
0.861
0.352
-1.0
0.405
0.440
0.867
0.467
-1.0
0.440
560
Matlab
Self-Sup
0.4562
1.0
0.3030
-1.0
0.4562
0.5133
1.0
0.4000
-1.0
0.5133
440
HRNet-W32
6 keys
Self-Sup
0.5145
1.0
0.3634
-1.0
0.5145
0.5667
1.0
0.4667
-1.0
0.5667
330
Nvidia
Self-Sup
0.2858
0.61174
0.10396
-1.0
0.2858
0.32667
0.32667
0.13333
-1.0
0.32667
510
Matlab
Self-Sup
0.1477
0.4738
0.0901
-1.0
0.1477
0.26
0.6667
0.20
-1.0
0.260
480
Table 3: The second scenario : results of fine tuning ResNet-50 on small ACID dataset (3.5% of whole ACID dataset) from different transfer learning models used one of three main pose estimation datasets. The initial self-supervised feature extraction was trained over the whole ACID dataset.
Status
Model
AP
AR
Epochs
Model
AP
AR
Epochs
Dataset: NVIDIA simulated dataset
Supervised
ResNet-50
0.817
0.863
150
HRNet-w32
0.814
0.871
380
Self-sup
0.834
0.888
200
0.940
0.961
600
Dataset: MATLAB simulated dataset
Supervised
ResNet-50
0.688
0.696
120
HRNet-w32
0.690
0.696
90
Self-sup
0.696
0.699
130
0.684
0.7
550
Table 4: Results of pose estimation model training with both supervised and self-supervised approaches on the two simulated datasets with different backbones. Feature extraction in self-supervised stage was trained over whole ACID dataset.
Figure 4: Samples results of the self-supervised model originally pretrained with ImageNet, and the proposed approach. Red circle shows a new keypoint detection, while the blue box shows a correction in the detection.
Figure 5: Samples from supervised and self-supervised models trained on datasets from NVIDIA Isaac and Simscape Multibody using the ResNet-50 backbone.
Figure 6: Samples of automatic bounding box computation in green labeling using the minimum bounding box of the six keypoints of excavator pose.
This paper revisits camera pose estimation through the lens of self-supervised pretraining, focusing on inverse-dynamics pretraining as a scalable alternative to the current trend of fully supervised training with 3D annotations. Concretely, we employ inverse- and forward-dynamics models to learn latent action representations, similar to Genie from large-scale driving videos. Our idea is simple yet effective. Existing methods use latent actions in their original capacity, that is, as action conditioning of world-models or as proxies of robot action parameters in policy networks. Our method, dubbed LA-Pose, repurposes the latent action features as inputs to a camera pose estimator, finetuned on a limited set of high-quality 3D annotations. This formulation enables accurate and generalizable pose prediction while maintaining feed-forward efficiency. Extensive experiments on driving benchmarks show that LA-Pose achieves competitive and even superior performance to state-of-the-art methods while using orders of magnitude less labeled data. Concretely, on the Waymo and PandaSet benchmarks, LA-Pose achieves over 10% higher pose accuracy than recent feed-forward methods. To our knowledge, this work is the first to demonstrate the power of inverse-dynamics self-supervised learning for pose estimation.
We present an improvement on previous spacecraft pose estimation architectures that results in the lowest published mean rotation errors we know of on the SPEED+ lightbox and sunlamp test sets for a known, non-cooperative spacecraft. By using a previously established heatmap-based pose estimation architecture and adapting a large self-supervised ViT foundation model (DINOv3) in place of the smaller convolutional and ViT encoders of previous work, we show that pose estimation accuracy improves from 300M to 840M parameters with no saturation yet observed. We also evaluate our 840M model on a Jetson Orin NX 16GB, measuring single-pass network inference at 133.8 ms per crop with a board draw of 32.0 W. These measurements demonstrate embedded inference feasibility on a processor family with orbital flight heritage. Our resulting model outperforms previous models across lightbox and sunlamp domains while training only on synthetic data. Our best model, using DINOv3 840M adapted with LoRA as the encoder (rank 64, three-seed ensemble with four-rotation test-time augmentation), results in 1.56∘ mean rotation error on sunlamp and 1.17∘ on lightbox, compared to the previous best mean rotation errors we know of on these test sets, 2.66∘ and 1.75∘ by EagerNet.
John Church, Vazghen Nikolian
Both authors contributed equally to this work; author order is alphabetical.
Unsupervised domain adaptation (UDA) enables robust transfer of knowledge from simulated to real environments while exploiting a subset of unlabeled target data to improve real-world performance. Existing UDA methods for Object pose estimation often rely on global feature matching, multi-stage larger frameworks, or image translation pipelines, which tend to overlook the pose-specific information embedded in feature representations. To bridge this limitation, we introduce CAPLR that targets the adaptation of pose-sensitive features in localized regions, ensuring that domain alignment preserves the geometric cues essential for accurate pose estimation. CAPLR achieves UDA with three key components: (1) Efficient Cross-Domain Pairing strategy leveraging intermediate features to identify pose similar image pairs across domains without supervision; (2) Contrastive Alignment to perform feature alignment at localised regions in both intermediate and task-specific representations; and (3) Consistency-Based Pseudo-Label Refinement to improve reliability by encouraging stable target predictions. Extensive experiments demonstrate that CAPLR achieves state-of-the-art performance across multiple well-known object pose estimation benchmarks featuring diverse and challenging scenarios.