A Self-Supervised Pose Estimation Approach for Construction Machines
Authors: Alaa Alshubbak, Daniel Görges
Organizations: Institute of Electromobility, University of Kaiserslautern-Landau, Kaiserslautern, Germany · German Jordanian University, Amman, Jordan.
Pose estimation is a computer vision task used to estimate a skeleton of dynamic systems to predict future movements. Most of the research in this direction is based on a supervised learning approach which requires a massive amount of labeled datasets. In this paper, a self-supervised three-stage model based on a contrastive learning approach is introduced for estimating a skeleton of dynamic construction machines; such as excavators without using any labeled images for the first stage. The whole model structure is divided into three stages: the pre-train stage using the SimCLR contrastive approach, and two fine-tuning stages for the transfer learning and the downstream task. The model can leverage the features and learn from a huge unlabeled dataset called ACID to two small datasets generated from NVIDIA Isaac and MATLAB Simscape simulators as well as transfer the knowledge to a smaller dataset with a ratio of 3.5% from the original ACID dataset. The results show that the proposed approach can improve the accuracy of pose estimation for heavy construction machines in real images by 11% and 13% in comparison to the normal self-supervised approach with two backbones ResNet-50 and HRNet-W32, respectively.
Table 1: Definition of machine keypoints in literature and this work.
Figure 1: General diagram of a self-supervised model trained over unlabeled data.
Figure 2: Diagram of the approach with three-stage self-supervised paradigm.
Figure 3: Image samples from ACID manually annotated for pose estimation.
Model
Datasets
Status
AP
AP 50
AP 75
AP M
AP L
AR
AR 50
AR 75
AR M
AR L
Epochs
ResNet-50
ACID p
Sup
0.4455
1.0
0.3307
-1.0
0.4455
0.4867
1.0
0.4667
-1.0
0.4867
120
ImageNet
Self-Sup
0.37962
0.80528
0.17327
-1.0
0.37962
0.42
0.86667
0.2
-1.0
0.42
590
HRNet-W32
ACID p
Sup
0.4228
1.0
0.3187
-1.0
0.4228
0.520
1.0
0.5333
-1.0
0.520
120
ImageNet
Self-Sup
0.3832
0.7958
0.2911
-1.0
0.3832
0.4200
0.8667
0.3333
-1.0
0.4200
450
Table 2: The first scenario: supervised learning versus normal self-supervised learning approach for learning features for a pose estimation downstream task. There is no intermediate stage in this approach.
Model
Data
Status
AP
AP 50
AP 75
AP M
AP L
AR
AR 50
AR 75
AR M
AR L
Epochs
ResNet-50
6 keys
Self-Sup
0.4913
0.9261
0.3964
-1.0
0.4913
0.5400
0.9333
0.5333
-1.0
0.5400
100
Nvidia
Self-Sup
0.405
0.861
0.352
-1.0
0.405
0.440
0.867
0.467
-1.0
0.440
560
Matlab
Self-Sup
0.4562
1.0
0.3030
-1.0
0.4562
0.5133
1.0
0.4000
-1.0
0.5133
440
HRNet-W32
6 keys
Self-Sup
0.5145
1.0
0.3634
-1.0
0.5145
0.5667
1.0
0.4667
-1.0
0.5667
330
Nvidia
Self-Sup
0.2858
0.61174
0.10396
-1.0
0.2858
0.32667
0.32667
0.13333
-1.0
0.32667
510
Matlab
Self-Sup
0.1477
0.4738
0.0901
-1.0
0.1477
0.26
0.6667
0.20
-1.0
0.260
480
Table 3: The second scenario : results of fine tuning ResNet-50 on small ACID dataset (3.5% of whole ACID dataset) from different transfer learning models used one of three main pose estimation datasets. The initial self-supervised feature extraction was trained over the whole ACID dataset.
Status
Model
AP
AR
Epochs
Model
AP
AR
Epochs
Dataset: NVIDIA simulated dataset
Supervised
ResNet-50
0.817
0.863
150
HRNet-w32
0.814
0.871
380
Self-sup
0.834
0.888
200
0.940
0.961
600
Dataset: MATLAB simulated dataset
Supervised
ResNet-50
0.688
0.696
120
HRNet-w32
0.690
0.696
90
Self-sup
0.696
0.699
130
0.684
0.7
550
Table 4: Results of pose estimation model training with both supervised and self-supervised approaches on the two simulated datasets with different backbones. Feature extraction in self-supervised stage was trained over whole ACID dataset.
Figure 4: Samples results of the self-supervised model originally pretrained with ImageNet, and the proposed approach. Red circle shows a new keypoint detection, while the blue box shows a correction in the detection.
Figure 5: Samples from supervised and self-supervised models trained on datasets from NVIDIA Isaac and Simscape Multibody using the ResNet-50 backbone.
Figure 6: Samples of automatic bounding box computation in green labeling using the minimum bounding box of the six keypoints of excavator pose.