To date, all natural scene skeleton detection follows the paradigm of taking RGB images as the sole input; despite notable progress, methods under this paradigm suffer significant performance degradation on complex-content images. We observe that depth images are inherently insensitive to color and texture, and can provide clear regional contours and inter-region spatial relationships, which naturally alleviates the difficulty of skeleton detection in complex scenarios. Motivated by this observation, this paper proposes for the first time a novel skeleton detection paradigm where depth images serve as the dominant modality and RGB images act as the auxiliary, and accordingly presents a model DDSkel (short for Depth-Dominant Skeleton Detection) under this paradigm. DDSkel employs an asymmetric encoder design to fuse RGB information into depth features, with the RGB modality branch having only 12% the parameters of the depth modality branch. DDSkel has a simple structure without intricate designs. Nevertheless, with only 36% of the trainable parameters of the current best method, DDSkel outperforms all state-of-the-art approaches on SymPASCAL, the most challenging dataset with a large volume of complex images.
Figures & tables
Figure 1: Examples of the comparison of detection results between DDSkel and 3 state-of-the-art skeleton detection methods, BlumNet, AdaLSN, and DeepFlux.
Figure 2: The overall architecture of DDSkel, which consists of an encoder and a decoder. The encoder comprises two backbones, namely ConvNeXtV2-Atto and DINOV3-Tiny, which process RGB images and depth images respectively. RGB features are fused with depth features via the CFM (Cross-stage Fusion Module), DFM (Dual-modality Fusion Module), and DAFM (Dual-modality Attention-based Fusion Module) to enhance depth features, which are then fed into the decoder to produce the final skeleton detection results.
Method
SK506
SK-LARGE
WH-SYMMAX
SymPASCAL
HED
0.542
0.497
0.732
0.369
SRN
0.632
0.658
0.780
0.443
Hi-Fi
0.681
0.724
0.805
0.454
DeepFlux
0.695
0.732
0.840
0.502
AdaLSN
0.740
0.786
0.851
0.497
FMRN
0.751
0.789
0.865
0.508
Table 1: Quantitative comparison on 4 datasets with 9 methods using F-measure scores. Values highlighted in bold in table correspond to the highest-ranked F-measure scores.
Method
Params (M)
FLOPs (G)
Time (ms)
HED
14.72
103.75
5.18
SRN
14.72
103.75
5.23
DeepFlux
18.32
125.93
6.39
Ada-LSN
32.38
217.07
15.05
BlumNet
99.32
121.0
22.8
BlumNet+
100.80
–
–
Table 2: Computational-efficiency comparison with existing skeleton-detection methods. The symbol ’–’ means no results provided by authors.
Figure 3: The qualitative comparison between DDSkel and three state-of-the-art methods, BlumNet, AdaLSN, and DeepFlux.
Configuration
SymPASCAL
SK-LARGE
RGB-Only
0.6197
0.8008
Depth-Only
0.6602
0.8051
RGB–RGB
0.6345
0.8046
Depth–Depth
0.6633
0.8102
RGB–Depth
0.6622
0.8138
Depth–RGB(DDSkel)
0.6740
0.8250
Table 3: Comparison of different modality assignments under a fixed network architecture and training protocol. For dual-branch settings, the first modality is assigned to the primary backbone and the second modality to the auxiliary branch.
RGB Auxiliary Branch
SymPASCAL
SK-LARGE
ConvNeXtV2-Tiny (60.85M)
0.6802
0.8292
ConvNeXtV2-Pico (41.15M)
0.6753
0.8264
ConvNeXtV2-Femto (37.23M)
0.6759
0.8247
ConvNeXtV2-Atto (35.66M, Ours)
0.6740
0.8250
Table 4: Ablation of the RGB auxiliary-branch capacity with the depth primary branch and fusion configuration fixed. The values in parentheses denote the total parameter count of the complete dual-stream model. Values highlighted in bold in table correspond to the highest-ranked F-measure scores.
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China. · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore, Singapore. · School of Geodesy and Geomatics, Wuhan University, Wuhan, China. +2