Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance
Authors: Guodong Sun, Junjie Liu, Gaoyang Zhang, Bo Wu, Yang Zhang
Organizations: School of Mechanical Engineering, Hubei University of Technology, Wuhan, 430068, China · Hubei Key Laboratory of Modern Manufacturing Quality Engineering, Hubei University of Technology, Wuhan, 430068, China · Vehicle Measurement, Control and Safety Key Laboratory of Sichuan Province, Xihua University, Chengdu, 610039, China · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, 201210, China · National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210023, China
Scene understanding plays a critical role in enabling intelligence and autonomy in robotic systems. Traditional approaches often face challenges, including occlusions, ambiguous boundaries, and the inability to adapt attention based on task-specific requirements and sample variations. To address these limitations, this paper presents an efficient RGB-D scene understanding model that performs a range of tasks, including semantic segmentation, instance segmentation, orientation estimation, panoptic segmentation, and scene classification. The proposed model incorporates an enhanced fusion encoder, which effectively leverages redundant information from both RGB and depth inputs. For semantic segmentation, we introduce normalized focus channel layers and a context feature interaction layer, designed to mitigate issues such as shallow feature misguidance and insufficient local-global feature representation. The instance segmentation task benefits from a non-bottleneck 1D structure, which achieves superior contour representation with fewer parameters. Additionally, we propose a multi-task adaptive loss function that dynamically adjusts the learning strategy for different tasks based on scene variations. Extensive experiments on the NYUv2, SUN RGB-D, and Cityscapes datasets demonstrate that our approach outperforms existing methods in both segmentation accuracy and processing speed.
Figures & tables
Figure 1: Multi-task outputs of the network. Semantic segmentation provides foreground masks for instance segmentation, and pixel-level instances are generated by combining instance centers with instance offsets. Panoptic segmentation is achieved by integrating semantic and instance segmentation.
Figure 2: Multi-task scene understanding network structure. This network features an improved feature fusion encoder that handles redundant information from RGB and depth to enhance feature extraction. The semantic decoder includes NFCL and CFIL to enrich scene representations from various dimensions. The instance decoder employs the non-bottleneck 1D architecture to generate instance centers, instance offsets, and raw orientations. The integration of instance segmentation with semantic segmentation enables panoptic segmentation. Scene classification is performed by a task head with a fully connected layer. In addition, the multi-task adaptive loss function optimizes the training strategy based on data variations.
Figure 3: Fusion encoder. This encoder adopts merging layers for channel expansion and downsampling. Given the similarity between channel features, the fusion blocks extract redundant channel features to optimize computational efficiency.
Figure 4: Normalized focus channel layer. This layer learns the variance adjustment parameters via batch normalization. It then computes channel weights that quantify the importance of each channel. The input features are rearranged, weighted according to these channel weights, and subsequently processed through an activation function.
Figure 5: Context feature interaction layer. This layer addresses the limitation of the semantic decoder in integrating local and global information by employing a multi-scale pooling operation and efficient channel compression.
Figure 6: Semantic segmentation heat maps. Combining the feature guidance of CFIL and NFCL enables better handling of objects with similar colors and low-light scenes.
Figure 7: Non-bottleneck 1D module. This module decomposes 2D convolution into 1D convolutions combined with a non-linear activation function. This decomposition reduces model redundancy and enhances the non-linear decision-making capability.
Method
Instance
Panoptic
Scene
Semantic
PQ ↑
MAAE ↓
mIoU ↑
PQ ↑
RQ ↑
SQ ↑
MAAE ↓
bAcc ↑
mIoU ↑
Swin v2 [ 5 ]
58.49
21.09
50.51
43.08
52.22
81.34
20.46
77.11
49.76
ConvNeXt v2 [ 54 ]
41.04
31.24
27.20
18.21
24.11
72.20
29.29
61.79
27.69
MPViT [ 53 ]
57.77
21.18
47.09
39.43
47.78
81.13
19.82
75.16
47.44
MetaFormer [ 53 ]
53.31
23.69
44.75
34.01
42.96
77.86
22.10
74.51
43.27
Ours
58.59
18.67
47.37
40.35
49.18
81.03
17.88
74.67
46.83
Table 1: Encoder comparison results on the NYUv2 dataset. The symbols ↑ and ↓ indicate whether higher or lower values are better, respectively. The best metric is indicated in bold .
Figure 8: Feature visualization of backbones. We visualize the model inputs and the features extracted by Swin Transformer v2 and our method. Compared to the blurred features of Swin Transformer v2, our method more distinctly highlights objects in the scene, demonstrating better suitability for segmentation tasks.
Method
Instance
Panoptic
Scene
Semantic
PQ ↑
MAAE ↓
mIoU ↑
PQ ↑
RQ ↑
SQ ↑
MAAE ↓
bAcc ↑
mIoU ↑
SPPELAN [ 55 ]
58.79
19.18
47.55
41.09
49.60
81.66
18.18
77.39
46.51
ASPP [ 56 ]
58.75
18.14
48.40
41.81
50.51
81.61
17.83
76.73
47.98
SimSPPF [ 57 ]
58.34
19.07
47.62
40.56
49.12
81.37
18.03
74.70
46.40
RFB [ 58 ]
58.98
18.08
48.46
42.15
50.83
81.88
17.75
76.51
47.14
Ours
59.25
18.27
50.16
42.83
51.78
81.60
17.56
77.00
49.72
Table 2: Comparison of context modules. We compare the proposed context feature interaction layer with existing context modules. They are used in the multi-layer feature fusion stage of the semantic decoder.
Method
Instance
Panoptic
Scene
Semantic
PQ ↑
MAAE ↓
mIoU ↑
PQ ↑
RQ ↑
SQ ↑
MAAE ↓
bAcc ↑
mIoU ↑
P0
58.39
18.41
46.93
40.61
49.14
81.54
18.06
77.58
46.36
P1
59.37
19.34
48.39
41.81
51.02
80.78
18.58
76.23
47.72
P2
58.71
16.63
48.17
41.44
50.21
81.55
15.63
77.89
47.12
P3
59.49
17.90
48.99
42.40
51.24
81.56
17.25
75.08
49.25
P4
58.88
19.78
47.62
40.52
49.05
81.60
18.41
75.21
47.13
Table 3: Comparison of the context feature interaction layer positions. P0: CFIL not used. P1: encoder. P2: instance decoder. P3: semantic decoder and instance decoder. P4: encoder and semantic decoder. Ours: semantic decoder.
Method
Instance
Panoptic
Scene
Semantic
PQ ↑
MAAE ↓
mIoU ↑
PQ ↑
RQ ↑
SQ ↑
MAAE ↓
bAcc ↑
mIoU ↑
L0
59.15
19.03
49.37
42.00
50.61
81.97
17.74
76.48
48.93
L1
58.78
19.88
49.24
42.52
51.60
81.21
18.08
74.64
48.88
L2
59.37
18.42
50.01
42.74
51.71
81.5
17.78
76.42
49.53
L3
58.97
18.03
48.41
41.65
50.32
81.76
17.41
76.35
48.45
Ours
59.90
19.64
50.21
43.21
52.58
81.20
18.79
76.57
49.82
Table 4: Comparison of the normalized focus channel layers on the semantic decoder. L0: 1st, 2nd, 3rd, and 4th layers. L1: 2nd and 3rd layers. L2: 1st and 2nd layers. L3: 2nd, 3rd, and 4th layers. Ours: 1st, 2nd, and 3rd layers.
Method
Instance
Panoptic
Scene
Semantic
PQ ↑
MAAE ↓
mIoU ↑
PQ ↑
RQ ↑
SQ ↑
MAAE ↓
bAcc ↑
mIoU ↑
BasicBlock [ 59 ]
58.51
18.40
49.33
42.29
51.17
81.45
17.40
76.87
49.17
Bottleneck [ 59 ]
57.97
18.80
50.33
41.97
50.93
81.23
18.20
76.41
50.26
MobileBottleneck [ 60 ]
58.24
20.45
48.46
40.85
49.45
81.38
19.54
75.03
48.66
GhostBottleneck [ 61 ]
58.14
20.29
48.93
41.82
51.02
80.74
19.72
75.86
48.75
Ours
59.25
18.27
50.16
42.83
51.78
81.60
17.56
77.00
49.72
Table 5: Comparison of feature extraction modules. In the instance decoder layer, we compare the non-bottleneck 1D module with BasicBlock, Bottleneck, MobileBottleneck, and GhostBottleneck.
Figure 9: Comparison plot of multi-task adaptive loss. Figure a and Figure b represent the loss comparison and loss variance, respectively. The blue line represents the original fixed-weight loss, while the orange line represents the multi-task adaptive loss. To make the comparison clearer, we record the loss values and variances from epoch 100 onward. Our method makes the training process more convergent and stable.
Method
Instance
Panoptic
Scene
Semantic
PQ ↑
MAAE ↓
mIoU ↑
PQ ↑
RQ ↑
SQ ↑
MAAE ↓
bAcc ↑
mIoU ↑
Fixed
58.59
18.67
47.37
40.35
49.18
81.03
17.88
74.67
46.83
0.1
57.66
19.90
47.17
40.05
49.05
80.63
19.15
72.99
47.21
0.05
58.86
19.06
47.72
40.07
48.74
80.96
18.33
73.59
46.74
0.02
59.40
19.02
48.25
41.56
50.34
81.44
18.21
74.59
47.63
0.01
59.37
19.34
48.39
41.81
51.02
80.78
18.58
76.23
47.72
Table 6: Comparison of adjustment factors for multi-task adaptive loss. "Fixed" represents the original fixed weight. The numbers denote the adjustment factors, with our method using an adjustment factor of 0.01.
Fusion
Adaptive
CFIL
NFCL
Instance
Panoptic
Scene
Semantic
PQ ↑
MAAE ↓
mIoU ↑
PQ ↑
RQ ↑
SQ ↑
MAAE ↓
bAcc ↑
mIoU ↑
58.49
21.09
50.51
43.08
52.22
81.34
20.46
77.11
49.76
✓
58.59
18.67
47.37
40.35
49.18
81.03
17.88
74.67
46.83
✓
✓
59.37
19.34
48.39
41.81
51.02
80.78
18.58
76.23
47.72
✓
✓
✓
59.25
18.27
50.16
42.83
51.78
81.60
17.56
77.00
49.72
✓
✓
✓
✓
59.90
19.64
50.21
43.21
52.58
81.20
18.79
76.57
49.82
Table 7: Comparison results of the ablation study on the overall framework. Fusion: improved fusion encoder. Adaptive: multi-task adaptive loss. CFIL: context feature interaction layer. NFCL: normalized focus channel layer.
Figure 10: Visualization of the ablation experiment. We present the panoptic heatmaps for the model with the improvements: fusion encoder, multi-task adaptive loss, context feature interaction layer, and normalized focus channel layer. Benefiting from the adaptive learning strategy and cross-dimensional feature guidance, the model achieves more accurate recognition and contour representation.
Method
Params
FLOPs
FPS
VRAM
Semantic mIoU ↑
Instance PQ ↑
EMSAFormer [ 4 ]
72.08 M
50.66 G
16.32
3188 MiB
49.76
58.49
ConvNeXt v2 [ 54 ]
111.84 M
112.38 G
14.42
4592 MiB
27.69
41.04
MetaFormer [ 52 ]
76.14 M
72.06 G
12.01
3620 MiB
43.27
53.31
MPViT [ 53 ]
92.76 M
235.24 G
9.94
5266 MiB
47.44
57.77
Ours
71.82 M
75.28 G
20.33
3293 MiB
49.82
59.90
Table 8: Comparison results of model complexity. Based on EMSAFormer, we compare the proposed model with models using different encoders. The comparison metrics include the number of parameters, FLOPs, FPS, and video memory usage (VRAM).
Figure 11: Panoptic segmentation maps on NYUv2. Based on EMSAFormer, we compare our proposed model with models using different encoders. Our method withstands lighting variations and provides more complete segmentation of indoor plants and furniture.
Method
Mode
Backbone
Semantic mIoU ↑
MaskSup [ 62 ]
RGB
U-Net++
39.31
DIANet [ 35 ]
RGB
ResNet50
42.60
MTML [ 63 ]
RGB
ResNet50
41.51
FDNet [ 64 ]
RGB
DenseNet264
47.40
PGT [ 65 ]
RGB
Swin-S
46.43
SOSD-Net [ 31 ]
RGB
DeepLabv3 +
45.00
Table 9: Comparison results on the NYUv2 dataset. We compare our proposed model with existing semantic segmentation models and list the input modalities and backbones of these models.
Figure 12: Panoptic segmentation maps on the SUN RGB-D dataset. We compare the panoptic segmentation maps of our proposed model with those of EMSAFormer. Our method accurately identifies and segments objects under low-lighting and occlusion conditions.
Method
Mode
Backbone
Semantic mIoU ↑
DIANet [ 35 ]
RGB
ResNet50
43.10
CI-Net [ 33 ]
RGB
ResNet101
44.30
SSMA [ 23 ]
RGB
ResNet50
38.40
SSMA [ 23 ]
RGB-D
ResNet50
43.90
3DGNN [ 18 ]
RGB-D
VGG-Net
40.20
D-CNN [ 19 ]
RGB-D
VGG-Net
42.00
Table 10: Comparison results on the SUN RGB-D dataset. We compare our proposed model with existing semantic segmentation models, using semantic mIoU as the evaluation metric.
Figure 13: Visualization results on the Cityscapes dataset. We present the input information, ground truth (GT), the heatmap and segmentation map for semantic segmentation, as well as the instance center and offset for instance segmentation.
Method
Mode
Backbone
Semantic mIoU ↑
IkshanaNet [ 69 ]
RGB
IkshanaNet
54.40
SegNet [ 70 ]
RGB
VGG-Net
56.10
ENet [ 71 ]
RGB
ENet
58.30
Lovasz [ 72 ]
RGB
ENet
63.06
ESPNet [ 73 ]
RGB
ESPNet
60.30
EMSANet [ 3 ]
RGB-D
ResNet34 NBt1D
41.92
Table 11: Comparison results on the Cityscapes dataset. Semantic mIoU is adopted as the evaluation metric.
Previous RGB-D semantic segmentation methods commonly employ dual encoders to separately process RGB and depth inputs, followed by dedicated modules for cross-modal feature fusion. However, such designs often inadequately capture depth representations and consequently limit effective cross-modal interaction, while the additional encoder branch introduces redundant computation that hinders lightweight execution. To tackle these challenges, we propose URNet, a Unified Reparameterized RGB-D Network that performs simultaneous multi-modal feature extraction and cross-modal fusion within a single encoder. Specifically, we adopt a reparameterization strategy to compact the network architecture and facilitate fast inference. Within each Reparameterized Block (RepBlock), a Linear Gated Attention (LGA) module is introduced to fully exploit complementary RGB and depth cues across different feature scales. Furthermore, considering that decoder design has been relatively underexplored in existing RGB-D segmentation models, we develop a concise yet effective universal decoder, termed the Pyramid Merging Decoder (PMD). Extensive experiments on multiple RGB-D segmentation benchmarks demonstrate that URNet achieves state-of-the-art performance while maintaining high efficiency. Code will be available at https://github.com/Wild-Stephen/URNet.
Guoan Xu, Zhengxue Wang, Yang Xiao +3
University of Technology Sydney Sydney, New South Wales, Australia · Nanjing University of Science and Technology Nanjing, China · Honor Device Co., Ltd. Beijing, China +1
In this paper, we propose RGBD20K, a novel dataset for facilitating the development of more robust and general RGB-D semantic segmentation by encompassing abundant categories and high-quality annotations. RGBD20K possesses several attractive properties: (1) Expanded Semantic Space. In particular, it covers 160 fine-grained categories, largely surpassing the category diversity of existing popular RGB-D benchmarks (e.g., NYUv2 with 40 classes and SUN RGB-D with 37 classes). With such enriched semantic coverage, we expect to promote the learning of more generalizable segmentation models. (2) Larger Scale. Compared with current benchmarks, RGBD20K offers 20,000 RGB-D image pairs, providing a substantially larger training resource that benefits the development of more powerful deep models. (3) High-Fidelity Annotation. We perform rigorous re-evaluation and correction of existing labels to resolve long-standing annotation noise, resulting in a clean and reliable ground-truth foundation. Furthermore, we propose a novel score-purified fusion (SPF) method, which achieves state-of-the-art performance across all evaluated benchmarks, demonstrating the effectiveness of our approach in leveraging high-quality multimodal information for RGB-D semantic segmentation. The dataset is here: https://github.com/ShaohuaDong2021/RGBD20K/.
RGB-D semantic segmentation has made notable progress by fusing RGB and Depth, yet mainstream models still learn features almost exclusively from pixel-level supervision, lacking direct high-level semantic constraints. This raises a central question-can external knowledge such as language priors inject stronger semantic discriminability into mainstream RGB-D segmentation models. We present DTFormer, a novel tri-modal (RGB-D-Text) semantic segmentation framework. At its core is Text-guided Semantic Alignment Module (TSAM) that first encodes textual cues into a set of semantic prototypes and then explicitly aligns multi-modal RGB-D features with these prototypes at multiple encoder and decoder layers. This design imposes strong semantic regularization on representation learning, guiding the network toward more discriminative features. Extensive experiments on multiple benchmarks show that DTFormer delivers consistent gains while remaining simple and efficient. Our results demonstrate that explicit semantic alignment offers an effective and practical route to improving RGB-D semantic segmentation. The code will be released upon acceptance.
Ziang Wei, Yinlong Liu, Yan Xia +2
Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Germany · City University of Macau, China · University of Science and Technology of China, China +1