Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance

Guodong Sun; Junjie Liu; Gaoyang Zhang; Bo Wu; Yang Zhang

cs.CVMar 8, 2026

Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance

Authors: Guodong Sun, Junjie Liu, Gaoyang Zhang, Bo Wu, Yang Zhang

Organizations: School of Mechanical Engineering, Hubei University of Technology, Wuhan, 430068, China · Hubei Key Laboratory of Modern Manufacturing Quality Engineering, Hubei University of Technology, Wuhan, 430068, China · Vehicle Measurement, Control and Safety Key Laboratory of Sichuan Province, Xihua University, Chengdu, 610039, China · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, 201210, China ·

Abstract

Scene understanding plays a critical role in enabling intelligence and autonomy in robotic systems. Traditional approaches often face challenges, including occlusions, ambiguous boundaries, and the inability to adapt attention based on task-specific requirements and sample variations. To address these limitations, this paper presents an efficient RGB-D scene understanding model that performs a range of tasks, including semantic segmentation, instance segmentation, orientation estimation, panoptic segmentation, and scene classification. The proposed model incorporates an enhanced fusion encoder, which effectively leverages redundant information from both RGB and depth inputs. For semantic segmentation, we introduce normalized focus channel layers and a context feature interaction layer, designed to mitigate issues such as shallow feature misguidance and insufficient local-global feature representation. The instance segmentation task benefits from a non-bottleneck 1D structure, which achieves superior contour representation with fewer parameters. Additionally, we propose a multi-task adaptive loss function that dynamically adjusts the learning strategy for different tasks based on scene variations. Extensive experiments on the NYUv2, SUN RGB-D, and Cityscapes datasets demonstrate that our approach outperforms existing methods in both segmentation accuracy and processing speed.

Figures & tables

Figure 1: Multi-task outputs of the network. Semantic segmentation provides foreground masks for instance segmentation, and pixel-level instances are generated by combining instance centers with instance offsets. Panoptic segmentation is achieved by integrating semantic and instance segmentation.

Figure 2: Multi-task scene understanding network structure. This network features an improved feature fusion encoder that handles redundant information from RGB and depth to enhance feature extraction. The semantic decoder includes NFCL and CFIL to enrich scene representations from various dimensions. The instance decoder employs the non-bottleneck 1D architecture to generate instance centers, instance offsets, and raw orientations. The integration of instance segmentation with semantic segmentation enables panoptic segmentation. Scene classification is performed by a task head with a fully connected layer. In addition, the multi-task adaptive loss function optimizes the training strategy based on data variations.

Figure 3: Fusion encoder. This encoder adopts merging layers for channel expansion and downsampling. Given the similarity between channel features, the fusion blocks extract redundant channel features to optimize computational efficiency.

Figure 4: Normalized focus channel layer. This layer learns the variance adjustment parameters via batch normalization. It then computes channel weights that quantify the importance of each channel. The input features are rearranged, weighted according to these channel weights, and subsequently processed through an activation function.

Figure 5: Context feature interaction layer. This layer addresses the limitation of the semantic decoder in integrating local and global information by employing a multi-scale pooling operation and efficient channel compression.

Panel of Figure 6 — Figure 6: Semantic segmentation heat maps. Combining the feature guidance of CFIL and NFCL enables better handling of objects with similar colors and low-light scenes.

Figure 7: Non-bottleneck 1D module. This module decomposes 2D convolution into 1D convolutions combined with a non-linear activation function. This decomposition reduces model redundancy and enhances the non-linear decision-making capability.

Method	Instance		Panoptic					Scene	Semantic
Method	PQ $\uparrow$	MAAE $\downarrow$	mIoU $\uparrow$	PQ $\uparrow$	RQ $\uparrow$	SQ $\uparrow$	MAAE $\downarrow$	bAcc $\uparrow$	mIoU $\uparrow$
Swin v2 [ 5 ]	58.49	21.09	50.51	43.08	52.22	81.34	20.46	77.11	49.76
ConvNeXt v2 [ 54 ]	41.04	31.24	27.20	18.21	24.11	72.20	29.29	61.79	27.69
MPViT [ 53 ]	57.77	21.18	47.09	39.43	47.78	81.13	19.82	75.16	47.44
MetaFormer [ 53 ]	53.31	23.69	44.75	34.01	42.96	77.86	22.10	74.51	43.27
Ours	58.59	18.67	47.37	40.35	49.18	81.03	17.88	74.67	46.83

Table 1: Encoder comparison results on the NYUv2 dataset. The symbols

\uparrow

and

\downarrow

indicate whether higher or lower values are better, respectively. The best metric is indicated in bold .

Panel of Figure 8 — Figure 8: Feature visualization of backbones. We visualize the model inputs and the features extracted by Swin Transformer v2 and our method. Compared to the blurred features of Swin Transformer v2, our method more distinctly highlights objects in the scene, demonstrating better suitability for segmentation tasks.

Method	Instance		Panoptic					Scene	Semantic
Method	PQ $\uparrow$	MAAE $\downarrow$	mIoU $\uparrow$	PQ $\uparrow$	RQ $\uparrow$	SQ $\uparrow$	MAAE $\downarrow$	bAcc $\uparrow$	mIoU $\uparrow$
SPPELAN [ 55 ]	58.79	19.18	47.55	41.09	49.60	81.66	18.18	77.39	46.51
ASPP [ 56 ]	58.75	18.14	48.40	41.81	50.51	81.61	17.83	76.73	47.98
SimSPPF [ 57 ]	58.34	19.07	47.62	40.56	49.12	81.37	18.03	74.70	46.40
RFB [ 58 ]	58.98	18.08	48.46	42.15	50.83	81.88	17.75	76.51	47.14
Ours	59.25	18.27	50.16	42.83	51.78	81.60	17.56	77.00	49.72

Table 2: Comparison of context modules. We compare the proposed context feature interaction layer with existing context modules. They are used in the multi-layer feature fusion stage of the semantic decoder.

Method	Instance		Panoptic					Scene	Semantic
Method	PQ $\uparrow$	MAAE $\downarrow$	mIoU $\uparrow$	PQ $\uparrow$	RQ $\uparrow$	SQ $\uparrow$	MAAE $\downarrow$	bAcc $\uparrow$	mIoU $\uparrow$
P0	58.39	18.41	46.93	40.61	49.14	81.54	18.06	77.58	46.36
P1	59.37	19.34	48.39	41.81	51.02	80.78	18.58	76.23	47.72
P2	58.71	16.63	48.17	41.44	50.21	81.55	15.63	77.89	47.12
P3	59.49	17.90	48.99	42.40	51.24	81.56	17.25	75.08	49.25
P4	58.88	19.78	47.62	40.52	49.05	81.60	18.41	75.21	47.13

Table 3: Comparison of the context feature interaction layer positions. P0: CFIL not used. P1: encoder. P2: instance decoder. P3: semantic decoder and instance decoder. P4: encoder and semantic decoder. Ours: semantic decoder.

Method	Instance		Panoptic					Scene	Semantic
Method	PQ $\uparrow$	MAAE $\downarrow$	mIoU $\uparrow$	PQ $\uparrow$	RQ $\uparrow$	SQ $\uparrow$	MAAE $\downarrow$	bAcc $\uparrow$	mIoU $\uparrow$
L0	59.15	19.03	49.37	42.00	50.61	81.97	17.74	76.48	48.93
L1	58.78	19.88	49.24	42.52	51.60	81.21	18.08	74.64	48.88
L2	59.37	18.42	50.01	42.74	51.71	81.5	17.78	76.42	49.53
L3	58.97	18.03	48.41	41.65	50.32	81.76	17.41	76.35	48.45
Ours	59.90	19.64	50.21	43.21	52.58	81.20	18.79	76.57	49.82

Table 4: Comparison of the normalized focus channel layers on the semantic decoder. L0: 1st, 2nd, 3rd, and 4th layers. L1: 2nd and 3rd layers. L2: 1st and 2nd layers. L3: 2nd, 3rd, and 4th layers. Ours: 1st, 2nd, and 3rd layers.

Method	Instance		Panoptic					Scene	Semantic
Method	PQ $\uparrow$	MAAE $\downarrow$	mIoU $\uparrow$	PQ $\uparrow$	RQ $\uparrow$	SQ $\uparrow$	MAAE $\downarrow$	bAcc $\uparrow$	mIoU $\uparrow$
BasicBlock [ 59 ]	58.51	18.40	49.33	42.29	51.17	81.45	17.40	76.87	49.17
Bottleneck [ 59 ]	57.97	18.80	50.33	41.97	50.93	81.23	18.20	76.41	50.26
MobileBottleneck [ 60 ]	58.24	20.45	48.46	40.85	49.45	81.38	19.54	75.03	48.66
GhostBottleneck [ 61 ]	58.14	20.29	48.93	41.82	51.02	80.74	19.72	75.86	48.75
Ours	59.25	18.27	50.16	42.83	51.78	81.60	17.56	77.00	49.72

Table 5: Comparison of feature extraction modules. In the instance decoder layer, we compare the non-bottleneck 1D module with BasicBlock, Bottleneck, MobileBottleneck, and GhostBottleneck.

Panel of Figure 9 — Figure 9: Comparison plot of multi-task adaptive loss. Figure a and Figure b represent the loss comparison and loss variance, respectively. The blue line represents the original fixed-weight loss, while the orange line represents the multi-task adaptive loss. To make the comparison clearer, we record the loss values and variances from epoch 100 onward. Our method makes the training process more convergent and stable.

Method	Instance		Panoptic					Scene	Semantic
Method	PQ $\uparrow$	MAAE $\downarrow$	mIoU $\uparrow$	PQ $\uparrow$	RQ $\uparrow$	SQ $\uparrow$	MAAE $\downarrow$	bAcc $\uparrow$	mIoU $\uparrow$
Fixed	58.59	18.67	47.37	40.35	49.18	81.03	17.88	74.67	46.83
0.1	57.66	19.90	47.17	40.05	49.05	80.63	19.15	72.99	47.21
0.05	58.86	19.06	47.72	40.07	48.74	80.96	18.33	73.59	46.74
0.02	59.40	19.02	48.25	41.56	50.34	81.44	18.21	74.59	47.63
0.01	59.37	19.34	48.39	41.81	51.02	80.78	18.58	76.23	47.72

Table 6: Comparison of adjustment factors for multi-task adaptive loss. "Fixed" represents the original fixed weight. The numbers denote the adjustment factors, with our method using an adjustment factor of 0.01.

Fusion	Adaptive	CFIL	NFCL	Instance		Panoptic					Scene	Semantic
Fusion	Adaptive	CFIL	NFCL	PQ $\uparrow$	MAAE $\downarrow$	mIoU $\uparrow$	PQ $\uparrow$	RQ $\uparrow$	SQ $\uparrow$	MAAE $\downarrow$	bAcc $\uparrow$	mIoU $\uparrow$
				58.49	21.09	50.51	43.08	52.22	81.34	20.46	77.11	49.76
✓				58.59	18.67	47.37	40.35	49.18	81.03	17.88	74.67	46.83
✓	✓			59.37	19.34	48.39	41.81	51.02	80.78	18.58	76.23	47.72
✓	✓	✓		59.25	18.27	50.16	42.83	51.78	81.60	17.56	77.00	49.72
✓	✓	✓	✓	59.90	19.64	50.21	43.21	52.58	81.20	18.79	76.57	49.82

Table 7: Comparison results of the ablation study on the overall framework. Fusion: improved fusion encoder. Adaptive: multi-task adaptive loss. CFIL: context feature interaction layer. NFCL: normalized focus channel layer.

Panel of Figure 10 — Figure 10: Visualization of the ablation experiment. We present the panoptic heatmaps for the model with the improvements: fusion encoder, multi-task adaptive loss, context feature interaction layer, and normalized focus channel layer. Benefiting from the adaptive learning strategy and cross-dimensional feature guidance, the model achieves more accurate recognition and contour representation.

Method	Params	FLOPs	FPS	VRAM	Semantic mIoU $\uparrow$	Instance PQ $\uparrow$
EMSAFormer [ 4 ]	72.08 M	50.66 G	16.32	3188 MiB	49.76	58.49
ConvNeXt v2 [ 54 ]	111.84 M	112.38 G	14.42	4592 MiB	27.69	41.04
MetaFormer [ 52 ]	76.14 M	72.06 G	12.01	3620 MiB	43.27	53.31
MPViT [ 53 ]	92.76 M	235.24 G	9.94	5266 MiB	47.44	57.77
Ours	71.82 M	75.28 G	20.33	3293 MiB	49.82	59.90

Table 8: Comparison results of model complexity. Based on EMSAFormer, we compare the proposed model with models using different encoders. The comparison metrics include the number of parameters, FLOPs, FPS, and video memory usage (VRAM).

Panel of Figure 11 — Figure 11: Panoptic segmentation maps on NYUv2. Based on EMSAFormer, we compare our proposed model with models using different encoders. Our method withstands lighting variations and provides more complete segmentation of indoor plants and furniture.

Method	Mode	Backbone	Semantic mIoU $\uparrow$
MaskSup [ 62 ]	RGB	U-Net++	39.31
DIANet [ 35 ]	RGB	ResNet50	42.60
MTML [ 63 ]	RGB	ResNet50	41.51
FDNet [ 64 ]	RGB	DenseNet264	47.40
PGT [ 65 ]	RGB	Swin-S	46.43
SOSD-Net [ 31 ]	RGB	DeepLabv3 $+$	45.00

Table 9: Comparison results on the NYUv2 dataset. We compare our proposed model with existing semantic segmentation models and list the input modalities and backbones of these models.

Panel of Figure 12 — Figure 12: Panoptic segmentation maps on the SUN RGB-D dataset. We compare the panoptic segmentation maps of our proposed model with those of EMSAFormer. Our method accurately identifies and segments objects under low-lighting and occlusion conditions.

Method	Mode	Backbone	Semantic mIoU $\uparrow$
DIANet [ 35 ]	RGB	ResNet50	43.10
CI-Net [ 33 ]	RGB	ResNet101	44.30
SSMA [ 23 ]	RGB	ResNet50	38.40
SSMA [ 23 ]	RGB-D	ResNet50	43.90
3DGNN [ 18 ]	RGB-D	VGG-Net	40.20
D-CNN [ 19 ]	RGB-D	VGG-Net	42.00

Table 10: Comparison results on the SUN RGB-D dataset. We compare our proposed model with existing semantic segmentation models, using semantic mIoU as the evaluation metric.

Figure 13: Visualization results on the Cityscapes dataset. We present the input information, ground truth (GT), the heatmap and segmentation map for semantic segmentation, as well as the instance center and offset for instance segmentation.

Method	Mode	Backbone	Semantic mIoU $\uparrow$
IkshanaNet [ 69 ]	RGB	IkshanaNet	54.40
SegNet [ 70 ]	RGB	VGG-Net	56.10
ENet [ 71 ]	RGB	ENet	58.30
Lovasz [ 72 ]	RGB	ENet	63.06
ESPNet [ 73 ]	RGB	ESPNet	60.30
EMSANet [ 3 ]	RGB-D	ResNet34 NBt1D	41.92

Table 11: Comparison results on the Cityscapes dataset. Semantic mIoU is adopted as the evaluation metric.

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 295 papers

URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation
Aug 6, 2026Guoan Xu, Zhengxue Wang, Yang Xiao +3Multi-Scale Feature Fusion Multimodal Fusion
University of Technology Sydney Sydney, New South Wales, Australia · Nanjing University of Science and Technology Nanjing, China · Honor Device Co., Ltd. Beijing, China +1
RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation
Sep 24, 2026Shaohua Dong, Zexuan Meng, Haiyan Sun +6Multimodal Fusion Semantic Segmentation
University of North Texas. Denton, TX 76207, USA.
DTFormer: Text-Guided Semantic Alignment for RGB-D Segmentation
Oct 4, 2026Ziang Wei, Yinlong Liu, Yan Xia +2Semantic Segmentation Multimodal Semantic Segmentation
Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Germany · City University of Macau, China · University of Science and Technology of China, China +1

Aug 6, 2026J/K move · Enter open · S save

URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation

Guoan Xu, Zhengxue Wang, Yang Xiao, Ligeng Chen+2

University of Technology Sydney Sydney, New South Wales, Australia · Nanjing University of Science and Technology Nanjing, China · Honor Device Co., Ltd. Beijing, China+1

Multi-Scale Feature Fusion Multimodal Fusion Semantic Segmentation

Open paper AskPDF arXiv

Previous RGB-D semantic segmentation methods commonly employ dual encoders to separately process RGB and depth inputs, followed by dedicated modules for cross-modal feature fusion. However, such designs often inadequately capture depth representations and consequently limit effective cross-modal interaction, while the additional encoder branch introduces redundant computation that hinders lightweight execution. To tackle these challenges, we propose URNet, a Unified Reparameterized RGB-D Network that performs simultaneous multi-modal feature extraction and cross-modal fusion within a single encoder. Specifically, we adopt a reparameterization strategy to compact the network architecture and facilitate fast inference. Within each Reparameterized Block (RepBlock), a Linear Gated Attention (LGA) module is introduced to fully exploit complementary RGB and depth cues across different feature scales. Furthermore, considering that decoder design has been relatively underexplored in existing RGB-D segmentation models, we develop a concise yet effective universal decoder, termed the Pyramid Merging Decoder (PMD). Extensive experiments on multiple RGB-D segmentation benchmarks demonstrate that URNet achieves state-of-the-art performance while maintaining high efficiency. Code will be available at https://github.com/Wild-Stephen/URNet.

Abstract

3D Scene Understanding

Explore similar work

URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation

RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

DTFormer: Text-Guided Semantic Alignment for RGB-D Segmentation