RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

Shaohua Dong; Zexuan Meng; Haiyan Sun; Bing Fan; Cuicui Zhang; Dylan Joseph; Kewei Sha; Yunhe Feng

cs.CVSep 24, 2026

RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

Authors: Shaohua Dong, Zexuan Meng, Haiyan Sun, Bing Fan, Cuicui Zhang, Dylan Joseph, Kewei Sha, Yunhe Feng, +1 more

Organizations: University of North Texas. Denton, TX 76207, USA.

Abstract

In this paper, we propose RGBD20K, a novel dataset for facilitating the development of more robust and general RGB-D semantic segmentation by encompassing abundant categories and high-quality annotations. RGBD20K possesses several attractive properties: (1) Expanded Semantic Space. In particular, it covers 160 fine-grained categories, largely surpassing the category diversity of existing popular RGB-D benchmarks (e.g., NYUv2 with 40 classes and SUN RGB-D with 37 classes). With such enriched semantic coverage, we expect to promote the learning of more generalizable segmentation models. (2) Larger Scale. Compared with current benchmarks, RGBD20K offers 20,000 RGB-D image pairs, providing a substantially larger training resource that benefits the development of more powerful deep models. (3) High-Fidelity Annotation. We perform rigorous re-evaluation and correction of existing labels to resolve long-standing annotation noise, resulting in a clean and reliable ground-truth foundation. Furthermore, we propose a novel score-purified fusion (SPF) method, which achieves state-of-the-art performance across all evaluated benchmarks, demonstrating the effectiveness of our approach in leveraging high-quality multimodal information for RGB-D semantic segmentation. The dataset is here: https://github.com/ShaohuaDong2021/RGBD20K/.

Figures & tables

Panel of Figure 1 — Fig. 1 : Comparison of the proposed RGBD20K with other RGB-D datasets.

Panel of Figure 2 — Fig. 2 : Visualization of RGBD20K distribution.

Dataset	Year	Categories	Images	Scenarios	Masks	Avg./img
NYUv1 [ 8 ]	2011	13	2,347	7	–	–
NYUv2 [ 9 ]	2012	40	1,449	26	33,749	23.3
SUN RGB-D [ 10 ]	2015	37	10,335	47	146,617	14.2
RGBD20K (Ours)	2026	160	20,000	75	368,312	18.4

TABLE I : Comparison of RGB-D semantic segmentation benchmarks.

Panel of Figure 4 — Fig. 3: Example images with annotations from RGBD20K.

Dataset	Images	Task
SUN RGB-D [ 10 ]	10,335	SS
RGB-D mirror [ 32 ]	824	SS
VidSOD [ 33 ]	100	SOD
DepthTrack [ 34 ]	98	VOT
RGBD1K [ 35 ]	235	VOT
ARKitTrack [ 36 ]	43	VOT

TABLE II: Summary of RGBD20K Dataset Collection Sources. (SS: Semantic Segmentation, SOD: Salient Object Detection, VOT: Visual Object Tracking, PT: Pre-training.)

Fig. 4 : The overall architecture of the proposed score-purified fusion model.

Panel of Figure 7 — Fig. 5 : Comparison of annotation quality. (a) Previous dataset, (b) our high-quality results.

	Backbone	Params	NYUv2	SUN RGB-D	SUN RGB-D †	RGBD20K
PGDENet [ 17 ]	ResNet-34	100.7M	53.7	51.0	54.6	22.4
TokenFusion [ 18 ]	MiT-B3	45.9M	54.2	51.4	56.2	32.8
GeminiFusion [ 21 ]	Swin-L-384	369.2M	60.2	54.6	61.5	45.4
MultiMAE [ 40 ]	ViT-B	95.2M	56.0	51.1	55.7	42.2
CMX [ 20 ]	MiT-B5	181.1M	56.9	52.4	55.7	43.9
CMNeXt [ 26 ]	MiT-B4	119.6M	56.9	51.9	55.4	43.1

TABLE III: Comparison of RGBD20K to existing datasets using mean IoU. † denotes our re-annotation.

Structure	mIoU (%)
Score-Purified Fusion	46.3
without Projection	45.7 (-0.6)
without Score	45.5 (-0.8)
with Score (K)	46.0 (-0.3)
with Score (V)	45.9 (-0.4)
with Score (RGB)	45.8 (-0.5)

TABLE IV: Ablation study on different components.

Table 10

Explore similar work

RDVSv2: A Large-scale Benchmark for RGB-D Video Salient Object Detection
Jul 28, 2026Tianyu Li, Jiahao He, Keren Fu +1Rgb-Thermal Object Detection Rgb-Depth Cameras
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, China · College of Computer Science, Sichuan University, Chengdu, China
Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropout
Jul 22, 2026Xuchen Zhu, Yajuan Wei, Shuang Hao +3Dropout Semantic Segmentation
Xi’an University of Posts & Telecommunications, Xi’an, China · Xi’an Jiaotong University, Xi’an, China · Huazhong University of Science and Technology, Wuhan, China +1
URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation
Aug 6, 2026Guoan Xu, Zhengxue Wang, Yang Xiao +3U-Net Cross-Modal Fusion
University of Technology Sydney Sydney, New South Wales, Australia · Nanjing University of Science and Technology Nanjing, China · Honor Device Co., Ltd. Beijing, China +1

Jul 28, 2026, cs.CVJ/K move · Enter open · S save

RDVSv2: A Large-scale Benchmark for RGB-D Video Salient Object Detection

Tianyu Li, Jiahao He, Keren Fu, Qijun Zhao

National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, China · College of Computer Science, Sichuan University, Chengdu, China

Rgb-Thermal Object Detection Rgb-Depth Cameras Dense Annotation Large-Scale Benchmark

Open paper AskPDF arXiv

We introduce RDVSv2, a large-scale benchmark for RGB-D video salient object detection (RGB-D VSOD) with dense frame-level annotations. Existing datasets in this emerging field are often limited in scale and annotation quality, while also relying on less geometry-consistent depth cues. To address these limitations, RDVSv2 is built from publicly accessible stereoscopic online videos and contains 249 video sequences with 29,077 annotated frames. It includes depth maps derived from stereoscopic videos, together with frame-wise salient object masks annotated with eye-tracking guidance. Compared with existing datasets, RDVSv2 is much larger in scale and covers more diverse and challenging scenarios. In addition, we establish a strong baseline for RGB-D VSOD based on Segment Anything Model 2 (SAM2). Specifically, we employ a parameter-efficient fine-tuning (PEFT) strategy to adapt the SAM2 encoder to jointly encode RGB, depth, and optical flow cues. Extensive experiments show that RDVSv2 is substantially more challenging for existing RGB-D VSOD methods. Meanwhile, the proposed baseline achieves state-of-the-art results on RDVSv2 and existing RGB-D VSOD benchmarks. We hope that RDVSv2 and the provided baseline will serve as useful resources for future research on RGB-D VSOD and related multi-modal video understanding tasks. Our dataset and code will be available at https://github.com/ltynick/RDVSv2.

Abstract

Explore similar work

RDVSv2: A Large-scale Benchmark for RGB-D Video Salient Object Detection

Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropout

URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation