GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depth

Krishna Jaganathan; Patricio Vela

cs.CVApr 13, 2026

GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depth

Authors: Krishna Jaganathan, Patricio Vela

Organizations: Georgia Institute of Technology

Abstract

Multimodal perception systems for robotics and embodied AI often assume reliable RGB-D sensing, but in practice, depth is frequently missing, noisy, or corrupted. We thus present GeomPrompt, a lightweight cross-modal adaptation module that synthesizes a task-driven geometric prompt from RGB alone for the fourth channel of a frozen RGB-D semantic segmentation model, without depth supervision. We further introduce GeomPrompt-Recovery, an adaptation module that compensates for degraded depth by predicting the fourth channel correction relevant for the frozen segmenter. Both modules are trained solely with downstream segmentation supervision, enabling recovery of the geometric prior useful for segmentation, rather than estimating depth signals. On SUN RGB-D, GeomPrompt improves over RGB-only inference by +6.1 mIoU on DFormer and +3.0 mIoU on GeminiFusion, while remaining competitive with strong monocular depth estimators. For degraded depth, GeomPrompt-Recovery consistently improves robustness, yielding gains up to +3.6 mIoU under severe depth corruptions. GeomPrompt is also substantially more efficient than monocular depth baselines, reaching 7.8 ms latency versus 38.3 ms and 71.9 ms. These results suggest that task-driven geometric prompting is an efficient mechanism for cross-modal compensation under missing and degraded depth inputs in RGB-D perception.

Figures & tables

Panel of Figure 2 — Figure 2: Model architecture diagrams. (a) Overview of the GeomPrompt architecture. (b) Depth handling in GeomPrompt-Recovery.

	DFormer		GeminiFusion
Method	mIoU $\uparrow$	PA $\uparrow$	mIoU $\uparrow$	PA $\uparrow$
GT Depth	51.2	83.4	52.7	82.8
RGB-only	41.7	78.3	43.4	78.8
DA2	44.0	80.8	47.7	81.4
DA2 [Hypersim]	47.5	81.8	44.5	79.6
Metric3Dv2	46.6	81.8	46.6	80.6

Table 1: Evaluation on DFormer and GeminiFusion. GT Depth is shown as an upper bound. Best non-upper-bound results for each benchmark are in bold.

Figure 3: Degraded depth vs. our recovered prompt across severities and degradation types.

Degradation	Mean mIoU Gain $\uparrow$	High-Severity Gain $\uparrow$
Quantization	+1.4	+2.3
Dropout	+2.0	+1.5
Noise	+2.5	+3.6

Table 2: Recovered gains across different degradation types.

Model	Lat. (ms) $\downarrow$	FLOPs (G) $\downarrow$	Params (M) $\downarrow$
DA2	38.3	280.7	97.5
Metric3Dv2	71.9	315.2	37.5
$\bigstar$ GeomPrompt	7.8	44.0	23.4

Table 3: Efficiency comparison across methods. Lower is better.

Method	DFormer $\uparrow$	GeminiFusion $\uparrow$
RGB-only	41.7	43.4
Luminance Gray	25.4	43.2
Laplacian Edges	40.3	42.4
Scharr	39.8	43.0
Canny	38.6	43.5
$\bigstar$ GeomPrompt	47.8	46.4

Table 4: Naive baselines and references, separated by segmenter. Best results in each segmenter group are in bold.

Figure 4: mIoU of training ablations vs. our baseline.

Figure 5: Qualitative comparison of segmentation outputs and geometric representations.

Figure 6: Visual comparison of our synthesized geometric prompt against the original RGB input and ground truth depth.

Figure 7: GeomPrompt-Recovery under simulated depth failures.

Explore similar work

Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropout
Jul 22, 2026Xuchen Zhu, Yajuan Wei, Shuang Hao +3Missing-Modality Learning Multimodal Robustness
Xi’an University of Posts & Telecommunications, Xi’an, China · Xi’an Jiaotong University, Xi’an, China · Huazhong University of Science and Technology, Wuhan, China +1
When Depth Hurts: Reliability-Aware Geometry Distillation for Depth-Free RGB-D Salient Object Detection
Sep 3, 2026Xuehao Wang, Jiaxin Hua, Runmei Li +4Salient Object Detection
University of International Business and Economics · State Key Laboratory of Virtual Reality Technology and Systems · Southwest Jiaotong University +2
Any to Full: Prompting Depth Anything for Depth Completion in One Stage
Mar 5, 2026Zhiyuan Zhou, Ruofeng Liu, Taichi Liu +4Depth Completion Prompt Tuning
Rutgers University, USA · Michigan State University, USA · JD Logistics, China +1

Jul 22, 2026J/K move · Enter open · S save

Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropout

Xuchen Zhu, Yajuan Wei, Shuang Hao, Jiwei Jiang+2

Xi’an University of Posts & Telecommunications, Xi’an, China · Xi’an Jiaotong University, Xi’an, China · Huazhong University of Science and Technology, Wuhan, China+1

Missing-Modality Learning Multimodal Robustness Semantic Segmentation

Open paper AskPDF arXiv

RGB-D semantic segmentation has achieved remarkable progress, yet most models assume that RGB and depth are always available. In practice, failures or occlusions of surveillance sensors often remove one modality. Although RGB or depth alone can contain sufficient cues, models trained only on full-modality inputs fail to exploit the remaining modality once one is missing, causing severe degradation. We tackle this issue with a simple continued-training paradigm, Condition Dropout (ConD), which mitigates degradation while preserving full-modality accuracy. Starting from a pretrained RGB-D model, ConD adds a second stage that randomly simulates complete, RGB-missing, and depth-missing inputs, freezes the original encoders, and trains copied encoders with zero-initialized feature injection. Experiments on NYU-Depth V2 and SUN RGB-D show that ConD improves robustness under missing modalities and even yields slight gains when modalities are complete. Our code will be made publicly available upon acceptance.

Abstract

Explore similar work

Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropout

When Depth Hurts: Reliability-Aware Geometry Distillation for Depth-Free RGB-D Salient Object Detection

Any to Full: Prompting Depth Anything for Depth Completion in One Stage