Organizations: Department of Aerospace Engineering, Indian Institute of Science, Bengaluru, India · Robert Bosch Centre for Cyber Physical Systems, Indian Institute of Science, Bengaluru, India
Domain adaptation for semantic segmentation is challenging when target-domain images are unavailable during training. Zero-shot domain adaptation addresses this by adapting models using only textual descriptions of the target domain. However, existing prompt-learning methods are not well suited for ResNet-based CLIP vision backbones or pixel-level cross-modal feature alignment. We propose CROMPT, a prompt-learning framework for zero-shot domain adaptation in semantic segmentation. CROMPT introduces Cross-Modal Feature Augmentation (CROMA), combining Random Visual Prompting with Hybrid Textual Prompting (HTP) to jointly adapt visual and textual representations. We further propose CROMA-VL, which leverages vision-language models to generate domain-conditioned prompts, and a progressive easy-to-hard adaptation strategy for improved domain invariance. Experiments on two benchmark datasets demonstrate consistent improvements over strong baselines and state-of-the-art methods, achieving 1.63-2.63% mIoU gains while maintaining inference-time complexity. These results establish CROMPT as an effective and efficient framework for zero-shot multi-domain semantic segmentation. OpenReview: https://openreview.net/forum?id=FxqFsIL685
Figures & tables
Figure 1: Overview of the proposed zero-shot feature augmentation framework. The source image is processed by the CLIP image encoder (V) to produce target-style features, which are simultaneously aligned with domain-specific target embeddings (e.g., τnight,τrain ). RVP adds prompts directly to the image, while HTP combines hard prompts with learnable textual prompts for the CLIP text encoder to guide domain-aware feature adaptation.
Figure 2: Model adaptation with simulated features using CROMPT. Step 1 applies learnable prompts to hard domains and hard-coded prompts to easy domains, producing the intermediate model Mmid . In Step 2, the prompt assignments are swapped, learnable prompts are applied to easy domains and hard-coded prompts to hard domains, and the intermediate model Mmid is used to initialize this stage, producing the final adapted model Madp .
Figure 3: Comparison of output predictions under adverse weather conditions between PODA and our model. Our method produces more accurate and consistent segmentation across challenging scenarios.
Table 1: Performance comparison (mIoU %) for zero-shot domain adaptation. Cityscapes is used as the source domain, with ACDC and GTA5 as target domains. * denotes results reproduced from official code, and TrgPrompt refers to the target prompt.
Method
CROMA
CROMA-VL
CROMPT
Mean (mIoU)
ULDA
39.25
ULDA
✓
40.69
ULDA
✓
✓
40.81
ULDA
✓
✓
✓
40.88
ULDA
✓
✓
40.91
Table 2: Ablation study showing the incremental contribution of CROMA, CROMA-VL, and CROMPT on top of ULDA.
Source
Target Eval.
One-shot SM-PPM
Zero-shot CROMPT
CS ↦
ACDC Night
13.07 / 14.60 ( Δ =1.53)
18.31 / 23.29 ( Δ = 4.98 )
CS ↦
ACDC Snow
32.60 / 35.61 ( Δ =3.01)
39.28 / 44.04 ( Δ = 4.76 )
CS ↦
ACDC Rain
29.78 / 32.23 ( Δ =2.45)
38.20 / 44.41 ( Δ = 6.21 )
Table 3: Segmentation performance (mIoU %) of source and adapted models, with Δ indicating gains over the source-only baseline. SM-PPM uses one unlabeled target image per domain for adaptation, whereas CROMPT operates in a fully zero-shot manner without target data.
Easy domains (R/S/F)
Hard domains (R/F/N)
Mean IoU
CoOp
CoOp
39.45
CoOp
Hand-coded Description
40.43
Hand-coded Description
CoOp
39.56
Qwen-VL
Qwen-VL
39.99
CoOp
Qwen-VL
40.81
Table 4: Comparison of prompting strategies across Easy (R/S/F) and Hard (R/F/N) domains.
Stage 1
Stage 2
mIoU
Learnable: S/G/F Hard Desc.: N/R/F
Learnable: N/R/F Hard Desc.: S/G/F
40.19
Learnable: N/R/F Hard Desc.: S/G/F
Learnable: S/G/F Hard Desc.: N/R/F
40.91
Learnable: N/R/F Qwen-VL: S/G/F
Learnable: S/G/F Qwen-VL: N/R/F
40.76
Learnable: N/R/F Hard Desc.: S/G/F
Learnable: S/G/F Qwen-VL: N/R/F
40.88
Table 5: Two-stage progressive adaptation analysis comparing Learnable prompts, Hard Prompt Descriptions, and Qwen-VL across easy (S, G, F) and hard (N, R, F) domains.
Prompts
Setting
ACDC Mean
Visual Prompts (Learnable)
Padded Random Patch Corner + Middle
39.34 39.65 39.62
Visual Prompts (Not Learnable)
Corner + Middle
40.81
Table 6: Ablation study comparing learnable visual prompts with non-learnable (noise-based) prompts.
Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia +1