Organizations: Department of Aerospace Engineering, Indian Institute of Science, Bengaluru, India · Robert Bosch Centre for Cyber Physical Systems, Indian Institute of Science, Bengaluru, India
Domain adaptation for semantic segmentation is challenging when target-domain images are unavailable during training. Zero-shot domain adaptation addresses this by adapting models using only textual descriptions of the target domain. However, existing prompt-learning methods are not well suited for ResNet-based CLIP vision backbones or pixel-level cross-modal feature alignment. We propose CROMPT, a prompt-learning framework for zero-shot domain adaptation in semantic segmentation. CROMPT introduces Cross-Modal Feature Augmentation (CROMA), combining Random Visual Prompting with Hybrid Textual Prompting (HTP) to jointly adapt visual and textual representations. We further propose CROMA-VL, which leverages vision-language models to generate domain-conditioned prompts, and a progressive easy-to-hard adaptation strategy for improved domain invariance. Experiments on two benchmark datasets demonstrate consistent improvements over strong baselines and state-of-the-art methods, achieving 1.63-2.63% mIoU gains while maintaining inference-time complexity. These results establish CROMPT as an effective and efficient framework for zero-shot multi-domain semantic segmentation. OpenReview: https://openreview.net/forum?id=FxqFsIL685
Figures & tables
Figure 1: Overview of the proposed zero-shot feature augmentation framework. The source image is processed by the CLIP image encoder (V) to produce target-style features, which are simultaneously aligned with domain-specific target embeddings (e.g., τnight,τrain ). RVP adds prompts directly to the image, while HTP combines hard prompts with learnable textual prompts for the CLIP text encoder to guide domain-aware feature adaptation.
Figure 2: Model adaptation with simulated features using CROMPT. Step 1 applies learnable prompts to hard domains and hard-coded prompts to easy domains, producing the intermediate model Mmid . In Step 2, the prompt assignments are swapped, learnable prompts are applied to easy domains and hard-coded prompts to hard domains, and the intermediate model Mmid is used to initialize this stage, producing the final adapted model Madp .
Figure 3: Comparison of output predictions under adverse weather conditions between PODA and our model. Our method produces more accurate and consistent segmentation across challenging scenarios.
Table 1: Performance comparison (mIoU %) for zero-shot domain adaptation. Cityscapes is used as the source domain, with ACDC and GTA5 as target domains. * denotes results reproduced from official code, and TrgPrompt refers to the target prompt.
Method
CROMA
CROMA-VL
CROMPT
Mean (mIoU)
ULDA
39.25
ULDA
✓
40.69
ULDA
✓
✓
40.81
ULDA
✓
✓
✓
40.88
ULDA
✓
✓
40.91
Table 2: Ablation study showing the incremental contribution of CROMA, CROMA-VL, and CROMPT on top of ULDA.
Source
Target Eval.
One-shot SM-PPM
Zero-shot CROMPT
CS ↦
ACDC Night
13.07 / 14.60 ( Δ =1.53)
18.31 / 23.29 ( Δ = 4.98 )
CS ↦
ACDC Snow
32.60 / 35.61 ( Δ =3.01)
39.28 / 44.04 ( Δ = 4.76 )
CS ↦
ACDC Rain
29.78 / 32.23 ( Δ =2.45)
38.20 / 44.41 ( Δ = 6.21 )
Table 3: Segmentation performance (mIoU %) of source and adapted models, with Δ indicating gains over the source-only baseline. SM-PPM uses one unlabeled target image per domain for adaptation, whereas CROMPT operates in a fully zero-shot manner without target data.
Easy domains (R/S/F)
Hard domains (R/F/N)
Mean IoU
CoOp
CoOp
39.45
CoOp
Hand-coded Description
40.43
Hand-coded Description
CoOp
39.56
Qwen-VL
Qwen-VL
39.99
CoOp
Qwen-VL
40.81
Table 4: Comparison of prompting strategies across Easy (R/S/F) and Hard (R/F/N) domains.
Stage 1
Stage 2
mIoU
Learnable: S/G/F Hard Desc.: N/R/F
Learnable: N/R/F Hard Desc.: S/G/F
40.19
Learnable: N/R/F Hard Desc.: S/G/F
Learnable: S/G/F Hard Desc.: N/R/F
40.91
Learnable: N/R/F Qwen-VL: S/G/F
Learnable: S/G/F Qwen-VL: N/R/F
40.76
Learnable: N/R/F Hard Desc.: S/G/F
Learnable: S/G/F Qwen-VL: N/R/F
40.88
Table 5: Two-stage progressive adaptation analysis comparing Learnable prompts, Hard Prompt Descriptions, and Qwen-VL across easy (S, G, F) and hard (N, R, F) domains.
Prompts
Setting
ACDC Mean
Visual Prompts (Learnable)
Padded Random Patch Corner + Middle
39.34 39.65 39.62
Visual Prompts (Not Learnable)
Corner + Middle
40.81
Table 6: Ablation study comparing learnable visual prompts with non-learnable (noise-based) prompts.
Segmenting images is critical for visual understanding but demands extensive pixel-level annotations. Foundational models have enabled new paradigms for predicting new classes guided by textual prompts, without annotations from the target domain. Yet, on specialized target domains, far from the original pre-training, their performance degrades. We study the errors of existing methods under such domain-shift, finding that misclassification rather than mask generation is the main culprit. To address this, we introduce the novel problem of Few-Shot Visual Adaptation for text-prompted Segmentation. This kind of adaptation has been largely studied for image classification, but it remains unexplored for segmentation. We tackle this task with Prototype Adaptation (PrAda), a novel, parameter-efficient method that adapts a frozen text-prompted segmentation model. Our approach learns class-specific prototypes by combining fine-grained pixel features and high-level transformer representations, which are then fused with the original text-based predictions through a learned importance factor. This preserves the model's zero-shot potential while enabling strong adaptation to new domains. Experiments across semantic, instance, and panoptic segmentation on five benchmarks demonstrate that PrAda yields significant improvements over state-of-the-art and proposed baselines.
Domain generalization (DG) aims to improve the generalization of the model to any unseen domains. Targeting this goal, existing DG methods mainly focus on learning domain-invariant features or prompts from the given source domains. However, these methods rely on source-domain data, and collecting and annotating large-scale single- or multi-source datasets is often infeasible, limiting the practical applicability of DG. To tackle this challenge, we propose {Do}main Prompt Learning (DoP), a method built on a large vision-language model (e.g., CLIP) for zero-shot domain generalization, which recognizes unseen classes in unseen target domains without access to source-domain data. DoP essentially learns the target domain knowledge via prompt both in text and visual modalities to overcome the domain shift in text-visual modalities under the test-time adaptation formulation. Specifically, DoP first takes text domain learning (TDL) to simulate various distribution shifts by learning diverse style tokens implicitly in the text space with inter-style diversity loss and intra-style consistency loss. Subsequently, DoP employs visual domain learning (VDL) to further refine the target-domain-relevant style token generated by TDL by making use of the test visual data with self-supervised learning. Extensive experiments on popular DG benchmark datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet show that our DoP consistently outperforms the state-of-the-art DG approaches. Codes are available at: https://github.com/shiming-chen/DoP.
Shiming Chen, Shuhuang Chen, Salman Khan +1
Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia +1
Adapting vision-language models to remote sensing imagery presents a fundamental challenge: both the visual and linguistic distributions of satellite data lie far outside natural image pretraining corpora. Despite this, prompting remains the dominant deployment paradigm, driven by the assumption that domain-specific language can guide frozen model representations toward specialized tasks. We test this assumption directly on a domain where the mismatch is prominent: cloud segmentation for satellite imagery. Using CLIPSeg on the CloudSEN12+ cloud segmentation benchmark, we evaluate 60 prompt variants spanning simple labels, domain terminology, appearance descriptors, and contextual cues, finding that every variant underperforms the zero-shot baseline (0.255 mIoU), with engineered prompts scoring as low as 0.07 mIoU. No amount of linguistic refinement bridges the gap between CLIP's natural image representations and satellite spectral imagery. In contrast, supervised fine-tuning with just 0.1% labeled data (~8 images) surpasses zero-shot performance overall, and 5-10% data recovers ~85% of maximum achievable mIoU. Full fine-tuning consistently outperforms low-rank adaptation by 0.03-0.09 mIoU, with the largest gaps for spectrally ambiguous classes, and at 0.5 to 1% labeled data, fine-tuning temporarily degrades performance on these classes before recovering, a supervision dip that aggregate mIoU can mask. For practitioners adapting vision-language models to specialized imagery, our results deliver a clear message: labeled data is not the expensive alternative to prompting; it is the worthwhile path.