Hierarchical Prompt Injector for Domain Generalization Segmentation

Xin Kun Lin; Ruoyu Guo; Jiaqi Guo; Maurice Pagnucco; Yang Song

cs.CVSep 5, 2026

Hierarchical Prompt Injector for Domain Generalization Segmentation

Authors: Xin Kun Lin, Ruoyu Guo, Jiaqi Guo, Maurice Pagnucco, Yang Song

Organizations: School of Computer Science and Engineering, University of New South Wales, Sydney, Australia

Abstract

Domain Generalized Semantic Segmentation (DGSS) is a challenging task, as vision models often rely on low-level appearance cues that change across domains. In contrast, structural attributes exhibit cross-domain stability, motivating the use of structural priors for DGSS. Existing methods use prompt learning to transfer such priors into DGSS models, but typically encode each class as a single holistic prompt. Moreover, these methods apply prompts uniformly to all pixels, offering no mechanism to adapt when only a subset of object regions is visible due to viewpoint changes, occlusion, and environmental variation. We address this with Spatial Hierarchical Prompts (SHP) that enrich each class with region-level geometric anchors capturing structural appearance from distinct viewing angles, ensuring complementary coverage under arbitrary viewpoints. Additionally, we propose the Hierarchical Prompt Injector (HPI), which enables spatially adaptive prompt injection in foundation models. HPI spatially grounds prompts by modeling their semantic relevance and spatial influence with visual features. Considering the difficulty of learning spatially and semantically aware prompt injection, we further introduce auxiliary supervision to align hierarchical prompts with their corresponding object regions. We achieve 70.62% and 72.74% mIoU on synthetic-to-real and real-to-real benchmarks, respectively. Code and checkpoints are released at https://github.com/MosukFate/HPI

Figures & tables

Figure 1: Class-level prompts (left) versus our Spatial Hierarchical Prompts (right). Class-level prompts encode each category as a single holistic sentence, producing diffuse attention that spreads across background regions. SHP decomposes each class into region-level prompts built from four complementary components: Region Cue , Part , Geometry , and Boundary Relation .

Figure 2: (a) Hierarchical Prompt Injector. Region-level prompts serve as queries in cross-attention, and the resulting residuals are modulated by SAC and SCC before injection. (b) Overall architecture. HPI augments the VLM backbone with SHP-guided residuals, while DINOv2 provides spatial features that are fused into the VLM stream for dense prediction. (c) Supervision losses. $\mathcal{L}_{\text{spatial}}$ aligns each prompt’s cross-attention map $\bar{\mathbf{A}}^{x}$ with ground-truth segmentation masks, supervising prompt-to-region correspondences. $\mathcal{L}_{\text{semantic}}$ supervises class-level logits $\hat{\mathbf{s}}$ against binary class-presence labels.

Methods	Backbone	Synthetic-to-real				Real-to-real
Methods	Backbone	G $\to$ C	G $\to$ B	G $\to$ M	Avg.	C $\to$ B	C $\to$ M	Avg.
SAN-SAW [ 37 ]	RN101	45.33	41.18	40.77	42.43	54.73	61.27	58.00
WildNet [ 29 ]	RN101	45.79	41.73	47.08	44.87	47.01	50.94	48.98
SHADE [ 63 ]	RN101	46.66	43.66	45.50	45.27	50.95	60.67	55.81
TLDR [ 28 ]	RN101	47.58	44.88	48.80	47.09	–	–	–
FAMix [ 19 ]	RN101	49.47	46.40	51.97	49.28	54.07	58.72	56.40

Table 1: Performance comparison (mIoU) under the synthetic-to-real (G

\to

{C, B, M}) and real-to-real (C

\to

{B, M}) settings.

\dagger

denotes methods that additionally employ a frozen DINOv2-L encoder. Best and second-best in bold and underlined .

Methods	Backbone	Synthetic-to-real
Methods	Backbone	S $\to$ C	S $\to$ B	S $\to$ M	Avg.
SAN-SAW [ 37 ]	RN101	40.87	35.98	37.26	38.04
TLDR [ 28 ]	RN101	42.60	35.46	37.46	38.51
IBAFormer [ 44 ]	MiT-B5	50.92	44.66	50.58	48.72
Rein [ 48 ]	DINOv2-L	48.59	44.42	48.64	47.22
SET [ 55 ]	DINOv2-L	49.65	45.45	49.45	48.18

Table 2: Performance comparison (mIoU) under the synthetic-to-real setting (S

\to

{C, B, M}).

\dagger

denotes methods that additionally employ a frozen DINOv2-L encoder. Best and second-best in bold and underlined .

Figure 3: Qualitative comparison on G $\to$ {C, B, M}. From left to right — input image, Rein [ 48 ] , MFuser [ 60 ] , HPI (Ours), and ground truth. Highlighted boxes indicate regions with partial occlusion or fine-grained structures where prior methods produce erroneous predictions.

Number of regional prompts ( $R$ )
Configuration	G $\to$ C	G $\to$ B	G $\to$ M	Avg.
$R{=}2$	72.68	62.26	73.49	69.48
$R{=}4$	73.01	62.58	73.48	69.69
Geometrical Cues
w/o Region Cue	72.33	62.24	73.26	69.28
w/o Part	72.49	62.40	73.30	69.40

Table 3: Spatial hierarchical prompt ablation study (mIoU) with DINOv2-L+CLIP-L.

Configuration	G $\to$ C	G $\to$ B	G $\to$ M	Avg.
w/o SAC	73.83	61.61	73.76	69.73
w/o SCC	73.17	62.63	74.03	69.94
Ours	74.05	63.14	74.67	70.62

Table 4: Adaptive injection ablation study (mIoU) with DINOv2-L+CLIP-L.

Figure 4: Cross-attention maps before (left) and after (middle) HPI injection, with the difference map (right). HPI concentrates attention on semantically meaningful object parts while suppressing background activations.

Configuration	CLIP-L				EVA02-L
Configuration	G $\to$ C	G $\to$ B	G $\to$ M	Avg.	G $\to$ C	G $\to$ B	G $\to$ M	Avg.
w/o HPI	71.57	61.29	72.98	68.61	71.63	63.48	72.08	69.06
w/o DINOv2	61.58	53.60	65.81	60.33	69.20	60.67	70.01	66.63
Freeze [ 48 ]	53.70	48.70	55.00	52.40	56.50	53.60	58.60	56.20
Rein [ 48 ]	57.10	54.70	60.50	57.40	65.30	60.50	64.90	63.60
SET [ 55 ]	58.20	55.30	61.40	58.30	66.40	61.80	65.60	64.60

Table 5: HPI component ablation study (mIoU). We additionally report CLIP/EVA-only baselines from Rein [ 48 ] for reference under the same G

\to

{C,B,M} setting.

Layer	G $\to$ C	G $\to$ B	G $\to$ M	Avg.	Layer	G $\to$ C	G $\to$ B	G $\to$ M	Avg.
5	72.29	62.25	73.43	69.32	17	72.79	62.72	73.75	69.75
9	72.66	62.43	73.49	69.53	21	73.15	62.98	73.59	69.91
13	72.59	62.78	73.36	69.58	Ours (23)	74.05	63.14	74.67	70.62

Table 6: Injection layer ablation study (mIoU) with DINOv2-L+CLIP-L.

Configuration	G $\to$ C	G $\to$ B	G $\to$ M	Avg.
w/o $\mathcal{L}_{\text{spatial}}$ + $\mathcal{L}_{\text{semantic}}$	72.51	62.69	73.01	69.40
w/o $\mathcal{L}_{\text{spatial}}$	72.99	62.07	73.56	69.54
w/o $\mathcal{L}_{\text{semantic}}$	73.10	62.87	73.76	69.91
Ours	74.05	63.14	74.67	70.62

Table 7: Supervision loss ablation study (mIoU) with DINOv2-L+CLIP-L.

Domain Generalization for Semantic Segmentation

See all 14 papers

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation
Apr 19, 2026Evren Çetinkaya, Sangmin Lee, Jung Uk Kim +2Domain Generalization Medical Image Segmentation
Technical University of Munich · Korea University · Kyung Hee University +1
Prompting Diffusion Models for Zero-Shot Instance Segmentation
Jun 21, 2026Irem Zeynep Alagöz, Nils Morbitzer, Andrea Ramazzina +3Interactive Image Segmentation Instance Segmentation
Technical University of Munich (TUM) · Munich Center of Machine Learning (MCML) · Mercedes-Benz AG +1
SegRAG: Retrieval Augmented Spatial Prompting for Open Vocabulary Semantic Segmentation
May 17, 2026Abderrahmene Boudiaf, Irfan Hussain, Sajid JavedImage Segmentation Promptable Image Segmentation
Khalifa University, United Arab Emirates.

Apr 19, 2026J/K move · Enter open · S save

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation

Evren Çetinkaya, Sangmin Lee, Jung Uk Kim, Hong Joo Lee+1

Technical University of Munich · Korea University · Kyung Hee University+1

Domain Generalization Medical Image Segmentation Visual Prompt Learning

Open paper AskPDF arXiv

Visual prompting has emerged as a powerful method for adapting pre-trained models to new domains without updating model parameters. However, existing prompting methods typically optimize a single prompt per domain and apply it uniformly to all inputs, limiting their ability to generalize under intra and inter-domain variability, which is especially critical in the medical field. To address this, we propose APEX, an Adaptive Prompt EXtraction framework that retrieves input-specific prompts from a learnable prompt memory. The memory stores diverse, domain-discriminative prompt representations and is queried via domain features extracted from the Fourier spectrum. To learn robust and discriminative domain features, we introduce a novel Low-Frequency Feature Contrastive (LFC) learning framework that clusters representations from the same domain while separating those from different domains. Extensive experiments on two medical segmentation tasks demonstrate that APEX significantly improves generalization across both seen and unseen domains. Furthermore, it complements any existing backbones and consistently enhances performance, confirming its effectiveness as a plug-and-play prompting solution in medical fields. The code is available at https://github.com/cetinkayaevren/apex/

Abstract

Domain Generalization for Semantic Segmentation

Explore similar work

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation

Prompting Diffusion Models for Zero-Shot Instance Segmentation

SegRAG: Retrieval Augmented Spatial Prompting for Open Vocabulary Semantic Segmentation