Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement

Hyun-Kurl Jang; Jihun Kim; Kuk-Jin Yoon

cs.CVSep 28, 2026

Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement

Authors: Hyun-Kurl Jang, Jihun Kim, Kuk-Jin Yoon

Organizations: Visual Intelligence Lab. KAIST

Abstract

Open-vocabulary semantic segmentation (OVSS) enables pixel-level prediction over arbitrary text-specified vocabularies and has shown strong generalization on common benchmarks. However, OVSS performance often degrades in specialized domains such as medical imaging, remote sensing, and industrial inspection, where dense pixel-level masks for adaptation are costly to obtain and require domain-specific expertise. We propose a preference-guided adaptation framework that replaces dense mask supervision with binary preferences. We observe that different prompt templates produce systematically different segmentations for the same image, a phenomenon we call prompt disagreement, and we repurpose it as a built-in source of preference supervision. Building on this, we mine localized preference queries from regions of high cross-template uncertainty, and adapt the OVSS model with Region-Localized Preference Optimization (RLPO) together with consistency regularization that stabilizes updates outside the queried region. Across extensive experiments on the MESS benchmark, the proposed method achieves consistent gains across diverse OVSS backbones without any pixel-level annotation, and remains effective under noisy preferences. Our code is available at https://github.com/blue-531/pref-ovss.

Figures & tables

Figure 1: Left: For each target-domain image, our framework runs multiple prompt templates through a OVSS backbone, mines a localized preference query from regions where the templates disagree most, and uses a binary judgment ("which prediction is closer to the intended concept on this region?") to drive an update of OVSS model. Right: Validation mIoU across diverse specialized domains [ 41 ] when only the prompt template is varied. Across all domains, performance varies widely across templates. Our method harnesses this variance as supervision and lifts performance above the entire per-domain spread, turning prompt disagreement into training signal.

Figure 2: Given a training image, our method generates $K$ prompt-conditioned predictions with model $f_{\theta}$ . It mines a preference query by selecting a high-uncertainty region $R$ from prompt-ensemble entropy and the most disagreeing template pair within $R$ . A binary preference identifies the winner and loser predictions. RLPO updates $f_{\theta}$ by increasing the winner’s score relative to the loser’s inside R, while the consistency regularization aligns the loser prediction with the winner pseudo-label.

VLM	Method	General	Earth Monit.	Medical Sci.	Engineering	Agri. & Biology	Mean
CLIP ViT-B/16	SAN-B	21.60	26.90	33.51	28.19	15.99	25.29
	+ Ours	22.09 $\pm$ 0.18	27.69 $\pm$ 0.60	52.97 $\pm$ 1.04	33.47 $\pm$ 0.21	30.36 $\pm$ 0.23	32.39 $\pm$ 0.09
	+ Dense-mask	24.74 $\pm$ 0.33	28.59 $\pm$ 0.44	44.37 $\pm$ 0.60	35.64 $\pm$ 0.72	31.48 $\pm$ 2.95	32.41 $\pm$ 0.46
	CAT-Seg-B	34.51	34.52	37.82	29.95	28.95	33.12
	+ Ours	36.06 $\pm$ 1.85	35.88 $\pm$ 0.60	52.66 $\pm$ 2.48	43.82 $\pm$ 2.16	31.04 $\pm$ 0.46	39.67 $\pm$ 0.08
	+ Dense-mask	38.69 $\pm$ 0.13	38.22 $\pm$ 0.35	60.08 $\pm$ 1.37	46.01 $\pm$ 1.37	41.61 $\pm$ 0.87	44.26 $\pm$ 0.49

Table 1: Quantitative evaluation on the MESS benchmark (mIoU, %). Numbers for our method are mean

\pm

standard deviation. Dense-mask rows use ground-truth mask supervision under the same streaming protocol, providing the dense-supervision counterpart.

Domain	Baseline	Strategies
Domain	Baseline	MC Dropout	TTA	Prompt
General	39.36	36.48	40.95	42.48
Earth	35.64	32.10	37.73	40.28
Medical	29.52	42.64	53.15	51.83
Engin.	34.22	47.66	50.68	52.68
Agri.& Bio	36.41	36.90	42.36	42.86

Table 2: Comparison of candidate generation strategies. We contrast prompt disagreement against MC Dropout and test-time augmentation, with matched candidate size (

K=14

Domain	Baseline	Strategies
Domain	Baseline	MC Dropout	TTA	Prompt
General	39.36	36.48	40.95	42.48
Earth	35.64	32.10	37.73	40.28
Medical	29.52	42.64	53.15	51.83
Engin.	34.22	47.66	50.68	52.68
Agri.& Bio	36.41	36.90	42.36	42.86

Table 2: Comparison of candidate generation strategies. We contrast prompt disagreement against MC Dropout and test-time augmentation, with matched candidate size (

K=14

Domain	Baseline	Ablation
Domain	Baseline	w/o $\mathcal{L}_{\mathrm{RLPO}}$	w/o $\mathcal{L}_{\mathrm{cons}}$	Ours
General	39.36	41.80	41.80	42.48
Earth	35.64	38.52	38.11	40.28
Medical	29.52	36.25	48.93	51.83
Engin.	34.22	44.83	47.88	52.68
Agri.& Bio	36.41	39.97	41.41	42.86

Table 3: Loss function ablation. We ablate the two components of our objective:

\mathcal{L}_{\text{RLPO}}

(preference loss within the queried region) and

\mathcal{L}_{\text{cons}}

(consistency regularizer outside)

Figure 3: Qualitative comparison on the MESS benchmark. Each row shows one sample, comparing the zero-shot baseline against our adapted prediction at both CLIP scales (ViT-B/16 and ViT-L/14). Top: SAN on CUB-200 (Agriculture & Biology). Bottom: CAT-Seg on iSAID (Earth Monitoring).

	Number of training images
Domain group	$0$	$4$	$8$	$16$	$32$	$64$	$128$
General	39.36	39.57 (+0.21)	40.52 (+1.16)	40.85 (+1.49)	41.96 (+2.60)	42.48 (+3.12)	42.56 (+3.20)
Earth Monitoring	35.64	38.22 (+2.58)	37.38 (+1.74)	39.50 (+3.86)	38.69 (+3.05)	40.28 (+4.64)	39.35 (+3.71)
Medical Sciences	29.52	41.03 (+11.51)	42.75 (+13.23)	46.30 (+16.78)	48.89 (+19.37)	51.83 (+22.31)	51.42 (+21.90)
Engineering	34.22	36.28 (+2.06)	39.19 (+4.97)	45.20 (+10.98)	50.57 (+16.35)	52.68 (+18.46)	54.13 (+19.91)
Agriculture & Biology	36.41	36.42 (+0.01)	38.12 (+1.71)	40.16 (+3.75)	41.36 (+4.95)	42.86 (+6.45)	41.93 (+5.52)

Table 4: Effect of training set size. We vary the number of target-domain training images per dataset from 0, corresponding to the zero-shot baseline, to 128. We report mIoU for each domain group and the gain over the zero-shot baseline.

Figure 4: Robustness to noisy preferences. We flip a fraction $p$ of preference labels and report the mIoU gain over the zero-shot baseline for each domain as well as the mean across datasets.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	General				Earth Monit.				Medical			Engineering				Agri. & Bio.
Subset	BDD100K	MHP v1	FoodSeg103	ATLANTIS	iSAID	WorldFloods	FloodNet	UAVid	Kvasir-Inst.	CHASE_DB1	PAXRay-4	Corrosion CS	DeepCrack	PST900	ZeroWaste-f	SUIM	CUB-200	CWFID	Mean
Baseline	48.23	30.77	32.92	45.51	19.67	39.94	41.05	41.90	65.49	3.32	19.75	7.47	25.27	78.73	25.42	49.75	21.89	37.58	35.26
Sentence-only	50.76	36.95	36.73	45.70	35.74	39.90	41.19	43.76	80.21	19.65	46.05	21.40	60.22	80.75	24.33	55.74	22.38	46.69	43.79
Scale-only	50.89	34.67	37.41	45.41	32.59	40.33	41.16	46.19	76.12	20.14	47.09	20.94	61.44	79.98	24.76	57.71	22.47	46.48	43.65
Full pool	50.58	36.41	37.41	45.50	36.17	41.13	41.07	42.74	87.41	20.74	47.34	22.90	74.39	80.96	32.47	58.49	22.22	47.87	45.88

Appendix

Table 5: Per-dataset adaptation performance for three template subsets that vary along different diversity axes. The best per dataset across the three subsets is highlighted in bold; the zero-shot baseline is shown for reference.

CAT-Seg [ 30 ]
optimizer	AdamW
weight decay	1e-4
learning rate	3e-3
$\beta$	0.1
$\lambda_{\text{cons}}$	0.1
$\tau_{\text{conf}}$	0.8

Appendix

Table 6: Detailed Implementation configuration.

Tier	IoU margin	#Judgments	Agreement
Easy	$\geq 0.18$	200	0.967
Medium	$[0.04,\ 0.18)$	200	0.953
Hard	$<0.04$	200	0.840
Overall	—	600	0.920

Appendix

Table 7: Agreement between human annotators and the preference oracle. Mined preference pairs are split into three difficulty tiers by the IoU margin between the two candidates, with boundaries set to the terciles of the margin distribution observed during actual adaptation runs. Each of the

20

participants judged

30

pairs (

10

per tier), for

600

judgments in total.

Condition	$\tau$	General	Earth Monit.	Medical Sci.	Engineering	Agri. & Biology	Mean
Zero-shot	—	39.36	35.64	29.52	34.22	36.41	35.26
Clean (oracle)	—	42.48	40.28	51.83	52.68	42.86	45.88
Random	0.05	42.39	39.83	51.60	49.38	43.74	45.13
	0.10	42.08	39.18	50.66	51.95	40.85	44.85
	0.15	42.21	37.52	50.57	48.68	42.93	44.12
Flip	0.05	42.74	39.25	51.30	51.78	41.62	45.21

Appendix

Table 8: Robustness to preference noise placed where humans actually disagree (CAT-Seg-L, mIoU %). Noise is injected only into near-tie queries whose IoU margin falls below

\tau

. Random replaces the oracle choice with a coin flip; Flip actively inverts it, an adversarial upper bound on annotator error.

Method	Annotation	General	Earth Monit.	Medical Sci.	Engineering	Agri. & Biology	Mean
CAT-Seg-L	—	39.36	35.64	29.52	34.22	36.41	35.26
+ Prompt ens.	—	39.53	37.66	25.11	34.97	33.78	34.74
+ Ours	binary preference / image	42.48 $\pm$ 0.35	40.28 $\pm$ 0.35	51.83 $\pm$ 0.81	52.68 $\pm$ 0.41	42.86 $\pm$ 1.45	45.88 $\pm$ 0.38
+ Weakly-sup. (point)	click / object	41.79 $\pm$ 0.62	34.20 $\pm$ 0.86	51.86 $\pm$ 2.70	44.15 $\pm$ 2.07	42.43 $\pm$ 0.97	42.41 $\pm$ 0.68
+ Prompt selection	dense mask / image	39.61 $\pm$ 0.29	40.15 $\pm$ 0.73	35.05 $\pm$ 0.10	39.13 $\pm$ 0.04	35.67 $\pm$ 0.26	38.20 $\pm$ 0.08
+ Dense-mask	dense mask / image	44.04 $\pm$ 0.41	41.82 $\pm$ 1.69	55.48 $\pm$ 1.98	49.18 $\pm$ 0.69	44.48 $\pm$ 0.78	46.67 $\pm$ 0.74

Appendix

Table 9: Comparison with alternative adaptation strategies on CAT-Seg-L (mIoU, %). Annotation denotes target-domain supervision as type / scope , with rows ordered by annotation cost. Grayed rows require stronger supervision than binary preferences and are included as references. Results are mean

\pm

standard deviation over three runs.

Param	Value	General	Earth Monit.	Medical Sci.	Engineering	Agri. & Bio.	Mean
$\beta$	0.05	42.50	39.95	52.02	50.27	43.01	45.33
	0.1	42.47	40.28	51.83	52.68	42.86	45.88
	0.2	41.77	40.31	51.61	51.84	43.24	45.57
$\lambda_{\text{cons}}$	0.05	41.60	40.21	50.98	52.06	42.42	45.31
	0.1	42.47	40.28	51.83	52.68	42.86	45.88
	0.2	41.90	39.83	51.96	50.41	41.80	44.99

Appendix

Table 10: Sensitivity to the hyperparameters of our method. Default values and the best per domain group within each hyperparameter group are shown in bold.

Metric	CAT-Seg-L	SAN-L
Full model parameters	433.7 M	436.7 M
Trainable parameters	71,681	71,681
Vision LoRA	65,536	65,536
Text residual adapter	6,145	6,145
Trainable fraction	0.017 $\,\%$	0.016 $\,\%$
Inference cost

Appendix

Table 11: Compute and memory profile of our adaptation on single NVIDIA H200 GPU.

#	Template
1	a photo of a {}.
2	This is a photo of a {}.
3	There is a {} in the scene.
4	There is the {} in the scene.
5	a photo of a {} in the scene.
6	a photo of a small {}.

Appendix

Table 12: The

K=14

prompt templates from the ViLD prompt pool used throughout this work.

Dataset	License	# classes	Classes
General Scenes
BDD100K [ 76 ]	custom	19	[road; sidewalk; building; wall; fence; pole; traffic light; …]
MHP v1 [ 75 ]	custom	19	[others; hat; hair; sunglasses; upper clothes; skirt; pants; …]
FoodSeg103 [ 74 ]	Apache 2.0	104	[background; candy; egg tart; french fries; chocolate; biscuit; …]
ATLANTIS [ 73 ]	Flickr (images)	56	[bicycle; boat; breakwater; bridge; building; bus; canal; …]
Earth Monitoring

Appendix

Table 13: Datasets in the MESS benchmark used in this work, grouped by domain.

		General				Earth Monit.				Medical			Engineering				Agri. & Bio.
Backbone	Method	BDD100K	MHP v1	FoodSeg103	ATLANTIS	iSAID	WorldFloods	FloodNet	UAVid	Kvasir-Inst.	CHASE_DB1	PAXRay-4	Corrosion CS	DeepCrack	PST900	ZeroWaste-f	SUIM	CUB-200	CWFID	Mean
SAN-B	base	35.36	9.39	8.40	33.25	4.18	30.15	33.95	39.31	62.38	18.47	19.69	4.53	49.27	40.69	18.25	36.54	5.79	5.63	25.29
	+ Ours	37.78	10.69	6.59	33.31	4.19	32.59	33.73	40.25	73.30	46.43	39.18	21.01	48.23	46.49	18.16	40.64	6.77	43.68	32.39
	+ Dense.	40.04	12.40	12.77	33.75	6.70	33.30	34.70	39.66	47.71	46.68	38.71	21.01	61.59	42.20	17.76	49.58	6.82	38.05	32.41
CAT-Seg-B	base	47.03	23.89	26.67	40.43	19.34	38.52	37.16	43.04	48.20	23.99	41.26	12.46	32.71	57.13	17.51	44.82	10.41	31.61	33.12
	+ Ours	48.12	29.00	27.78	39.35	24.70	36.60	37.91	44.29	84.05	23.16	50.77	22.33	61.49	73.79	17.65	47.08	9.43	36.60	39.67

Appendix

Table 14: Per-dataset results on the MESS benchmark across four OVSS backbones (mIoU, %). Within each backbone, the better of the zero-shot baseline and our method ( + Ours ) is highlighted in bold. ( + Dense. ) indicates the supervised reference.

	General				Earth Monit.				Medical			Engineering				Agri. & Bio.
Method	BDD100K	MHP v1	FoodSeg103	ATLANTIS	iSAID	WorldFloods	FloodNet	UAVid	Kvasir-Inst.	CHASE_DB1	PAXRay-4	Corrosion CS	DeepCrack	PST900	ZeroWaste-f	SUIM	CUB-200	CWFID	Mean
Baseline	48.23	30.77	32.92	45.51	19.67	39.94	41.05	41.90	65.49	3.32	19.75	7.47	25.27	78.73	25.42	49.75	21.89	37.58	35.26
w/o $\mathcal{L}_{\text{RLPO}}$	50.08	35.41	36.65	45.05	31.67	40.83	40.01	41.56	70.68	3.32	34.76	20.88	61.25	80.62	16.57	55.04	21.54	43.32	40.51
w/o $\mathcal{L}_{\text{cons}}$	50.36	34.48	36.79	45.56	25.45	41.29	42.07	43.64	84.86	20.52	41.40	23.78	59.33	80.22	28.18	54.90	21.15	48.17	43.45
Ours	50.58	36.41	37.41	45.50	36.17	41.13	41.07	42.74	87.41	20.74	47.34	22.90	74.39	80.96	32.47	58.49	22.22	47.87	45.88

Appendix

Table 15: Per-dataset loss ablation. The best performance is highlighted in bold.

	General				Earth Monit.				Medical			Engineering				Agri. & Bio.
Method	BDD100K	MHP v1	FoodSeg103	ATLANTIS	iSAID	WorldFloods	FloodNet	UAVid	Kvasir-Inst.	CHASE_DB1	PAXRay-4	Corrosion CS	DeepCrack	PST900	ZeroWaste-f	SUIM	CUB-200	CWFID	Mean
Baseline	48.23	30.77	32.92	45.51	19.67	39.94	41.05	41.90	65.49	3.32	19.75	7.47	25.27	78.73	25.42	49.75	21.89	37.58	35.26
MC Dropout	47.59	17.67	35.41	45.23	18.63	29.23	38.28	42.27	81.48	15.01	31.43	23.22	70.32	77.16	19.93	45.85	21.33	43.52	39.09
TTA	47.59	34.41	35.78	46.01	27.12	40.02	41.11	42.66	88.12	21.06	50.28	22.30	72.84	74.21	33.36	57.35	22.56	47.17	44.66
Ours	50.58	36.41	37.41	45.50	36.17	41.13	41.07	42.74	87.41	20.74	47.34	22.90	74.39	80.96	32.47	58.49	22.22	47.87	45.88

Appendix

Table 16: Per-dataset candidate generation comparison. With matched candidate size (

K=14

), we contrast prompt disagreement (Ours) against MC Dropout and test-time augmentation (TTA). The best performance is highlighted in bold.

	General				Earth Monit.				Medical			Engineering				Agri. & Bio.
# images	BDD100K	MHP v1	FoodSeg103	ATLANTIS	iSAID	WorldFloods	FloodNet	UAVid	Kvasir-Inst.	CHASE_DB1	PAXRay-4	Corrosion CS	DeepCrack	PST900	ZeroWaste-f	SUIM	CUB-200	CWFID	Mean
0	48.23	30.77	32.92	45.51	19.67	39.94	41.05	41.90	65.49	3.32	19.75	7.47	25.27	78.73	25.42	49.75	21.89	37.58	35.26
4	49.59	31.71	33.30	43.69	29.30	39.96	41.04	42.57	68.67	20.24	34.17	15.21	26.02	79.57	24.33	52.39	18.64	38.24	38.26
8	50.24	32.36	34.10	45.40	26.51	39.78	41.14	42.10	69.34	20.74	38.19	23.04	29.24	79.86	24.63	52.54	21.08	40.75	39.50
16	50.73	31.89	35.67	45.13	34.12	40.47	41.22	42.20	74.79	20.74	43.38	22.89	53.77	80.58	23.53	54.73	19.25	46.51	42.31
32	50.78	34.43	37.20	45.43	30.62	40.27	41.25	42.63	80.26	20.74	45.68	22.24	72.25	80.31	27.46	53.58	22.64	47.85	44.20

Appendix

Table 17: Per-dataset effect of training set size. The best performance is highlighted in bold.

Figure 5: Qualitative results on SAN-B. Each row shows one sample from one of the five MESS domain groups (top to bottom: General, Earth Monitoring, Medical Sciences, Engineering, Agriculture & Biology). Columns show the input image, ground-truth segmentation, zero-shot baseline prediction, and our adapted prediction.

Figure 6: Qualitative results on CAT-Seg-B. Each row shows one sample from one of the five MESS domain groups (top to bottom: General, Earth Monitoring, Medical Sciences, Engineering, Agriculture & Biology). Columns show the input image, ground-truth segmentation, zero-shot baseline prediction, and our adapted prediction.

Figure 7: Qualitative results on SAN-L. Each row shows one sample from one of the five MESS domain groups (top to bottom: General, Earth Monitoring, Medical Sciences, Engineering, Agriculture & Biology). Columns show the input image, ground-truth segmentation, zero-shot baseline prediction, and our adapted prediction.

Figure 8: Qualitative results on CAT-Seg-L. Each row shows one sample from one of the five MESS domain groups (top to bottom: General, Earth Monitoring, Medical Sciences, Engineering, Agriculture & Biology). Columns show the input image, ground-truth segmentation, zero-shot baseline prediction, and our adapted prediction.

Figure 9: Visualization of preference query mining. For each example (one per row), we show the input image, ground truth, winner prediction, loser prediction, and cross-template entropy ( $K=14$ ViLD templates). The selected query region $\mathcal{R}$ is overlaid on the entropy map as a bounding box.

Explore similar work

Test-Time Prototype Adaptation for Open-Vocabulary Semantic Segmentation
Aug 8, 2026Haozhe Wang, Jintao Cheng, Weibin Li +1Open-Vocabulary Segmentation Semantic Segmentation
South China Normal University · The Hong Kong University of Science and Technology
Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation
Aug 30, 2026Chandler Timm C. Doloriel, Yunbei Zhang, Sarthak Kumar Maharana +5Vision-Language Model Adaptation Open-Vocabulary Segmentation
Faculty of Science and Technology (REALTEK), Norwegian University of Life Sciences (NMBU) · Tulane University · University of Texas at Dallas
RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation
Jun 25, 2026Shanwen Wang, Xin Sun, Sirui Wang +1Remote Sensing Image Segmentation Semantic Segmentation
Faculty of Data Science, City University of Macau, 999078, SAR Macao, China · Chair of Data Science in Earth Observation, Technical University of Munich (TUM), Germany · Munich Center for Machine Learning, Munich, Germany

Aug 8, 2026, cs.CVJ/K move · Enter open · S save

Test-Time Prototype Adaptation for Open-Vocabulary Semantic Segmentation

Haozhe Wang, Jintao Cheng, Weibin Li, Xiaoyu Tang

South China Normal University · The Hong Kong University of Science and Technology

Open-Vocabulary Segmentation Semantic Segmentation Open-Vocabulary Recent Vision Foundation Models Test-Time Adaptation

Open paper AskPDF arXiv

Open-vocabulary semantic segmentation (OVSS) repurposes a pretrained CLIP encoder for dense prediction without additional labeled supervision. Existing methods improve CLIP's spatial behavior either by redesigning its internal attention or by injecting features from auxiliary vision foundation models; both require access to the host's internal computation and are tailored to its specific forward pass. In this work, we propose Test-time Prototype Adaptation (TPA), a training-free plug-in that operates at the output level, leaving the host's forward pass and weights unmodified. By leveraging a lightweight transductive adaptation phase, TPA identifies confident anchor patches from the host's own output predictions on a small pool of unlabeled deployment-domain images, and aggregates their frozen DINO features into per-class prototypes; at inference, a single cosine similarity lookup against this frozen bank provides an auxiliary score fused linearly with the host's logits. TPA composes with five representative OVSS hosts spanning attention-redesign and VFM-injection designs, across three CLIP backbones, eight benchmarks, and multiple internal VFM choices. Under a single set of hyper-parameters and without per-host tuning or parameter updates, TPA consistently improves segmentation accuracy, with as few as approximately 10% of unlabeled deployment-domain images sufficing for effective bank construction on most benchmarks.

Abstract

Explore similar work

Test-Time Prototype Adaptation for Open-Vocabulary Semantic Segmentation

Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation

RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation