Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation

Nikolay Kormushev; Josip Šarić; Matej Kristan

cs.CVMar 22, 2026

Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation

Authors: Nikolay Kormushev, Josip Šarić, Matej Kristan

Organizations: University of Ljubljana Faculty of Comp. and Inf. Science · ETH Zurich Dept. of Computer Science · University of Zagreb Faculty of Elec. Eng. and Computing

Abstract

Open-vocabulary panoptic segmentation remains hindered by two coupled issues: (i) mask selection bias, where objectness heads trained on closed vocabularies suppress masks of categories not observed in training, and (ii) limited regional understanding in vision-language models such as CLIP, which were optimized for global image classification rather than localized segmentation. We introduce OVRCOAT, a simple, modular framework that tackles both. First, a CLIP-conditioned objectness adjustment (COAT) updates background/foreground probabilities, preserving high-quality masks for out-of-vocabulary objects. Second, an open-vocabulary mask-to-text refinement (OVR) strengthens CLIP's region-level alignment to improve classification of both seen and unseen classes with markedly lower memory cost than prior fine-tuning schemes. The two components combine to jointly improve objectness estimation and mask recognition, yielding consistent panoptic gains. Despite its simplicity, OVRCOAT sets a new state of the art on ADE20K (+5.5% PQ) and delivers clear gains on Mapillary Vistas and Cityscapes (+7.1% and +3% PQ, respectively). The code is available at: https://github.com/nickormushev/OVRCOAT

Figures & tables

Figure 1: State-of-the-art (e.g., MAFT+ [ 14 ] , FC-CLIP [ 48 ] ) incorrectly discard detections on categories not present at training time (eg., paintings), leading to failed detection. OVRCOAT correctly classifies these instances by leveraging a CLIP-based update to background probabilities, improving performance on unseen classes.

Figure 2: OVRCOAT extracts mask-to-text aligned features by CLIP image encoder and mask transformer, adjusted by a novel open-vocabulary mask-to-text refinement finetuning protocol (OVR). A new CLIP-conditioned objectness adjustment (COAT) module reduces the objectness bias in the mask transformer, improving detection of vocabulary instances not observed in training. Our main contributed blocks are denoted by red.

Figure 3: OVR optimizes a combined objective: a mask proposal accuracy $\mathcal{L}_{\mathrm{M2F}}$ and mask classification $\mathcal{L}_{\mathrm{cls}}$ for aligning per-pixel and text embeddings, while preserving vocabulary generalization.

Method	ADE20k [ 52 ]			Mapillary Vistas [ 26 ]			Cityscapes [ 7 ]			COCO [ 21 ]
	PQ	SQ	RQ	PQ	SQ	RQ	PQ	SQ	RQ	PQ	SQ	RQ
MaskCLIP [ 8 ]	15.1	70.5	19.2	-	-	-	-	-	-	-	-	-
FreeSeg [ 27 ]	16.3	71.8	21.6	-	-	-	-	-	-	-	-	-
OPSNet [ 3 ]	19.0	52.4	23.0	-	-	-	41.5	67.5	50.0	52.4	83.5	62.1
ODISE [ 45 ]	23.4	78.1	28.3	14.2	61.0	17.2	23.9	75.3	29.0	55.4	-	-
FC-CLIP [ 48 ]	26.8	71.2	32.3	18.3	56.0	23.1	44.0	75.4	53.6	54.4	-	-

Table 1: Analysis on standard datasets. Gray indicates a test set with same vocabulary as used in training.

Figure 4: Per-class PQ differences between OVRCOAT and FC-CLIP [ 48 ] on ADE20k [ 52 ] for classes with largest PQ differences. Mean values are reported weighted with ground truth class frequency (weighted), and treating all classes equally (unweighted). OVRCOAT yields substantial improvements on unseen categories, demonstrating stronger generalisation, with stable performance on seen classes.

COAT	OVR	ADE20k	Mapillary	Cityscapes	COCO
✗	✗	26.8	18.3	44.0	54.4
✓	✗	27.6	18.8	44.6	53.7
✗	✓	27.6	19.2	44.5	55.5
✓	✓	28.6	19.6	45.3	54.6

Table 2: Ablation study of OVRCOAT individual modules.

Model	Backbone	A-847	A-150	PC-459	PC-59	PAS-20
OpenSeg [ 13 ]	ALIGN	8.8	28.6	12.2	48.2	72.2
OVSeg [ 20 ]	ViT-L	9.0	29.6	12.4	55.7	94.5
SAN [ 46 ]	ViT-L	12.4	32.1	15.7	57.7	94.6
ODISE [ 45 ]	ViT-L	11.1	29.9	14.5	57.3	-
FC-CLIP* [ 48 ]	ConvNeXt-L	14.8	34.0	18.2	58.4	95.4
MAFT+ sem [ 14 ]	ConvNeXt-L	15.1	36.1	21.6	59.4	96.5

Table 3: Open-vocabulary semantic segmentation performance (mIoU) across multiple datasets, with best performance bolded.

Mask Classifier	COAT	PQ	SQ	RQ
CLIP OVR	✗	24.2	75.9	29.2
CLIP OVR	✓	24.9	78.4	30.2
Ensemble	✗	27.6	74.1	33.4
Ensemble	✓	28.6	77.3	34.7

Table 4: Comparison of various mask classification methods.

Model	PQ	PQ seen	PQ unseen
CLIP	41.8	53.3	33.3
CLIP OVR	46.4	58.2	37.6

Table 5: Evaluation of OVR using oracle (ground truth) masks.

Method	PQ	SQ	RQ
OVR B,U	27.2	72.4	33.1
OVR RC,U	27.7	73.0	33.7
OVR G,U	27.7	71.3	33.6
OVR G,F	27.6	73.5	33.4
OVR RC,F	27.7	71.2	33.6
OVR	27.6	73.6	33.4

Table 6: Comparison of finetuning strategies on ADE20K [ 52 ] .

Model	PQ (ADE20K)	Memory [GB/image]
MAFT+	27.1	27.0
OVRCOAT	28.6	12.5

Table 7: Average training-time memory consumption.

Figure 5: OVRCOAT performance on ADE20K [ 52 ] remains stable over a range of trust factor

\gamma

values.

Figure 6: Qualitative analysis. OVRCOAT consistently recovers more object masks, including regions that MAFT+ [ 14 ] and FC-CLIP [ 48 ] misclassify as background, e.g., playthings in the bedroom and bottles in the bar. It also produces more accurate segmentations.

Explore similar work

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation
Aug 12, 2026Duy Tran Thanh, Yeejin Lee, Byeongkeun KangPanoptic Segmentation Instance Segmentation
Department of Electronic Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul, 01811, South Korea · Department of Electrical and Information Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul, 01811, South Korea · School of Electrical and Electronics Engineering, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, South Korea
DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction
Mar 14, 2026Jing Wang, Huimin Shi, Quan Zhou +3Open-Vocabulary Semantic Segmentation Text-Guided Image Segmentation
National Engineering Research Center of Communications and Networking, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China. · Institute for Advanced Ocean Research (Nantong), Southeast University, Nantong 226334, P. R. China. · Department of Internet of Things, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China +1
Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
Feb 26, 2026Tilemachos Aravanis, Vladan Stojnić, Bill Psomas +2Open-Vocabulary Semantic Segmentation Test-Time Adaptation of VLMs
VRG, FEE, Czech Technical University in Prague · University of Crete · Archimedes, Athena RC +1

Aug 12, 2026J/K move · Enter open · S save

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation

Duy Tran Thanh, Yeejin Lee, Byeongkeun Kang

Department of Electronic Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul, 01811, South Korea · Department of Electrical and Information Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul, 01811, South Korea · School of Electrical and Electronics Engineering, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, South Korea

Panoptic Segmentation Instance Segmentation Pseudo-Labeling

Open paper AskPDF arXiv

This work addresses the challenge of open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), which aim to recognize both predefined and unseen object categories without exhaustive human annotations. Existing methods often suffer from noisy pseudo-masks, limited visual-textual grounding, and difficulty handling synonyms or out-of-vocabulary (OOV) words. To overcome these challenges, we propose a multimodal framework that leverages pre-trained vision-language models for automatic pseudo-label generation, CLIP-guided synonym filtering, and GPT-based caption reconstruction. In our target-vocabulary-assisted pseudo-labeling setting, the framework first constructs pseudo segmentation masks, descriptive captions, and semantically aligned synonym sets using Grounded SAM, LLaVA, and CLIP, providing multimodal supervision without manual annotation. We then enhance visual-textual alignment through three complementary training objectives: an extended grounding loss that incorporates visually grounded synonyms, a semantic consistency loss, and a generative caption reconstruction loss. Extensive experiments on the COCO dataset demonstrate that the proposed method consistently outperforms previous state-of-the-art approaches under this protocol, achieving substantial improvements on both OVIS and OSPS benchmarks.

Abstract

Explore similar work

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation

DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction

Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?