PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation

Jianjian Yin; Tao Chen; Yi Chen; Gensheng Pei; Xiangbo Shu; Yazhou Yao; Fumin Shen

cs.CVMar 18, 2026

PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation

Authors: Jianjian Yin, Tao Chen, Yi Chen, Gensheng Pei, Xiangbo Shu, Yazhou Yao, Fumin Shen

Organizations: Nanjing University of Science and Technology · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery · Nanjing Normal University · Department of Electrical and Computer Engineering, Sungkyunkwan University ·

Abstract

Recent advances in vision-language models (VLMs) have garnered substantial attention in open-vocabulary semantic and part segmentation (OSPS). However, existing methods extract image-text alignment cues from cost volumes through a serial structure of spatial and class aggregations, leading to knowledge interference between class-level semantics and spatial context. Therefore, this paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the above challenge, enabling the model to capture richer vision-language alignment information from cost volumes. Specifically, we design an expert-driven perceptual learning (EPL) module that efficiently integrates semantic and contextual streams. It incorporates a multi-expert parser to extract complementary features from multiple perspectives. In addition, a coefficient mapper is designed to adaptively learn pixel-specific weights for each feature, enabling the integration of complementary knowledge into a unified and robust feature embedding. Furthermore, we propose a feature orthogonalization decoupling (FOD) strategy to mitigate redundancy between the semantic and contextual streams, which allows the EPL module to learn diverse knowledge from orthogonalized features. Extensive experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.

Figures & tables

Figure 1: Motivation of PCA-Seg. (a) Cost aggregation comparison among different methods for class runway . (b) The prevailing serial architecture. (c) Our proposed parallel cost aggregation paradigm. The FOD strategy decouples class-level semantics and spatial context, enabling the EPL module to parse diverse knowledge from the resulting orthogonalized features.

Figure 2: Comparison with other state-of-the-art methods on different benchmarks. “Ppart” and “Apart” are abbreviations for the Pascal-Part-116 [ 52 ] and ADE20K-Part-234 [ 52 ] datasets, respectively. ‘P’ and ‘O’ denote two different settings, namely Pred-All and Oracle-Obj. ‘H’ represents the harmonic IoU over both seen and unseen classes, while ‘U’ denotes the mIoU of unseen classes.

Figure 3: The framework of PCA-Seg on OVSS. Image and text features from CLIP’s visual and text encoders are combined via the Hadamard product to construct the cost volume $\mathcal{S}$ . This volume is then processed simultaneously through spatial and class aggregation to produce the spatial context feature $\mathcal{B}_{n}$ and the class-level semantic representation $\mathcal{E}_{n}$ . The multi-expert (ME-) parser extracts diverse and complementary knowledge from these two streams across multiple perspectives, while the coefficient (Co-) mapper adaptively learns weights to integrate the features parsed by the experts. $\mathcal{B}_{n}$ and $\mathcal{E}_{n}$ are decoupled using the feature orthogonalization decoupling (FOD) strategy to reduce redundancy, thereby providing enriched representations for expert-driven perceptual learning (EPL).

Method	Publication	Backbone	Training Dataset	A-847	PC-459	A-150	PC-59	PAS-20	PAS-20 b
LSeg+ [ 18 ]	ECCV’22	ALIGN	COCO-Stuff	2.5	5.2	13.0	36.0	-	59.0
ZegFormer [ 16 ]	CVPR’22	ViT-B/16	COCO-Stuff-156	4.9	9.1	16.9	42.8	86.2	62.7
ZSseg [ 59 ]	ECCV’22	ViT-B/16	COCO-Stuff	7.0	-	20.5	47.7	88.4	-
OVSeg [ 29 ]	CVPR’23	ViT-B/16	COCO-Stuff + Caption	7.1	11.0	24.8	53.3	92.6	-
GKC [ 20 ]	ICCV’23	ViT-B/16	COCO Panoptic	3.5	7.1	18.8	45.2	83.2	-
EBSeg [ 44 ]	CVPR’24	ViT-B/16	COCO-Stuff	11.1	17.3	30.0	56.7	94.6	-

Table 1: Comparison of performance with other state-of-the-art methods on multiple standard benchmarks in open-vocabulary semantic segmentation. See

\lx@sectionsign\ref{Compare_method}

for details.

			Pascal-Part-116						ADE20K-Part-234
			Pred-All			Oracle-Obj			Pred-All			Oracle-Obj
Method	Publication	Backbone	Seen	Unseen	h-IoU	Seen	Unseen	h-IoU	Seen	Unseen	h-IoU	Seen	Unseen	h-IoU
ZSSeg+ [ 59 ]	ECCV’22	ResNet-50	38.1	3.4	6.2	54.4	19.0	28.2	32.3	0.9	1.7	43.2	27.8	33.9
CLIPSeg [ 33 , 52 ]	CVPR’22	ViT-B/16	27.8	13.3	18.0	48.9	27.5	35.2	3.1	0.6	0.9	38.2	30.9	34.2
VLPart [ 48 ]	ICCV’23	ResNet-50	35.2	9.0	14.4	42.6	18.7	26.0	-	-	-	-	-	-
CAT-Seg [ 12 , 52 ]	CVPR’24	ViT-B/16	36.8	23.4	28.6	43.8	27.7	33.9	7.0	2.4	3.5	33.8	25.9	29.3

Table 2: Comparison of zero-shot performance with other state-of-the-art methods on the Pascal-Part-116 and ADE20K-Part-234 datasets in open-vocabulary part segmentation. See

\lx@sectionsign\ref{Compare_method}

for details.

Panel of Figure 4 — Figure 4: Comparison of qualitative results with serial cost aggregation-based state-of-the-art methods at different granularities. (a)–(d) illustrate open-vocabulary semantic segmentation on the PC-59 dataset, while (e)–(h) present open-vocabulary part segmentation on the Pascal-Part-116 benchmark. See $\lx@sectionsign\ref{Compare_method}$ for details.

Setting	A-847	PC-459	A-150	PC-59	PAS-20 b
W/o Agg.	14.1	20.7	34.4	59.3	79.1
Serial Agg.	15.3	21.4	36.3	60.6	81.3
Parallel Agg.	16.1	22.3	38.1	62.2	82.7

Table 3: Comparative analysis of serial aggregation and the proposed parallel aggregation. See

\lx@sectionsign\ref{abla_studies}

for details.

Setting	Param(M)	FLOPs(G)	Memory(G)	Time(s)
Entire Network	70.3	2121.32	26.4	0.89
Serial Block	0.68	53.67	0.60	0.11
Parallel Block	1.03	88.07	1.56	0.19

Table 4: Complexity analysis of each serial and parallel block under the same experimental setup. See

\lx@sectionsign\ref{abla_studies}

for details.

Table 5: A set of ablative studies on three datasets. (a) and (b) provide analyses of integrating the proposed components into the parallel and existing serial architectures, respectively. (c) and (d) show the impact of the number of expert blocks in the multi-expert parser and the weight

\lambda

of the FOD loss on model performance, while (e) presents the performance analysis of the integration strategy for the multiple knowledge parsed by the multi-expert (ME) parser. See

\lx@sectionsign\ref{abla_studies}

for details.

Figure 6: Knowledge coupling analysis between class-level semantics and spatial context under different designs. SCA/PCA denote serial/parallel cost aggregation. See

\lx@sectionsign\ref{abla_studies}

for details.

Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation labels image pixels using text-defined categories beyond a fixed training vocabulary.

See all 87 papers

Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
May 15, 2026Chenhao Wang, Yingrui Ji, Yu Meng +1Image Segmentation Open-Vocabulary Semantic Segmentation
Aerospace Information Research Institute, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhejiang University
DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction
Mar 14, 2026Jing Wang, Huimin Shi, Quan Zhou +3Open-Vocabulary Semantic Segmentation Text-Guided Image Segmentation
National Engineering Research Center of Communications and Networking, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China. · Institute for Advanced Ocean Research (Nantong), Southeast University, Nantong 226334, P. R. China. · Department of Internet of Things, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China +1
PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation
Mar 23, 2026Gensheng Pei, Xiruo Jiang, Xinhao Cai +3Graph-Based Label Propagation Open-Vocabulary Semantic Segmentation
Department of Electrical and Computer Engineering, Sungkyunkwan University · School of Computing and Artificial Intelligence, Southwest Jiaotong University · School of Computer Science and Engineering, Nanjing University of Science and Technology

May 15, 2026J/K move · Enter open · S save

Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation

Chenhao Wang, Yingrui Ji, Yu Meng, Yao Zhu

Aerospace Information Research Institute, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhejiang University

Image Segmentation Open-Vocabulary Semantic Segmentation Vision-Language Alignment Semantic Segmentation Compositional Generalization Fine-Grained Semantic Segmentation

Open paper AskPDF arXiv

Open-vocabulary segmentation models often struggle to generalize to unseen combinations of object categories and attributes, because fine-grained descriptions are typically encoded as holistic sentences that entangle multiple semantic units. We propose a Decomposed Vision-Language Alignment framework that explicitly factorizes textual prompts into a concept token and multiple attribute tokens, enabling separate cross-modal interactions for each semantic unit. At the feature level, we introduce a Feature-Gated Cross-Attention module that generates attribute-specific gating maps to fuse information in a multiplicative manner, effectively enforcing compositional semantics. At the scoring level, per-token similarities are aggregated in log-space, producing a stable and interpretable compositional matching. The method can be seamlessly integrated into existing transformer-based segmentation architectures and significantly improves generalization to unseen attribute-category compositions in fine-grained open-vocabulary segmentation benchmarks.

Abstract

Open-Vocabulary Semantic Segmentation

Explore similar work

Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation

DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction

PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation