PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation

Gensheng Pei; Xiruo Jiang; Xinhao Cai; Tao Chen; Yazhou Yao; Byeungwoo Jeon

cs.CVMar 23, 2026

PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation

Authors: Gensheng Pei, Xiruo Jiang, Xinhao Cai, Tao Chen, Yazhou Yao, Byeungwoo Jeon

Organizations: Department of Electrical and Computer Engineering, Sungkyunkwan University · School of Computing and Artificial Intelligence, Southwest Jiaotong University · School of Computer Science and Engineering, Nanjing University of Science and Technology

Abstract

Training-free open-vocabulary semantic segmentation (OVSS) promises rapid adaptation to new label sets without retraining. Yet, many methods rely on heavy post-processing or handle text and vision in isolation, leaving cross-modal geometry underutilized. Others introduce auxiliary vision backbones or multi-model pipelines, which increase complexity and latency while compromising design simplicity. We present PEARL, Procrustes alignment with text-aware Laplacian propagation, a compact two-step inference that follows an align-then-propagate principle. The Procrustes alignment step performs an orthogonal projection inside the last self-attention block, rotating keys toward the query subspace via a stable polar iteration. The text-aware Laplacian propagation then refines per-pixel logits on a small grid through a confidence-weighted, text-guided graph solve: text provides both a data-trust signal and neighbor gating, while image gradients preserve boundaries. In this work, our method is fully training-free, plug-and-play, and uses only fixed constants, adding minimal latency with a small per-head projection and a few conjugate-gradient steps. Our approach, PEARL, sets a new state-of-the-art in training-free OVSS without extra data or auxiliary backbones across standard benchmarks, achieving superior performance under both with-background and without-background protocols.

Figures & tables

Figure 1: Attention visualization at selected query points. We show an input resized to $512\times 512$ using the CLIP ViT-B/16 vision encoder. For each colored query, we compare the attention maps produced by CLIP [ 51 ] , NACLIP [ 26 ] , and our PEARL. CLIP exhibits diffuse and background-biased responses, while NACLIP improves localization but often fragments objects into isolated peaks. By performing Procrustes alignment in the last self-attention block, PEARL yields compact and object-consistent attention, suppresses background spill, and better preserves thin parts, resulting in stable focus with reduced grid artifacts.

Figure 2: Our PEARL framework with align-then-propagate . Given an image $\bm{I}$ and a label set $\mathcal{Y}$ , a frozen ViT vision encoder yields patch tokens. Procrustes alignment is inserted in the last self-attention block: for each head ( $J$ heads), queries and keys are weighted-centered and the keys are orthogonally rotated toward the query subspace, producing geometry-corrected patch features. A frozen text encoder maps $\mathcal{Y}$ to unit-norm prototypes $\bm{T}$ . Patch-text cosine similarity gives initial logits $\widetilde{\bm{Z}}$ . Text-aware Laplacian propagation then refines $\widetilde{\bm{Z}}$ on a compact grid using a class graph $\bm{G}$ induced by $\bm{T}$ and confidence weights, and the refined scores are upsampled to full resolution $\bm{F}$ . The final segmentation $\bm{S}$ is obtained by selecting, at each pixel, the class with the highest score.

Method	Pub. & Year	Extra data	Extra backbone	Training free	with background			without background					Avg.
Method	Pub. & Year	Extra data	Extra backbone	Training free	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
(a) Trained + Extra data
GroupViT [ 69 ]	CVPR’22	CC12M+RedCaps	✗	✗	50.4	18.7	27.5	79.7	23.4	15.3	11.1	9.2	29.4
TCL [ 10 ]	CVPR’23	CC3M+CC12M	✗	✗	51.2	24.3	30.4	77.5	30.3	19.6	23.1	14.9	33.9
CoDe [ 64 ]	CVPR’24	CC3M+RedCaps	✗	✗	57.7	30.5	32.3	-	-	23.9	28.9	17.7	-
Trained + Extra data & backkbone

Table 1: Quantitative results of open-vocabulary semantic segmentation. “Extra data” denotes external datasets ( e.g . , CC3M [ 55 ] , CC12M [ 11 ] , RedCaps [ 20 ] , COCO Captions [ 14 , 40 ] , and ImageNet-1K [ 19 ] ), and “Extra backbone” lists auxiliary models. “Training-free” indicates no extra training. All metrics are mIoU (%). Best results are highlighted with bold , and second best with underlined .

Method	Pub. & Year	Extra backbone	with background			without background					Avg.
Method	Pub. & Year	Extra backbone	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
Training-free with extra backbone
LaVG [ 30 ]	ECCV’24	DINOv1 (ViT-B/8)	89.3	48.7	74.8	91.1	58.9	39.1	68.5	37.0	63.4
ProxyCLIP ∗ [ 35 ]	ECCV’24	DINOv2 † (ViT-B/14)	86.6	52.0	75.9	88.4	63.4	43.4	74.9	49.1	66.7
CASS ∗ [ 32 ]	CVPR’25	DINOv2 † (ViT-B/14)	85.6	50.9	70.9	92.2	60.1	41.1	72.5	41.0	64.3
CASS ∗ [ 32 ]	CVPR’25	DINOv3 (ViT-B/16)	88.1	53.6	75.5	93.8	62.3	42.0	75.2	45.6	67.0

Table 2: Quantitative results of open-vocabulary semantic segmentation using average pixel accuracy (pAcc, %) . Rows are grouped by whether an extra vision backbone is used. Best results are highlighted with bold , and second best with underlined .

Figure 3: Qualitative results of open-vocabulary semantic segmentation. Results are shown on the Pascal VOC [ 23 ] , Pascal Context [ 44 ] , MS-COCO [ 40 ] , Cityscapes [ 54 ] , and ADE20K [ 79 ] datasets ( DS ), comparing PEARL ( ours ) with NACLIP [ 26 ] and SFP [ 28 ] . All methods use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Component		with BG			without BG					Avg.
PA	TLP	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
✗	✗	18.6	7.8	6.5	49.1	11.2	7.2	6.7	3.2	13.8
✓	✗	59.2	33.0	34.3	85.0	35.3	24.5	35.0	17.9	40.6
✗	✓	35.4	22.4	23.4	79.3	25.0	16.9	20.5	11.7	29.3
✓	✓	64.1	35.1	37.3	86.9	38.6	26.3	37.6	19.4	43.2

Table 3: Ablation analysis of key components, i.e . , PA ( cf . § 3.2 ) and TLP ( cf . § 3.3 ). “BG” means the background class.

Method	with BG			without BG					Avg.
Method	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
SCLIP [ 62 ]	59.1	30.4	30.5	80.4	34.1	22.4	32.2	16.1	38.2
+ TLP	63.4	34.5	36.2	85.7	37.5	25.7	35.9	18.7	42.2
NACLIP [ 26 ]	58.9	32.2	33.2	79.7	35.2	23.3	35.5	17.4	39.4
+ TLP	63.3	34.7	36.2	83.9	38.1	25.7	37.9	18.9	42.3
SFP [ 28 ]	56.8	32.3	32.1	83.4	36.0	24.0	34.1	18.1	39.6

Table 4: Ablation analysis of plug-and-play TLP ( cf . § 3.3 ).

Size	with BG			without BG					Avg.
Size	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
224px	59.1	33.1	36.1	86.1	36.4	24.9	29.5	17.7	40.4
280px	61.1	34.4	37.4	86.8	37.8	26.0	32.6	19.2	41.9
336px	64.1	35.1	37.3	86.9	38.6	26.3	33.9	19.4	42.7

Table 5: Ablation analysis of using different input image sizes.

Method	CLIP	with BG	without BG			Avg.
Method	CLIP	V21	PC59	Stuff	ADE	Avg.
SCLIP [ 62 ]	ViT-B/32	50.6	28.7	20.0	14.8	28.5
NACLIP [ 26 ]		51.1	32.4	21.2	14.9	29.9
SFP ∗ [ 28 ]		50.1	31.9	21.3	15.7	29.8
PEARL		54.5	34.7	23.3	16.6	32.3
SCLIP [ 62 ]	ViT-L/14	44.4	25.2	17.6	10.9	24.5

Table 6: Ablation analysis of using different CLIP backbones.

Panel of Figure 4 — Figure 4: Ablation analysis of (a) grid size and (b) efficiency.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Config	$\tau_{s}$	$\beta$	$\epsilon$	$\kappa$	$\lambda$	$\tau$
Value	0.5	10	$10^{-6}$	5	1	1

Appendix

Table A7 : Fixed hyperparameter setting used for all datasets.

Solver	with BG			without BG					Avg.
Solver	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
SVD	64.2	35.2	37.2	86.7	38.6	26.3	37.9	19.2	43.2
N-S	64.1	35.1	37.3	86.9	38.6	26.3	37.6	19.4	43.2

Appendix

Table B8 : Ablation analysis of different alignment solvers ( cf . § 3.2 ). “N-S” denotes the Newton-Schulz iterative algorithm.

Latency	with BG			without BG					Avg.
Latency	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
SVD	60.9	199.7	239.3	59.6	198.8	212.1	975.9	192.3	267.3
N-S	48.7	111.7	149.4	47.1	111.2	120.7	498.5	115.4	150.3

Appendix

Table B9 : Inference latency (ms/img) comparison of alignment solvers. “N-S” denotes the Newton-Schulz iterative algorithm.

Figure B5 : Diagnostics of Procrustes Alignment on V21 [ 23 ] . (a) PCA projection of centered queries and keys before and after PA. (b) Distribution of the per-(image, head) rotation magnitude $\|\bm{R}^{\star}-\bm{I}_{d}\|_{F}$ . (c) Correlation between alignment-error reduction $\Delta err$ and mIoU gain $\Delta\mathrm{mIoU}$ . (d) Component ablation of centering ( C ), whitening ( W ), rotation ( R ), full PA without weights ( F0 ), and full weighted PA ( FW ). (e) Stability of the Newton-Schulz iterations used in PA. (f) Stability of the conjugate-gradient iterations used in TLP.

key-key	with BG			without BG					Avg.
key-key	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
w/o	63.2	34.7	36.8	86.4	38.3	26.1	34.6	18.7	42.4
w/	64.1	35.1	37.3	86.9	38.6	26.3	37.6	19.4	43.2

Appendix

Table B10 : Ablation analysis of a key-key self-correlation term.

Iteration	Precision		Efficiency		PES
Iteration	mIoU	pAcc	Latency (ms/img)	Memory (GB)	PES
5	61.5	87.4	31.7	1.32	0.50
15	63.2	88.2	40.5	1.32	0.73
25	64.1	88.5	48.7	1.32	0.80
35	64.4	88.7	58.5	1.32	0.79
45	64.6	88.7	66.5	1.32	0.75

Appendix

Table B11 : Ablation analysis of CG iterations on V21 [ 23 ] ( cf . § 3.3 . “PES” denotes the precision-efficiency score.

Method	Pub. & Year	Extra data	Extra backbone	Training free	with background			without background					Avg.
Method	Pub. & Year	Extra data	Extra backbone	Training free	V21	PC60	Object	V20	PC59	Stuff	City	ADE	Avg.
w/ Mask Refinement
GroupViT [ 69 ]	CVPR’22	CC12M+RedCaps	✗	✗	51.1	19.0	27.9	81.5	23.8	15.4	11.6	9.4	30.0
TCL [ 10 ]	CVPR’23	CC3M+CC12M	✗	✗	55.0	30.4	31.6	83.2	33.9	22.4	24.0	17.1	37.2
CLIP - DINOiser [ 66 ]	ECCV’24	ImageNet-1K	DINOv1 (ViT-B/16)	✗	64.6	33.5	36.1	81.5	37.1	25.3	31.5	20.6	41.3
ReCo [ 57 ]	NeurIPS’22	ImageNet-1K	✗	✓	27.2	21.9	17.3	62.4	24.7	16.3	22.8	12.4	25.6

Appendix

Table B12 : Quantitative results of open-vocabulary semantic segmentation. “Extra data” denotes external datasets ( e.g . , CC3M [ 55 ] , CC12M [ 11 ] , RedCaps [ 20 ] , COCO Captions [ 14 , 40 ] , and ImageNet-1K [ 19 ] ), and “Extra backbone” lists auxiliary models. “Training-free” indicates no extra training. We evaluate prior methods using their default post-processing (official or re-implemented), while reporting PEARL without any post-processing. All metrics are mIoU (%). Best results are highlighted with bold , and second best with underlined .

Figure D6 : Qualitative results of open-vocabulary semantic segmentation. Results are shown on the V21 [ 23 ] dataset. Our PEARL use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Figure D7 : Qualitative results of open-vocabulary semantic segmentation. Results are shown on the PC60 [ 44 ] dataset. Our PEARL use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Figure D8 : Qualitative results of open-vocabulary semantic segmentation. Results are shown on the Object [ 40 ] dataset. Our PEARL use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Figure D9 : Qualitative results of open-vocabulary semantic segmentation. Results are shown on the V20 [ 23 ] dataset. Our PEARL use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Figure D10 : Qualitative results of open-vocabulary semantic segmentation. Results are shown on the PC59 [ 44 ] dataset. Our PEARL use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Figure D11 : Qualitative results of open-vocabulary semantic segmentation. Results are shown on the Stuff [ 8 ] dataset. Our PEARL use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Figure D12 : Qualitative results of open-vocabulary semantic segmentation. Results are shown on the City [ 17 ] dataset. Our PEARL use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Figure D13 : Qualitative results of open-vocabulary semantic segmentation. Results are shown on the ADE [ 79 ] dataset. Our PEARL use CLIP ViT-B/16 [ 51 ] , and no post-processing ( e.g . , PAMR [ 1 ] or DenseCRF [ 34 ] ) is applied for a fair comparison.

Graph-Based Label Propagation

See all 16 papers

Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation
Aug 4, 2026Wanli Ma, Jiangwen Lu, Qinmu Peng +1Open-Vocabulary Semantic Segmentation Vision-Language Alignment
School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · National Anti-counterfeit Engineering Research Center, Huazhong University of Science and Technology, Wuhan 430074, China · School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China
Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation
Apr 9, 2026Jiahao Li, Yang Lu, Yachao Zhang +3Open-Vocabulary Semantic Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University. · Hanjiang National Laboratory. · East China Normal University.
Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
Feb 26, 2026Tilemachos Aravanis, Vladan Stojnić, Bill Psomas +2Open-Vocabulary Semantic Segmentation Test-Time Adaptation of VLMs
VRG, FEE, Czech Technical University in Prague · University of Crete · Archimedes, Athena RC +1

Aug 4, 2026J/K move · Enter open · S save

Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation

Wanli Ma, Jiangwen Lu, Qinmu Peng, Xinge You

School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · National Anti-counterfeit Engineering Research Center, Huazhong University of Science and Technology, Wuhan 430074, China · School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China

Open-Vocabulary Semantic Segmentation Vision-Language Alignment VLM Calibration

Open paper AskPDF arXiv

Training-free open-vocabulary semantic segmentation (OVSS) partitions an image into semantically distinct regions based on arbitrary text descriptions, without learning any additional parameters. However, existing methods typically focus on improving visual representations while treating text embeddings that encode only generic category concepts as fixed classification references. The resulting semantic gap between these generic concepts and the visual representations that capture the specific appearances of target instances often causes incomplete masks and erroneous predictions in non-target regions. Inspired by the symbol-percept correspondence underlying perceptual anchoring, we propose Prototype-Guided Text Calibration (PTC) for training-free OVSS. In the Perceiving stage, PTC selects reliable visual evidence based on initial matching scores to construct category-specific visual prototypes. In the Anchoring stage, PTC uses these prototypes to calibrate their corresponding text embeddings, with the calibration strength adaptively adjusted based on the amount of visual evidence. Consequently, the calibrated text embeddings align more accurately with instance-specific visual representations while preserving generic category semantics and open-vocabulary generalization. Moreover, PTC requires neither additional training nor external models and can serve as a plug-and-play module for existing methods. Extensive experiments across eight benchmarks show that PTC significantly enhances the performance of six representative methods and yields more complete and accurate segmentation results. These results validate PTC as a simple and effective approach to improving visual-text alignment.

Abstract

Graph-Based Label Propagation

Explore similar work

Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation

Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation

Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?