SEER: Semantic Evidence for Executable Region Propagation in Open-Vocabulary Semantic Segmentation
Authors: Junyang Chen, Xiao Li, Yiguo Qiao, Wenbin Li
Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · Key Laboratory of Computer Network and Information Integration (Ministry of Education), Southeast University, Nanjing 211189, China · University of Bath, UK
Training-free open-vocabulary semantic segmentation (OVSS) has made strong progress by adapting CLIP-like representations to dense prediction, but its remaining errors are often local, structured, and tied to the active label space. A successful correction must be visually plausible, legal under the target vocabulary, and executable in the segmentation score field. We propose SEER, a training-free test-time repair layer for Semantic Evidence and Executable Region propagation. SEER casts post-prediction OVSS repair as evidence-conditioned label propagation. The OVSS score space proposes compact candidate labels, a frozen VLM verifies closed-set regional evidence, and an execution gate certifies whether the evidence is legally and semantically actionable. The accepted label is then propagated by an OVSS-side policy: coherent high-confidence regions allow broad propagation, while object- or boundary-risk cases use conservative score-grounded propagation. Across eight standard OVSS benchmarks, SEER reaches 58.0 average mIoU without updating OVSS weights. Matched-input controls isolate the repair mechanism, while resource measurements report the cost of the complete frozen system.
Figures & tables
Figure 1: SEER turns regional semantic evidence into executable OVSS label propagation. Starting from a fixed OVSS prediction, SEER selects ambiguous proposals, derives compact candidate labels from the active score space, and asks a VLM to verify regional evidence. The OVSS system executes certified evidence: score-supported seeds guide label propagation, while boundary-conservative guards restrict unsafe spreading near object and structure boundaries. The bar chart shows consistent mIoU gains across diverse OVSS benchmarks.
Figure 2: Overview of SEER . SEER converts regional VLM evidence into executable OVSS label propagation. From a fixed prediction, it selects ambiguous proposals, derives score-based candidates, and verifies them with multi-view closed-set reasoning. Score support and vocabulary legality certify the verified label, and the OVSS model controls its pixel propagation.
Method
Size
VOC21
VOC20
PC60
PC59
Object
Stuff
ADE
City
Avg.
Training-based
TCL Cha et al. (2023)
ViT-B/16
55.0
83.2
30.4
33.9
31.6
22.4
17.1
24.0
37.2
CLIP-DINOiser Wysoczańska et al. (2024)
62.1
80.9
32.4
35.9
34.8
24.6
20.0
31.7
40.3
CoDe Wu et al. (2024)
57.7
–
30.5
–
32.3
23.9
17.7
28.9
–
LoGoSeg Chen et al. (2026)
78.6
95.4
–
58.2
–
–
32.6
–
–
ESC-Net Lee et al. (2025)
80.1
97.3
–
59.0
–
–
35.6
–
–
Table 1: Complete-system accuracy on eight OVSS benchmarks. Published baseline numbers follow the CorrCLIP-style evaluation configuration and are grouped by CLIP size. SEER is an accuracy–cost point, not an equal-resource comparison: it adds fixed SAM2 proposals and sparse local Qwen3.5-9B verification to CorrCLIP. Paired end-to-end cost and matched-input attribution are reported in Tab. 7 and Tab. 4 .
Setting
VLM
Closed
Multi-view
Coherent
VOC21
PC60
City
ADE
Avg.
Initial prediction
72.88
44.01
47.41
28.63
48.23
Score-only rule
✓
+2.10
+3.80
+2.53
+2.13
+2.64
Free-form VLM
✓
✓
✓
+2.82
+0.00
+0.00
+0.00
+0.70
Single view
✓
✓
✓
+2.43
+0.30
+1.68
+0.67
+1.27
Conservative exec.
✓
✓
✓
+3.73
+2.47
+4.05
+7.06
+4.33
SEER
✓
✓
✓
✓
+6.72
+4.19
+9.49
+9.67
+7.52
Table 2: Component diagnostics. Each row is an alternative repair setting, not a cumulative build-up. The initial values are an independently frozen diagnostic S,Y0 , not the published CorrCLIP row in Tab. 1 ; every delta uses this identical state.
Configuration
Proposal gen.
OVSS
Audit
Writeback
Total
Overhead
Frozen OVSS
–
5.042
–
–
5.042
–
Cached SEER
–
5.042
0.424
0.043
5.509
+9.3%
Cold SEER
0.508
5.042
0.424
0.043
6.018
+19.3%
Table 7: End-to-end cost (s/image). Validation-weighted measurements on the same hardware. Cached SEER reuses fixed SAM2 proposals; cold SEER includes proposal generation. Audit load averages 0.71 calls per image.
Diagnostic
Observation
Execution outcome
GT purity ≥0.95
1,732 / 2,000 (86.6%)
+0.466 mean mIoU
GT purity <0.70
71 / 2,000 (3.55%)
43 / 71 filtered
Low-purity bin effect
−0.0055 / +0.0018
Near-zero aggregate effect
Correct high-confidence decisions
1,397
1,061 base confirmations
Correct non-base directions
336
327 executed (97.32%)
Empty supported execution
0
9 safely retain Y0
Table 8: Proposal quality and execution coverage. Purity and semantic correctness are analysis-only measurements over the fixed audit pool.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Study
Scope
Varied factor
Held fixed
Main
Full validation
Backbone; benchmark
Default verifier, prompt, candidates, policy
Components
Four benchmarks
Module ablation
Prediction, proposals, candidates, parser, split
Interface
Default audit pool
Response format
Prediction, proposals, executable gate, policy
Views
Default audit pool
Visual context
Candidates, verifier, executable gate, policy
Cross-base
Full pipeline
OVSS base
Same coefficients, thresholds, verifier, and rules
Budget
Ranked prefixes
Audit budget
Cached evidence, executable gate, policy
Appendix
Table 9: Unified protocol. Each row lists the experimental scope, the factor varied, and the quantities kept identical.
Ranks proposals before sparse VLM audit; weights are fixed once and not benchmark-fitted.
Risk predicate
Object-like candidate/base, ri>0.20 , or αi<0.015
Blocks broad writes for risky, small, or boundary-sensitive proposals.
Score support
(τs,τm,τl)=(0.02,0.20,0.25) in Eq. 14
Chooses score-compatible pixels for conservative propagation.
Propagation route
Broad support for coherent low-risk evidence; conservative support for object, boundary, small, or risky evidence
Keeps boundary evidence executable but restricts writes to selected support.
Parser and gate
Invalid JSON, out-of-set label, uncertain , base label, low/medium confidence, mixed , or unclear
Abstains with no pixel update; deterministic parser, no retry.
VLM audit
Local Qwen3.5-9B; three-view prompt; constrained JSON; deterministic decoding; 96-token cap
One audited proposal gives one sparse VLM call; no closed oracle model is used.
Appendix
Table 10: Deterministic execution contract. Fixed constants and routing rules are shared by all benchmarks; invalid or abstained audits never modify pixels.
Mask generator
VOC21
VOC20
PC59
PC60
City
ADE
Stuff
Object
Avg.
Mask2Former Cheng et al. (2022)
78.5
93.5
54.8
47.6
53.9
36.7
40.1
54.6
57.5
SAM2 Ravi et al. (2024)
79.6
94.2
55.2
48.2
56.9
38.3
38.6
52.9
58.0
EntitySeg Qi et al. (2023)
81.1
94.8
57.8
49.7
58.1
40.6
39.6
54.2
59.5
SAM3 Carion et al. (2025)
82.3
95.6
58.6
50.5
57.9
40.0
40.4
56.5
60.2
Appendix
Table 11: Proposal-source diagnostic. The main protocol uses SAM2 to align with the CorrCLIP-style configuration; this diagnostic varies only the mask generator that supplies spatial domains.
Interface
Target-in-set cov.
Sem. cand.
Repair output
Correct output
Base/no edit
Abstain/filter
Avg. gain
Base + 1 alt.
56.3%
2.00
41.3%
40.7%
50.1%
8.6%
+5.86
Base + 2 alt.
85.2%
2.97
47.6%
39.4%
46.3%
6.1%
+6.22
Base + 3 alt.
99.9%
3.35
58.8%
46.4%
28.7%
12.5%
+7.52
Appendix
Table 14: Candidate-set size and verifier outcomes. Rates are averaged over the four mechanism benchmarks with 2,000 audited proposals each. Base + k alt. denotes the current/base label plus up to k non-base score-derived alternatives; uncertain is an abstention output. Target-in-set coverage is diagnosis-only. Repair and correct-output rates are normalized by all audited proposals; executed precision is reported separately in Table 15 .
Benchmark
Cand. present
Raw high-conf. acc.
Executed precision
Filtered/no exec.
Invalid/OOS
VOC21
100.0%
62.4%
81.9%
38.9%
0.2%
PC60
99.5%
67.7%
92.7%
35.2%
0.4%
City
100.0%
48.5%
74.3%
36.2%
0.1%
ADE
100.0%
53.1%
80.2%
34.5%
0.1%
Average
99.9%
57.9%
82.3%
36.2%
0.2%
Appendix
Table 15: Execution filtering of VLM evidence. Candidate, filtered, and invalid/OOS rates use audited proposals as the denominator; raw accuracy and executed precision are measured on their corresponding pre- and post-filter subsets.
VOC20
VOC21
PC59
PC60
City
ADE
Stuff
Object
Avg.
Initial prediction
87.62
72.88
48.68
44.01
47.41
28.63
33.38
46.19
51.10
Coherent mIoU
91.92
77.38
53.18
46.91
53.61
35.03
36.38
49.19
55.45
Coherent gain
+4.30
+4.50
+4.50
+2.90
+6.20
+6.40
+3.00
+3.00
+4.35
Boundary mIoU
90.72
76.28
51.98
46.31
51.91
33.83
35.68
48.59
54.41
Boundary gain
+3.10
+3.40
+3.30
+2.30
+4.50
+5.20
+2.30
+2.40
+3.31
Appendix
Table 16: Full propagation-policy replay over eight benchmarks. Gains are measured over the fixed prediction. Beneficial/harmful pixels are diagnostic statistics computed after evaluation.
VOC20
VOC21
PC59
PC60
City
ADE
Stuff
Coherent harmful px
0.02
1.32
1.14
13.85
1.41
0.76
1.18
Boundary harmful px
0.00
1.28
0.85
13.29
0.67
0.47
0.58
Coherent beneficial px
2.32
8.22
50.97
99.13
90.54
48.00
125.65
Boundary beneficial px
1.24
6.48
35.88
71.09
38.63
34.13
84.63
Appendix
Table 17: Pixel exposure of propagation policies. Harmful pixels are shown in millions; full mIoU gains, including COCO-Object, are reported in Table 16 .
Benchmark
Random
Area-only
Uncertainty-only
Competition-only
Full priority
VOC21
+0.53
+4.55
+1.79
+3.70
+6.72
PC60
+0.27
+2.12
+2.00
+3.39
+4.19
City
+1.32
+4.73
+1.64
+3.55
+9.49
ADE
+1.17
+4.90
+3.42
+4.75
+9.67
Average
+0.82
+4.08
+2.21
+3.85
+7.52
Appendix
Table 18: Audited-pool priority replay. Values are mIoU gains over the fixed prediction when the same audited decisions are replayed under different proposal orderings.
Benchmark
Initial prediction
After relaxed gate
Gain
Executed actions
VOC21
72.88
78.58
+5.70
5,024
PC60
44.01
49.59
+5.58
7,572
City
47.41
56.56
+9.15
1,955
ADE
28.63
38.21
+9.58
1,975
COCO-Object
46.19
50.55
+4.36
1,608
Appendix
Table 19: Relaxing executable certification. Removing the target/legal gate increases recall but permits many more writeback actions.
OVSS base
Benchmark
Base
+ SEER
Δ
Δavg
ProxyCLIP
VOC21
61.3
64.9
+3.6
+4.53
PC60
35.3
39.5
+4.2
City
38.1
42.6
+4.5
ADE
20.2
26.0
+5.8
ResCLIP
VOC21
61.1
65.2
+4.1
+5.08
PC60
33.5
38.3
+4.8
Appendix
Table 26: Full-validation cross-base plug-in test. The same SEER rules are used for each OVSS base; Δavg averages four benchmark gains.
Benchmark
SEER closed
Oracle top-500
Oracle full pool
SEER/full
VOC21
+6.72
+13.01
+16.38
0.41
PC60
+4.19
+2.33
+6.56
0.64
City
+9.49
+13.92
+21.74
0.44
ADE
+9.67
+6.37
+19.54
0.49
Average
+7.52
+8.91
+16.05
0.50
Appendix
Table 29: Proposal-space repair headroom. Oracle rows are analysis-only diagnostics, not inference-time baselines. The full-pool oracle is the headroom row; the top-500 oracle is budget-limited and not a strict upper bound.
Benchmark
Images
Harmed img.
Med. loss
P75 loss
Mean gain
Neg. cls.
VOC21
1,449
5.8%
3.40
8.70
+4.70
0
PC60
5,104
0.9%
3.26
8.95
+2.83
0
City
500
5.0%
0.43
1.92
+6.18
0
ADE
2,000
3.8%
0.82
2.47
+2.91
5
Appendix
Table 30: Image/class harm. Harmed-image rates and losses are measured after SEER repair; negative classes count class-IoU drops.
Benchmark
Harm/mod.
Support spill
Sem. error
ADE
2.1%
69.8%
30.2%
City
2.1%
72.4%
27.6%
PC60
2.6%
57.2%
42.8%
VOC21
1.4%
58.7%
41.3%
Appendix
Table 31: Residual harm sources. Harmful-pixel rates are measured among modified pixels; source shares are normalized only over residual harmful pixels.
Benchmark
Uncertain/low
Candidate miss
Correct high-conf.
Conservative route
VOC21
4.8%
0.2%
89.6%
5.4%
PC60
3.7%
0.5%
87.2%
8.6%
City
3.4%
0.3%
89.5%
6.8%
ADE
4.6%
0.8%
84.9%
9.7%
Object
6.9%
0.7%
81.6%
10.8%
Appendix
Table 32: Regional-evidence routing taxonomy. Rates are computed over routed audited proposals. Conservative routing records boundary-aware execution; residual pixel harm is reported in the preceding table.
Figure 7: Boundary-sensitive and residual failure cases. SEER can recover proposal-supported regions absent from the initial top-1 prediction, but errors remain when proposals cross semantic boundaries, candidate labels are ambiguous, or the OVSS score field is weak. These cases motivate the execution gate and conservative propagation: plausible regional VLM evidence still needs score-field-constrained pixel support.
Figure 8: ADE20K visual examples. SEER repairs local semantic residues in indoor and scene-centric images while preserving the surrounding dense prediction structure.
Figure 9: Cityscapes visual examples. SEER targets structured road-scene residues such as missing stuff regions, fragmented objects, and label-space confusions in a fixed OVSS prediction.
Figure 10: PC59 visual examples. The repairs are localized to semantically supported regions and keep unrelated objects and stuff categories unchanged.
Figure 11: PC60 visual examples. The fixed prediction often contains plausible but incomplete regions; SEER uses closed-set evidence to propagate executable labels where the OVSS score field supports them.
Figure 12: VOC20 visual examples. Object-centric cases show that SEER repairs missing or confused object regions through closed-set evidence and score-space execution.
Figure 13: VOC21 visual examples. SEER improves residual object and background errors under the active label space, complementing the quantitative gains reported in the main paper.
Figure 14: COCO-Stuff visual examples. Stuff-heavy scenes highlight the value of label-space-aware evidence: SEER corrects broad semantic regions while retaining pixel-level execution in the OVSS model.
Figure 15: COCO-Object visual examples. Object-boundary cases are more conservative, but SEER still repairs reliable regions when the closed candidate and execution gate agree.
Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts, necessitating that existing methods possess pixel-level vision-language alignment capability. Typically, this capability involves computing the cosine similarity, \ie, logits, between visual and linguistic features, and minimizing the distribution discrepancy between the logits and the ground truth (GT) to generate optimal logits that are subsequently used to construct segmentation maps, yet it depends on time-consuming iterative training or model-specific attention modulation. In this work, we propose a more direct approach that eschews the logits-optimization process by directly deriving an analytic solution for the segmentation map. We posit a key hypothesis: the distribution discrepancy encodes semantic information; specifically, this discrepancy exhibits consistency across patches belonging to the same category but inconsistency across different categories. Based on this hypothesis, we directly utilize the analytic solution of this distribution discrepancy as the semantic maps. In other words, we reformulate the optimization of the distribution discrepancy as deriving its analytic solution, thereby eliminating time-consuming iterative training, freeing us from model-specific attention modulation, and achieving state-of-the-art performance on eight benchmark datasets.
Jiahao Li, Yang Lu, Yachao Zhang +3
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University. · Hanjiang National Laboratory. · East China Normal University.
Training-free open-vocabulary semantic segmentation (OVSS) promises rapid adaptation to new label sets without retraining. Yet, many methods rely on heavy post-processing or handle text and vision in isolation, leaving cross-modal geometry underutilized. Others introduce auxiliary vision backbones or multi-model pipelines, which increase complexity and latency while compromising design simplicity. We present PEARL, Procrustes alignment with text-aware Laplacian propagation, a compact two-step inference that follows an align-then-propagate principle. The Procrustes alignment step performs an orthogonal projection inside the last self-attention block, rotating keys toward the query subspace via a stable polar iteration. The text-aware Laplacian propagation then refines per-pixel logits on a small grid through a confidence-weighted, text-guided graph solve: text provides both a data-trust signal and neighbor gating, while image gradients preserve boundaries. In this work, our method is fully training-free, plug-and-play, and uses only fixed constants, adding minimal latency with a small per-head projection and a few conjugate-gradient steps. Our approach, PEARL, sets a new state-of-the-art in training-free OVSS without extra data or auxiliary backbones across standard benchmarks, achieving superior performance under both with-background and without-background protocols.
Gensheng Pei, Xiruo Jiang, Xinhao Cai +3
Department of Electrical and Computer Engineering, Sungkyunkwan University · School of Computing and Artificial Intelligence, Southwest Jiaotong University · School of Computer Science and Engineering, Nanjing University of Science and Technology
Open-vocabulary semantic segmentation (OVSS) repurposes a pretrained CLIP encoder for dense prediction without additional labeled supervision. Existing methods improve CLIP's spatial behavior either by redesigning its internal attention or by injecting features from auxiliary vision foundation models; both require access to the host's internal computation and are tailored to its specific forward pass. In this work, we propose Test-time Prototype Adaptation (TPA), a training-free plug-in that operates at the output level, leaving the host's forward pass and weights unmodified. By leveraging a lightweight transductive adaptation phase, TPA identifies confident anchor patches from the host's own output predictions on a small pool of unlabeled deployment-domain images, and aggregates their frozen DINO features into per-class prototypes; at inference, a single cosine similarity lookup against this frozen bank provides an auxiliary score fused linearly with the host's logits. TPA composes with five representative OVSS hosts spanning attention-redesign and VFM-injection designs, across three CLIP backbones, eight benchmarks, and multiple internal VFM choices. Under a single set of hyper-parameters and without per-host tuning or parameter updates, TPA consistently improves segmentation accuracy, with as few as approximately 10% of unlabeled deployment-domain images sufficing for effective bank construction on most benchmarks.
Haozhe Wang, Jintao Cheng, Weibin Li +1
South China Normal University · The Hong Kong University of Science and Technology