Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation

Dominik Engel; Yousef Rajeh; Leon Sick; Ivan Viola

cs.CVOct 22, 2026

Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation

Authors: Dominik Engel, Yousef Rajeh, Leon Sick, Ivan Viola

Organizations: King Abdullah University of Science and Technology (KAUST) · Ulm University

Abstract

Lifting vision-language features such as CLIP into 3D Gaussian reconstructions enables open-vocabulary segmentation through natural-language queries. Although CLIP represents a broad range of semantic concepts, the lifted features of individual scenes can exhibit strong directional concentration, while raw cosine similarity does not account for the scene-specific feature distribution. We show that substantial segmentation gains can be obtained by changing how a fixed lifted feature field is compared with text. Our method partially subtracts a shared scene direction from visual features and text prototypes and whitens text prototypes using a fixed reference vocabulary of generic terms. It then uses soft clustering for multiclass prediction and CSLS with a relevance margin against the reference vocabulary for single-query prediction. Beyond reducing directional concentration, scene centering improves class-relative feature separation in every evaluated scene. Across seven lifting methods and four datasets, the resulting query-time calibration consistently improves mean intersection over union in both multiclass and single-query segmentation. These gains require no retraining, source-image re-encoding, or feature re-lifting, demonstrating the value of scene-conditioned comparison for making better use of existing lifted representations.

Figures & tables

Panel of Figure 1 — Figure 1: Method Overview. Starting from a Gaussian scene with lifted CLIP features, we calibrate the scene and text query features using centering and text-whitening, based on a reference vocabulary. Instead of plain query-scene similarities, we further calibrate the similarity scores using soft clustering for multi-class and CSLS with a relevance margin for single-query. Our calibration results in increased class-relative separation, as visualized in the similarity maps on the bottom. Our final predictions achieve substantial improvements in segmentation mIoU for both multi-class (visualized in bar chart) and single class.

Panel of Figure 2 — Figure 2: Scene centering maps blue input vectors ${\bm{z}}$ to teal calibrated vectors ${\bm{u}}$ by subtracting the orange mean direction ${\bm{\mu}}$ and renormalizing.

Panel of Figure 3 — Figure 3: Score calibration. Starting from the Middle: First we use the reference vocabulary to determine the whitening statistics. Left: For multi-class queries ( $N>1$ ) we center and whiten all text queries and perform soft clustering of the primitive features, using fixed text-derived probabilities to guide assignments and weighted top-eight visual primitives to initialize the centers. After clustering, argmax decides the class of each primitive. Right: For a single-class query ( $N=1$ ) we center and whiten the query and the reference vocabulary, apply reduced CSLS and compute the margin between each primitive’s corrected query score and its highest corrected reference-bank score. To make a prediction we threshold this relevance margin using $\tau$ .

	LERF-OVS			3D-OVS			ScanNet (19-class)			ScanNet++ (100-class)
Method	Raw	Ours	$\Delta$	Raw	Ours	$\Delta$	Raw	Ours	$\Delta$	Raw	Ours	$\Delta$
Multi-Query $N>1$
SFS Xiong et al. (2026)	49.09	65.67	$+16.57$	70.79	89.80	$+19.01$	26.87	39.37	$+12.50$	19.92	23.74	$+3.81$
NormLift Zang et al. (2026)	58.55	60.25	$+1.70$	70.90	86.11	$+15.20$	31.40	39.55	$+8.14$	20.60	24.76	$+4.16$
LUDVIG ∗ Marrie et al. (2025)	38.05	45.44	$+7.39$	4.15	5.85	$+1.70$	29.32	39.83	$+10.51$	20.31	23.96	$+3.66$
Occam Cheng et al. (2025)	57.13	61.05	$+3.92$	66.19	84.21	$+18.01$	23.87	32.11	$+8.24$	20.18	23.16	$+2.98$

Table 1: Segmentation Results Raw vs. Calibrated . Top: multi-class (

N>1

). Bottom: single-query (

N=1

). The

N=1

results use the fixed held-out thresholds stated in Appendix C . Smaller ScanNet 10-class and ScanNet++ 20-class results, as well as precision and recall are reported in Appendix A .

Method	ScanNet	ScanNet++
Ours	$\mathbf{39.37}$ $(+12.50)$	$\mathbf{23.74}$ $(+3.81)$
TransCLIP	$35.54$ $(+8.68)$	$22.81$ $(+2.89)$
PTC	$27.92$ $(+1.05)$	$20.81$ $(+0.89)$
Frolic-inspired	$28.37$ $(+1.51)$	$16.90$ $(-3.02)$
UMFC	$11.31$ $(-15.55)$	$9.23$ $(-10.69)$

Table 2: Comparison to other adaptations. Values are mIoU (

\Delta

over raw) for

N>1

on SFS. Other methods are unsuitable for

N=1

C1	C2	C3	ScanNet	ScanNet++
			$26.87$	$19.92$
$\checkmark$			$31.83$ $(+4.97)$	$18.77$ $(-1.15)$
	$\checkmark$		$35.09$ $(+8.22)$	$23.66$ $(+3.73)$
$\checkmark$	$\checkmark$		$36.37$ $(+9.50)$	$21.80$ $(+1.88)$
$\checkmark$		$\checkmark$	$36.49$ $(+9.63)$	$21.15$ $(+1.23)$
$\checkmark$	$\checkmark$	$\checkmark$	$\mathbf{39.37}$ $(+12.50)$	$\mathbf{23.74}$ $(+3.81)$

Table 3: Component ablation. Values are mIoU (

\Delta

over raw) using SFS.

Panel of Figure 4 — Figure 4: Qualitative query maps on eight ScanNet++ scenes. Each row shows one query. In mesh-derived binary GT, yellow denotes the query and purple denotes zero. Raw cosine and fully calibrated score maps are independently min–max normalized (purple: low; yellow: high). Note how the calibrated scores increase contrast between foreground and background.

Panel of Figure 5 — Figure 5: Qualitative multiclass segmentation on 3D-OVS and LERF-OVS. Each 3D-OVS row or LERF-OVS row pair shows one annotated view. For $N>1$ , argmax assigns every supported pixel to a queried class, precluding background, merged $N=1$ masks instead retain background where all queries are rejected.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

				ScanNet			ScanNet++
Center	Whiten	CSLS	Bank	mIoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	mIoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
				26.93	36.44	63.20	22.65	36.00	40.78
$\checkmark$				26.40	39.05	54.18	18.60	31.22	35.32
			$\checkmark$	27.47	40.41	58.59	21.71	35.96	37.87
		$\checkmark$	$\checkmark$	28.60	40.94	60.00	24.01	36.43	41.29
$\checkmark$	$\checkmark$		$\checkmark$	30.71	47.82	57.31	20.56	34.53	38.75

Appendix

Table 4: Single-query component ablations. We report ScanNet and ScanNet++ using SFS in the single-query setting. Each ablations threshold is fit per method and dataset using the held-out calbiration scenes. Both Whiten and CSLS depends on Bank being present, as they are no-ops on a single query. Ablations with Bank use query vs. max-reference margin, otherwise the query score is thresholded directly.

	LERF-OVS									3D-OVS
	mIoU $\uparrow$			Precision $\uparrow$			Recall $\uparrow$			mIoU $\uparrow$			Precision $\uparrow$			Recall $\uparrow$
Method	Raw	Ours	$\Delta$	Raw	Ours	$\Delta$	Raw	Ours	$\Delta$	Raw	Ours	$\Delta$	Raw	Ours	$\Delta$	Raw	Ours	$\Delta$
Multi-Query $N>1$
SFS	49.09	65.67	$+16.57$	67.67	77.11	$+9.45$	61.46	71.23	$+9.77$	70.79	89.80	$+19.01$	75.94	94.09	$+18.15$	88.95	95.58	$+6.63$
NormLift	58.55	60.25	$+1.70$	74.58	68.42	$-6.16$	68.23	66.97	$-1.26$	70.90	86.11	$+15.20$	74.41	90.49	$+16.08$	88.00	93.11	$+5.10$
LUDVIG	38.05	45.44	$+7.39$	59.77	57.75	$-2.02$	49.76	58.34	$+8.57$	4.15	5.85	$+1.70$	10.50	16.80	$+6.30$	14.07	14.59	$+0.52$

Appendix

Table 5: Segmentation Results Raw vs. Calibrated across All Datasets. Each dataset pair reports multi-query (

N>1

), followed by single-query (

N=1

) results.

object	stuff	surface	material
background	structure	furniture	room
clutter	corner	space	interior
scene	region	pattern	shape
volume	boundary	arrangement	composition
portion	section	expanse	form
void	span	zone	patch

Appendix

Table 6: Reference Vocabulary.

$\lambda\backslash\alpha$	0.30	0.35	0.40	0.45	0.50
0.20	40.06 23.35	39.95 23.34	39.91 23.36	39.88 23.45	39.86 23.41
0.25	39.84 23.51	39.73 23.44	39.69 23.47	39.50 23.77	39.41 23.74
0.30	39.47 23.55	39.40 23.71	39.37 23.74	39.35 23.70	39.38 23.69
0.35	39.16 23.48	39.14 23.41	39.12 23.49	38.88 23.50	38.87 23.51
0.40	39.38 23.80	38.74 23.64	38.71 23.64	38.68 23.71	38.69 23.70

Appendix

Table 7: Parameter sensitivity. SFS multiclass mIoU (%) on ScanNet (19 classes) / ScanNet++ (100 classes) is shown on the top / bottom of each cell. The bold cell marks the

(\lambda,\alpha)

chosen by held-out calibration.

Dataset	Method	Low opacity	Invalid feature	Both	Either	GT dropped	GT unsupported
ScanNet	SFS	26.33	21.65	19.98	28.00	0.00	1.94
ScanNet	NormLift	26.33	21.67	19.55	28.46	0.00	2.14
ScanNet	LUDVIG	26.33	19.30	17.61	28.02	0.00	1.98
ScanNet	Occam	28.38	0.00	0.00	28.38	0.00	0.00
ScanNet	VALA	0.23	0.00	0.00	0.23	0.00	0.00
ScanNet	LangSplat	26.33	0.00	0.00	26.33	0.00	0.00

Appendix

Table 8: Discarded prediction statistics. Low opacity, invalid feature, their intersection (Both), and their union (Either) characterize ineligibility as percentages of primitives. GT dropped and GT unsupported report percentages of GT points omitted from evaluation or left without a usable prediction under prior evaluation rules. In contrast, our evaluation retains every active GT point and assigns it to an eligible Gaussian.

Open-Vocabulary 3D Segmentation

Open-vocabulary 3D segmentation assigns language-queryable semantic and instance masks to 3D scenes, including objects unseen during training.

See all 74 papers

SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation
Aug 6, 2026Mohamad Zamini, Diksha ShuklaMemory-Augmented VLMs Open-Vocabulary Semantic Segmentation
University of Wyoming Laramie, WY
DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction
Mar 14, 2026Jing Wang, Huimin Shi, Quan Zhou +3Open-Vocabulary Semantic Segmentation Text-Guided Image Segmentation
National Engineering Research Center of Communications and Networking, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China. · Institute for Advanced Ocean Research (Nantong), Southeast University, Nantong 226334, P. R. China. · Department of Internet of Things, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China +1
NormLift: From Lifted Features To Semantic Reliability In 3D Gaussian Splatting
Sep 16, 2026Yihan Zang, Da Li, Dominik Engel +2Open-Vocabulary 3D Segmentation Language-Embedded 3DGS
CEMSE Division, King Abdullah University of Science and Technology

Aug 6, 2026J/K move · Enter open · S save

SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation

Mohamad Zamini, Diksha Shukla

University of Wyoming Laramie, WY

Memory-Augmented VLMs Open-Vocabulary Semantic Segmentation VLM Inference

Open paper AskPDF arXiv

Training-free open-vocabulary segmentation remains limited by a missing inference abstraction. Frozen vision-language features are produced at patch level, yet dense prediction requires a unit that simultaneously governs feature interaction, spatial support, contextual recovery, and retrieval-based correction. We present SCI-CLIP, a segment-centric inference framework built around the principle that the same region abstraction should organize all stages of dense open-vocabulary prediction. SCI-CLIP first induces a region-consistent interaction graph over frozen visual tokens, then reconstructs dense features by propagating values over this graph, augmenting them with selective cross-window support only where local evidence is insufficient. The same segment abstraction is subsequently used to construct and query an offline reference memory, aligning exemplar retrieval with the units on which prediction is made. SCI-CLIP turns frozen CLIP-style features into spatially coherent, context-aware, and retrieval-compatible dense predictions without any training. SCI-CLIP consistently improves the structural quality of dense predictions, the robustness of contextual reasoning, and the alignment of exemplar-based correction, yielding stronger open-vocabulary segmentation across eight benchmarks. Project code is available at: https://github.com/mzamini92/SCICLIP.

Abstract

Open-Vocabulary 3D Segmentation

Explore similar work

SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation

DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction

NormLift: From Lifted Features To Semantic Reliability In 3D Gaussian Splatting