Token-Budgeted Candidate Retrieval for 3D Scene-Grounded VLMs: Category Names vs. Pairwise Geometry

Michał Zakrzewski; Mariia Shpir; Taras Rumezhak

cs.CVOct 9, 2026

Token-Budgeted Candidate Retrieval for 3D Scene-Grounded VLMs: Category Names vs. Pairwise Geometry

Authors: Michał Zakrzewski, Mariia Shpir, Taras Rumezhak

Organizations: AGH University of Krakow, Krakow, Poland · SoftServe · ETH Zurich

Abstract

Under a fixed context budget, which objects and relations should a 3D scene-grounded VLM see first? On ScanNet indoor referring benchmarks (Dai et al., 2017), matching the query to object category names outperforms pairwise geometry scoring at every reported K. The gaps are largest at high recall. Analytic pairwise cues beat per-object cue scoring and query-agnostic kNN graphs, including on paraphrases. MiniLM and BGE-small still score higher on those splits. We evaluate token-budgeted candidate retrieval with axis-aligned boxes, a token-ratio metric, training-free baselines, two small encoders that match q to each object's category name, and QCRR (Query-Conditioned Relation Retrieval), a deterministic pairwise-geometry scorer. Across ScanRefer gt_val plus a custom 20% scan-id split of Nr3D and Sr3D (35,504 queries; not official ReferIt3D val), additive noun matching reaches R@10 0.864 on ScanRefer against QCRR's 0.626; MiniLM and BGE-small reach 0.911 to 0.913 on ScanRefer and 0.941 to 0.947 on Nr3D. A corrected 3DGraphLLM harness (native [email protected] 0.67 on the full graph, n=300 ScanRefer queries) shows that context selection changes downstream accuracy under a matched object budget (Random 0.44, QCRR 0.56, MiniLM 0.67, oracle 0.70). Semantic retrieval beats hand-designed geometry on that harness. We will release the protocol, retriever implementations, and per-query outputs so new selectors can be swapped in without rewriting the analysis.

Figures & tables

Panel of Figure 1 — Figure 1: QCRR is a five-stage analysis instrument: parse spatial cues from $q$ , score ordered object pairs with analytic geometry on axis-aligned boxes, down-weight structural hubs, then return ranked relations and candidate objects.

Cue family	Signal
Left / Right	$\sigma(\pm\Delta x_{ij})$
Front / Behind	$\sigma(\mp\Delta y_{ij})$
Above / Below	$\sigma(\pm\Delta z_{ij})$
Near / Far	$1-d_{ij}/d_{\max}$ / $d_{ij}/d_{\max}$
On / Inside	vertical overlap $\times$ containment
Overlap	3D IoU

Table 1: Cue families mapped to geometric signals in the ScanNet world frame.

\sigma

is a logistic;

d_{\max}

is the scene diagonal;

\Delta x_{ij}

is the world-frame center offset of pair

(i,j)

Method	R@1	R@5	R@10	R@20
Random	0.032	0.163	0.322	0.606
$k$ NN-graph	0.027	0.170	0.326	0.609
Cue-only + noun	0.260	0.673	0.818	0.924
QCRR	0.202	0.493	0.626	0.844
Learned (MiniLM)	0.461	0.826	0.911	0.973
Learned (BGE-small)	0.455	0.830	0.913	0.974

Table 2: Retrieval on ScanRefer gt_val (

9{,}508

queries). Bold marks best per column among listed methods.

	ScanRefer				Nr3D				Sr3D
Method	R@1	R@5	R@10	R@20	R@1	R@5	R@10	R@20	R@1	R@5	R@10	R@20
Random	.032	.163	.322	.606	.028	.161	.321	.597	.022	.119	.226	.462
$k$ NN-graph	.027	.170	.326	.609	.031	.180	.332	.603	.016	.094	.200	.451
Cue-only + noun	.260	.673	.818	.924	.212	.735	.856	.935	.267	.960	.993	.999
QCRR	.202	.493	.626	.844	.182	.524	.634	.820	.254	.599	.616	.754
MiniLM	.461	.826	.911	.973	.271	.857	.941	.973	.262	.976	.996	.999

Table 3: Cross-benchmark retrieval. Nr3D/Sr3D use a custom

20\%

scan-id split (seed 42), not official ReferIt3D val; ScanRefer is gt_val, matching Table 2 . MiniLM and BGE-small match

q

to category names only.

Figure 2: ScanRefer Recall@10 vs. token ratio as relation budget

M

varies. Main tables use

M{=}60

Object text	R@1	R@5	R@10	R@20
Category name	0.461	0.826	0.911	0.973
Leave-one-out descriptions	0.587	0.914	0.971	0.996

Table 4: MiniLM on ScanRefer gt_val. Leave-one-out uses other val descriptions of the same object, not the current query.

		QCRR			Cue-only
Bucket	#queries	R@1	R@5	R@10	R@1	R@5	R@10
Exact-match	8,660	0.251	0.529	0.652	0.231	0.592	0.756
Non-exact	848	0.037	0.236	0.440	0.018	0.103	0.288

Table 5: ScanRefer gt_val stratified by verbatim category match. Non-exact queries use synonyms, hypernyms, or paraphrases (e.g., “wooden cabinets” for kitchen cabinets ).

Class	#queries	$S2$	$S4$	$\delta$
chair	2,196	0.15	0.74	0.79
cabinet	781	0.23	0.56	0.60
table	768	0.32	0.58	0.45
counter	84	0.71	0.79	0.09
bathtub	50	0.62	0.62	0.00

Table 6: Per-class disambiguation on ScanRefer val (Mask3D [email protected], not retrieval R@

K

S2

: lexical;

S4

: class oracle;

\delta\!=\!1{-}S2/S4

$N$ bin	#queries	$\bar{N}$	QCRR	Centroid	Cue-only + noun
1–20	1,061	16	0.926	0.685	0.951
21–40	4,932	30	0.781	0.391	0.864
41–60	2,549	48	0.594	0.217	0.741
61+	966	78	0.487	0.174	0.609

Table 7: Recall@10 by scene size on ScanRefer gt_val.

	Spatial lang. ( $7{,}140$ )				No spatial lang. ( $778$ )
Method	R@1	R@5	R@10	R@20	R@1	R@5	R@10	R@20
$k$ NN-graph	0.031	0.181	0.335	0.602	0.036	0.170	0.306	0.609
Cue-only	0.163	0.554	0.717	0.864	0.162	0.526	0.716	0.870
QCRR	0.177	0.520	0.627	0.829	0.244	0.558	0.647	0.748
Learned (MiniLM)	0.260	0.850	0.939	0.973	0.369	0.918	0.961	0.976

Table 8: Nr3D split by ReferIt3D uses_spatial_lang (not official view-dependence). MiniLM matches

q

to names.

Cue bucket	#queries	R@10
left/right	3,957	0.696
near/far	3,155	0.733
vertical (above/below)	1,303	0.773
non-directional (near/on/overlap)	1,091	0.681

Table 9: Dominant cue type on ScanRefer gt_val (QCRR Recall@10). Vertical cues are easiest in the gravity-aligned ScanNet frame.

Context selector	[email protected]	GT in budget
Oracle (GT-incident relations first)	0.697	1.00
Learned retriever (MiniLM)	0.673	0.96
Full graph (no budget)	0.670	1.00
Noun baseline	0.647	0.94
QCRR	0.557	0.82
Random	0.443	0.60

Table 10: Downstream [email protected] on ScanRefer gt_val (

n{=}300

, seed 42) under a matched object budget. GT-in-budget is a separate column. Full graph included as an unconstrained reference.

Config	R@1	R@5	R@10	R@20
A0: Full QCRR (max-edge)	0.202	0.493	0.626	0.844
A2: $-$ hub penalty	0.172	0.446	0.588	0.825
A3: $-$ noun boost	0.048	0.231	0.428	0.675
A6: sum_top3 aggregator	0.169	0.377	0.536	0.835

Table 11: QCRR ablations on ScanRefer gt_val (

9{,}508

queries), same protocol as Table 2 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	R@1	R@5	R@10	R@20	Token ratio
QCRR (full pairwise)	0.230	0.501	0.631	0.848	0.081
Noun-restricted hybrid	0.297	0.718	0.828	0.911	0.071

Appendix

Table 12: Noun-restricted hybrid vs. full pairwise QCRR on leftover ScanRefer val.

Metric	Original	Canonicalized	$\Delta$
R@1	0.178	0.205	$+0.027$
R@5	0.424	0.471	$+0.047$
R@10	0.588	0.616	$+0.027$

Appendix

Table 13: PCA room-axis alignment on ScanRefer val, left/right queries only (

n{=}4{,}272

Subset	#queries	S0 [email protected]	S2 [email protected]	S0 [email protected]	S2 [email protected]
ST-wo-d	2,099	0.022	0.322	0.020	0.289
ST-w-d	5,358	0.025	0.185	0.021	0.167
MT	2,757	0.037	0.312	0.032	0.286
ZT-w-d	378	0.000	0.421	0.000	0.421
ZT-wo-d	528	0.000	0.695	0.000	0.695
All	11,120	0.026	0.281	0.022	0.255

Appendix

Table 14: Multi3DRef val (

11{,}120

queries). S0: random; S2: lexical class match (grounding-level).

Class	#queries	$\%$ Unique	$S2$	$S4$	$\delta$
chair	2,196	3.9	0.153	0.736	0.79
cabinet	781	17.3	0.225	0.565	0.60
table	768	31.0	0.319	0.581	0.45
door	520	49.6	0.271	0.396	0.32
window	366	41.3	0.372	0.604	0.38
desk	238	39.5	0.244	0.433	0.44

Appendix

Table 15: Per-class breakdown of category-matching difficulty on ScanRefer val (

9{,}508

queries, Mask3D-detected boxes).

\%

Unique is the fraction of queries on which the target class has a single instance.

S2

is lexical [email protected];

S4

is the class-oracle ceiling;

\delta\!=\!1{-}S2/S4

Vision-Language Retrieval

Vision-language retrieval finds images, video, or documents that match natural-language queries and supports search across modalities.

See all 98 papers

SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs
May 27, 2026Jiawei Li, Ziyi Liu, Weijie Shi +33D Spatial Reasoning Latent Visual Reasoning
The Hong Kong University of Science and Technology · Soochow University
P $^3$ -VLM: A Point-based Alternative for Grounded 3D Vision-Language Models
Oct 20, 2026Anna-Maria Halacheva, Jan-Nico Zaech, Sombit Dey +23D VLMs 3D Visual Grounding
INSAIT, Sofia University “St. Kliment Ohridski”
Do 3D Large Language Models Really Use 3D Evidence?
Oct 17, 2026Yifei Zhao, Fanyu Zhao, Han Zhang +63D VLMs 3D VQA
Fudan University · University of Science and Technology of China

May 27, 2026J/K move · Enter open · S save

SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs

Jiawei Li, Ziyi Liu, Weijie Shi, Long Chen+2

The Hong Kong University of Science and Technology · Soochow University

3D Spatial Reasoning Latent Visual Reasoning Visually Grounded Reasoning 3D VLMs 3D Visual Grounding

Open paper AskPDF arXiv

3D object grounding localizes referred objects in a 3D scene from natural language. Unified instance-centric 3D-LLMs aim to solve grounding together with dialog, QA, and captioning, yet many rely on a single pointer-style grounding decision that compresses a relational instruction into one selection. This is brittle for fine-grained queries where multiple same-class candidates must be ruled out by context objects and spatial relations. We propose Structured Spatial Reasoning 3D-LLM (SSR3D-LLM), a structured grounding interface for unified 3D-LLMs. Given fixed Mask3D object proposals, the LLM writes a sequence of latent spatial reasoning steps and memory tokens from the query, and a geometry-aware scorer reads these latent steps in order to refine candidate rankings step by step with step-length masking. The latent steps are learned from standard benchmark target supervision with auxiliary referential-cue supervision during training, while inference uses only the input query and Mask3D proposals. Across ReferIt3D, ScanRefer, and Multi3DRef, SSR3D-LLM achieves the strongest results among unified 3D-LLM baselines, with substantial gains over the single-pointer QPG baseline on fine-grained grounding and consistent improvements over prior unified 3D-LLMs, while preserving the default language-task route.

Abstract

Vision-Language Retrieval

Explore similar work

SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs

P3^33-VLM: A Point-based Alternative for Grounded 3D Vision-Language Models

Do 3D Large Language Models Really Use 3D Evidence?

P $^3$ -VLM: A Point-based Alternative for Grounded 3D Vision-Language Models