TrianguLang: Geometry-Aware Semantic Consensus for Pose-Free 3D Localization

Bryce Grant; Aryeh Rothenberg; Atri Banerjee; Peng Wang

cs.CVMar 9, 2026

TrianguLang: Geometry-Aware Semantic Consensus for Pose-Free 3D Localization

Authors: Bryce Grant, Aryeh Rothenberg, Atri Banerjee, Peng Wang

Organizations: Case Western Reserve University, Cleveland, OH, USA

Abstract

Localizing objects and parts from natural language in 3D space is essential for robotics, AR, and embodied AI, yet existing methods face a trade-off between the accuracy and geometric consistency of per-scene optimization and the efficiency of feed-forward inference. We present TrianguLang, a feed-forward framework for 3D localization that requires no camera calibration at inference. Unlike prior methods that treat views independently, we introduce Geometry-Aware Semantic Attention (GASA), which utilizes predicted geometry to gate cross-view feature correspondence, suppressing semantically-plausible but geometrically-inconsistent matches without requiring ground-truth poses. Validated on five benchmarks including ScanNet++ and uCO3D, TrianguLang achieves state-of-the-art feed-forward text-guided segmentation and localization, reducing user effort from $O(N)$ clicks to a single text query. The model processes each frame at 1008x1008 resolution in $\sim$ 57ms ( $\sim$ 18 FPS) without optimization, enabling practical deployment for interactive robotics and AR applications. Code and checkpoints are available at https://cwru-aism.github.io/triangulang/.

Figures & tables

Figure 1: Overview of the TrianguLang architecture

Figure 3: Performance on uCO3D and ScanNet++ datasets. Left-to-right: RGB, depth map, ground truth, TrianguLang masks

Model	Setting	Train $\rightarrow$ Eval	mIoU ( $\uparrow$ )	mAcc ( $\uparrow$ )
MV-SAM	In-domain	ScanNet++ $\rightarrow$ ScanNet++	0.510	0.694
	In-domain	uCo3D $\rightarrow$ uCo3D	0.910	0.965
	Cross-domain	uCo3D $\rightarrow$ ScanNet++	0.194	0.251
	Cross-domain	ScanNet++ $\rightarrow$ uCo3D	0.322	0.517
	Large-scale	SA-1B $\rightarrow$ ScanNet++	0.489	0.635
	Large-scale	SA-1B $\rightarrow$ uCo3D	0.877	0.950

Table 1: In-domain, cross-dataset, and large-scale evaluation results. MV-SAM numbers from [ 13 ] . TrianguLang uses text-only prompts; MV-SAM uses 12 click prompts per object. SA-1B denotes MV-SAM trained on single-view SA-1B data (millions of images).

Method	mIoU	Loc. Acc.	Per-scene	Time
LERF [ 17 ]	37.4	73.6	✓	$\sim$ 45 min
LangSplat [ 32 ]	51.4	84.3	✓	$\sim$ 10 min
LangSplat-V2 [ 22 ]	59.9	84.1	✓	$\sim$ 10 min
TrianguLang †	59.2	89.1	✗	$\sim$ 58ms

Table 2: Comparison with per-scene language grounding methods on the LERF-OVS benchmark. Per-scene methods require calibrated poses and scene reconstruction. TrianguLang operates feed-forward using ScanNet++-trained weights without any training on LERF scenes.

Figure 4: Qualitative comparison on LERF-OVS scenes using uniform clip thresholds. Row 1: “toaster” query: LERF and LangSplatV2 produce diffuse activations across the scene while TrianguLang tightly focuses its relevancy map on the target object. Row 3: “stripes” query: TrianguLang achieves precise localization despite not training on this dataset, and runs 3 orders of magnitude faster ( $\sim$ 58ms vs. 10 to 45 min).

Method	Localization Accuracy (%)					Semantic Segmentation (mIoU %)
Method	Ramen	Teatime	Kitchen	Figurines	Overall	Ramen	Teatime	Kitchen	Figurines	Overall
Gaussian Grouping [ 45 ]	32.4	69.5	50.0	44.6	49.1	26.4	54.0	31.3	34.6	36.6
LEGaussians [ 38 ]	69.0	79.7	63.6	57.1	67.4	20.2	32.3	22.3	23.4	24.6
LERF [ 17 ]	62.0	84.8	72.7	75.0	73.6	28.2	45.0	37.9	38.6	37.4
GOI [ 33 ]	56.3	67.8	68.2	44.6	59.2	33.7	55.8	54.5	23.9	42.0
GAGS [ 31 ]	69.0	88.1	90.9	78.6	81.7	46.8	60.3	55.8	53.6	54.1

Table 3: Per-scene comparison on LERF-OVS benchmark. TrianguLang uses ScanNet++-trained weights without any LERF training (zero-shot transfer). Per-scene methods require 10 to 45 minutes of optimization per scene.

Configuration	mIoU	$\Delta$
Baseline (GASA + World PE)	54.7	$-$
$-$ GASA kernel (standard attn)	49.4	$-$ 5.3
$-$ World-space PE	49.3	$-$ 5.4
$-$ Both (no GASA, no PE)	46.3	$-$ 8.4
RBF kernel (vs. learned MLP)	44.0	$-$ 10.7

Table 4: Component ablation on ScanNet++ (230 scenes, 8 views, 504 resolution).

\Delta

measured against ablation baseline (54.7% mIoU). The full model in Table 1 achieves 62.4% at 1008 resolution with 230 scenes.

VGM	ScanNet++	uCO3D	LERF-OVS
DA3-NESTED [ 23 ]	62.4	72.1	59.2
MapAnything [ 15 ]	63.0	72.1	58.1
$\pi^{3}$ -VO [ 42 ]	62.7	66.8	51.8

Table 5: Sensitivity to visual geometry backbone. All models use the same GASA decoder architecture trained on ScanNet++ and evaluated zero-shot on uCO3D and LERF-OVS. mIoU (%) reported.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Qualifier	Synonyms	Selection Criterion
Base qualifiers (7 learnable embeddings)
Depth	nearest	closest, close	$\arg\min_{i}d_{i}$
	farthest	far, distant	$\arg\max_{i}d_{i}$
Horizontal	leftmost	left	$\arg\min_{i}x_{i}$
	rightmost	right	$\arg\max_{i}x_{i}$
Vertical	topmost	top, upper	$\arg\min_{i}y_{i}$

Appendix

Table 6: Spatial qualifier vocabulary (48 recognized phrases across 7 base categories). Index 0 is reserved for queries without spatial qualifiers. Ordinal and size-based extensions share learnable embedding indices with their related base qualifiers.

Configuration	mIoU
No spatial features	40.07
Spatial-as-points	43.54
Object-aware selection	47.01

Appendix

Table 7: Spatial feature ablation on ScanNet++. Object-aware selection achieves the best segmentation quality.

Data
Hyperparameter	Value
Scenes	230
Views per sample	8
Image resolution	1008
DA3 resolution	1008
Optimization

Appendix

Table 8: Training hyperparameters for TrianguLang.

Component	Parameters	Trainable
SAM3 Backbone	848M	Frozen
DA3 Depth (NESTED-GIANT-LARGE)	1.4B	Frozen
GASA Decoder	13.4M	✓
Query embeddings	26K	✓
Self-attention layers	1.2M	✓
Text cross-attention	1.2M	✓

Appendix

Table 9: Model parameter breakdown.

Method	Queries	Predicted	Oracle	Gap
SAM3 [ 2 ]	$\sim$ 200	49.8	79.9	30.1
TrianguLang	10	62.4	63.4	1.0

Appendix

Table 10: Oracle vs. predicted mIoU on ScanNet++ (49 val scenes, 100 frames/scene). Oracle selects the GT-best mask among candidates. The gap quantifies the mask selection bottleneck.

Config	GPUs	Time/epoch	Memory	Inference
Full (230s, 8v)	8 $\times$ A100	$\sim$ 10 min	45 GB	$\sim$ 107ms

Appendix

Table 11: Training and inference computational cost.

Type	Method	NVOS	SPIn-NeRF
Feed-forward	SAM2-Video	88.7	86.6
	MV-SAM	92.1	92.5
	TrianguLang	93.5	91.4
Per-scene optimization	SPIn-NeRF	$-$	90.7
	SA3D	91.1	92.4
	SAGA	92.6	93.7

Appendix

Table 12: Comparison on NVOS and SPIn-NeRF benchmarks (mIoU). Per-scene optimization methods require calibrated poses and 30 to 60 minutes of training per scene. TrianguLang and MV-SAM are feed-forward methods requiring no per-scene computation.

Method	ScanNet++		uCo3D
Method	mIoU ( $\uparrow$ )	mAcc ( $\uparrow$ )	mIoU ( $\uparrow$ )	mAcc ( $\uparrow$ )
Prompt Projection †	29.2	59.2	78.2	83.3
SAM2-Video [ 35 ]	46.1	61.4	81.9	91.3
SAM2-Long [ 9 ]	41.5	61.4	72.9	86.4
SAM3 [ 2 ]	48.6	63.4	82.4	91.4
MV-SAM	48.9	63.5	87.7	95.0

Appendix

Table 13: Comparison of mIoU and mAcc on ScanNet++ and uCo3D. All baselines use their respective SA-1B-trained weights. Prompt Projection † unprojects prompts into 3D and re-projects per view to run SAM2 independently. SAM3 alone (no cross-view reasoning) underperforms MV-SAM, confirming that TrianguLang’s gains come from GASA, not the backbone.

Method	fern	flower	fortress	horns_center	horns_left	leaves	trex	Avg.
SA3D [ 4 ]	82.90	94.60	98.30	96.20	90.20	93.20	81.99	91.06
SA3D [ 4 ]	94.39	98.74	99.68	99.33	99.36	99.57	97.41	98.35
SAGA [ 3 ]	83.53	96.62	98.16	98.06	93.59	93.51	80.81	92.57
SAGA [ 3 ]	94.60	99.17	99.65	99.50	99.51	99.59	97.26	98.55
OmniSeg3D [ 47 ]	82.70	95.30	98.50	97.70	95.60	92.70	87.40	92.84
OmniSeg3D [ 47 ]	94.30	98.90	99.70	99.60	99.70	99.50	98.30	98.57

Appendix

Table 14: Per-scene quantitative results on NVOS. The first row of each method corresponds to metric mIoU and the second row to metric mAcc. TrianguLang achieves state-of-the-art average mIoU (93.46%) among feed-forward methods, approaching per-scene optimization methods while using only text prompts and no per-scene training.

Method	room	orchids	horns	fern	fortress	leaves	fork	truck	lego	Avg.
SPIn-NeRF [ 27 ]	95.6	92.7	92.8	94.3	97.7	94.9	87.9	85.2	74.9	90.67
SPIn-NeRF [ 27 ]	99.4	98.8	98.7	99.2	99.7	99.7	99.5	95.1	99.2	98.81
SA3D [ 4 ]	88.22	83.55	94.49	97.05	98.33	97.18	89.41	90.82	92.15	92.36
SA3D [ 4 ]	98.33	96.86	99.02	99.59	99.75	99.85	99.55	96.66	99.75	98.82
SAGA [ 3 ]	96.91	90.55	92.96	96.49	96.16	95.52	85.84	95.71	93.17	93.70
SAGA [ 3 ]	99.59	98.29	98.71	99.51	99.41	99.75	99.42	98.53	99.79	99.22

Appendix

Table 15: Per-scene quantitative results on SPIn-NeRF. The first row of each method corresponds to metric mIoU and the second row to metric mAcc. TrianguLang achieves competitive results (91.35% mIoU) using only text prompts without any per-scene optimization, approaching per-scene methods like SA3D (92.36%) while being orders of magnitude faster.

Panel of Figure 5 — Figure 5: Spatial disambiguation on the NVOS T-Rex scene. Top: The query “dino” (97.6% IoU) segments the dominant triceratops skull in the scene. Bottom: The query “leftmost dino” (95.8% IoU) leverages spatial reasoning to disambiguate between the two skulls, correctly selecting only the left specimen. The depth map (second column) provides the geometric context that enables this: TrianguLang computes 3D centroids for each candidate mask and selects the one satisfying the spatial constraint ( $\arg\min_{i}x_{i}$ for “leftmost”), resolving ambiguity that would be impossible with object names alone.

Figure 6: Segmentation results on the SPIn-NeRF room scene (90.7% mean IoU). Each row shows a different viewpoint: RGB input, DA3 depth estimate, ground truth mask, predicted mask, and overlay. When queried for “table,” TrianguLang produces a clean segmentation of the table surface without including the conference equipment (microphones) present in the ground truth annotation, demonstrating learned semantic boundaries rather than memorized annotation artifacts.

Panel of Figure 7 — Figure 7: TSDF mesh reconstructions from TrianguLang segmentations on ScanNet++ scenes. Left: “sofa chair,” Center: “coffee table,” Right: “TV.” Each mesh is extracted by fusing masked metric depth maps across 8 views using TSDF integration. The clean geometry demonstrates that TrianguLang produces view-consistent segmentations suitable for 3D reconstruction.

3D Visual Grounding

Three-dimensional visual grounding links language or visual queries to specific objects, parts, regions, and spatial relations in 3D scenes.

See all 134 papers

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models
Mar 18, 2026Kevin Qu, Haozhe Qi, Mihai Dusmanu +33D VLMs 3D Spatial Reasoning
Microsoft Spatial AI Lab · ETH Zurich · EPFL
Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs
May 19, 2026Xueying Jiang, Wenhao Li, Quanhao Qian +43D Object Detection 3D VLMs
1Nanyang Technological University · 2DAMO Academy, Alibaba Group · 3HuPan Lab +1
$L^3$ :Scene-agnostic Visual Localization in the Wild
Mar 9, 2026Yu Zhang, Muhua Zhu, Yifei Xue +2Robot Localization
Hunan University, Changsha, Hunan 410082, China

Mar 18, 2026J/K move · Enter open · S save

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

Kevin Qu, Haozhe Qi, Mihai Dusmanu, Mahdi Rad+2

Microsoft Spatial AI Lab · ETH Zurich · EPFL

3D VLMs 3D Spatial Reasoning 3D Scene Understanding 3D VQA 3D Visual Grounding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm

Abstract

3D Visual Grounding

Explore similar work

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs

L3L^3L3:Scene-agnostic Visual Localization in the Wild

$L^3$ :Scene-agnostic Visual Localization in the Wild