Localizing objects and parts from natural language in 3D space is essential for robotics, AR, and embodied AI, yet existing methods face a trade-off between the accuracy and geometric consistency of per-scene optimization and the efficiency of feed-forward inference. We present TrianguLang, a feed-forward framework for 3D localization that requires no camera calibration at inference. Unlike prior methods that treat views independently, we introduce Geometry-Aware Semantic Attention (GASA), which utilizes predicted geometry to gate cross-view feature correspondence, suppressing semantically-plausible but geometrically-inconsistent matches without requiring ground-truth poses. Validated on five benchmarks including ScanNet++ and uCO3D, TrianguLang achieves state-of-the-art feed-forward text-guided segmentation and localization, reducing user effort from O(N) clicks to a single text query. The model processes each frame at 1008x1008 resolution in ∼57ms (∼18 FPS) without optimization, enabling practical deployment for interactive robotics and AR applications. Code and checkpoints are available at https://cwru-aism.github.io/triangulang/.
Figures & tables
Figure 1: Overview of the TrianguLang architecture
Figure 2: Overview of the GASA decoder.
Figure 3: Performance on uCO3D and ScanNet++ datasets. Left-to-right: RGB, depth map, ground truth, TrianguLang masks
Model
Setting
Train → Eval
mIoU ( ↑ )
mAcc ( ↑ )
MV-SAM
In-domain
ScanNet++ → ScanNet++
0.510
0.694
In-domain
uCo3D → uCo3D
0.910
0.965
Cross-domain
uCo3D → ScanNet++
0.194
0.251
Cross-domain
ScanNet++ → uCo3D
0.322
0.517
Large-scale
SA-1B → ScanNet++
0.489
0.635
Large-scale
SA-1B → uCo3D
0.877
0.950
Table 1: In-domain, cross-dataset, and large-scale evaluation results. MV-SAM numbers from [ 13 ] . TrianguLang uses text-only prompts; MV-SAM uses 12 click prompts per object. SA-1B denotes MV-SAM trained on single-view SA-1B data (millions of images).
Method
mIoU
Loc. Acc.
Per-scene
Time
LERF [ 17 ]
37.4
73.6
✓
∼ 45 min
LangSplat [ 32 ]
51.4
84.3
✓
∼ 10 min
LangSplat-V2 [ 22 ]
59.9
84.1
✓
∼ 10 min
TrianguLang †
59.2
89.1
✗
∼ 58ms
Table 2: Comparison with per-scene language grounding methods on the LERF-OVS benchmark. Per-scene methods require calibrated poses and scene reconstruction. TrianguLang operates feed-forward using ScanNet++-trained weights without any training on LERF scenes.
Figure 4: Qualitative comparison on LERF-OVS scenes using uniform clip thresholds. Row 1: “toaster” query: LERF and LangSplatV2 produce diffuse activations across the scene while TrianguLang tightly focuses its relevancy map on the target object. Row 3: “stripes” query: TrianguLang achieves precise localization despite not training on this dataset, and runs 3 orders of magnitude faster ( ∼ 58ms vs. 10 to 45 min).
Method
Localization Accuracy (%)
Semantic Segmentation (mIoU %)
Ramen
Teatime
Kitchen
Figurines
Overall
Ramen
Teatime
Kitchen
Figurines
Overall
Gaussian Grouping [ 45 ]
32.4
69.5
50.0
44.6
49.1
26.4
54.0
31.3
34.6
36.6
LEGaussians [ 38 ]
69.0
79.7
63.6
57.1
67.4
20.2
32.3
22.3
23.4
24.6
LERF [ 17 ]
62.0
84.8
72.7
75.0
73.6
28.2
45.0
37.9
38.6
37.4
GOI [ 33 ]
56.3
67.8
68.2
44.6
59.2
33.7
55.8
54.5
23.9
42.0
GAGS [ 31 ]
69.0
88.1
90.9
78.6
81.7
46.8
60.3
55.8
53.6
54.1
Table 3: Per-scene comparison on LERF-OVS benchmark. TrianguLang uses ScanNet++-trained weights without any LERF training (zero-shot transfer). Per-scene methods require 10 to 45 minutes of optimization per scene.
Configuration
mIoU
Δ
Baseline (GASA + World PE)
54.7
−
− GASA kernel (standard attn)
49.4
− 5.3
− World-space PE
49.3
− 5.4
− Both (no GASA, no PE)
46.3
− 8.4
RBF kernel (vs. learned MLP)
44.0
− 10.7
Table 4: Component ablation on ScanNet++ (230 scenes, 8 views, 504 resolution). Δ measured against ablation baseline (54.7% mIoU). The full model in Table 1 achieves 62.4% at 1008 resolution with 230 scenes.
VGM
ScanNet++
uCO3D
LERF-OVS
DA3-NESTED [ 23 ]
62.4
72.1
59.2
MapAnything [ 15 ]
63.0
72.1
58.1
π3 -VO [ 42 ]
62.7
66.8
51.8
Table 5: Sensitivity to visual geometry backbone. All models use the same GASA decoder architecture trained on ScanNet++ and evaluated zero-shot on uCO3D and LERF-OVS. mIoU (%) reported.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Qualifier
Synonyms
Selection Criterion
Base qualifiers (7 learnable embeddings)
Depth
nearest
closest, close
argminidi
farthest
far, distant
argmaxidi
Horizontal
leftmost
left
argminixi
rightmost
right
argmaxixi
Vertical
topmost
top, upper
argminiyi
Appendix
Table 6: Spatial qualifier vocabulary (48 recognized phrases across 7 base categories). Index 0 is reserved for queries without spatial qualifiers. Ordinal and size-based extensions share learnable embedding indices with their related base qualifiers.
Configuration
mIoU
No spatial features
40.07
Spatial-as-points
43.54
Object-aware selection
47.01
Appendix
Table 7: Spatial feature ablation on ScanNet++. Object-aware selection achieves the best segmentation quality.
Hyperparameter
Value
Data
Scenes
230
Views per sample
8
Image resolution
1008
DA3 resolution
1008
Optimization
Appendix
Table 8: Training hyperparameters for TrianguLang.
Component
Parameters
Trainable
SAM3 Backbone
848M
Frozen
DA3 Depth (NESTED-GIANT-LARGE)
1.4B
Frozen
GASA Decoder
13.4M
✓
Query embeddings
26K
✓
Self-attention layers
1.2M
✓
Text cross-attention
1.2M
✓
Appendix
Table 9: Model parameter breakdown.
Method
Queries
Predicted
Oracle
Gap
SAM3 [ 2 ]
∼ 200
49.8
79.9
30.1
TrianguLang
10
62.4
63.4
1.0
Appendix
Table 10: Oracle vs. predicted mIoU on ScanNet++ (49 val scenes, 100 frames/scene). Oracle selects the GT-best mask among candidates. The gap quantifies the mask selection bottleneck.
Config
GPUs
Time/epoch
Memory
Inference
Full (230s, 8v)
8 × A100
∼ 10 min
45 GB
∼ 107ms
Appendix
Table 11: Training and inference computational cost.
Type
Method
NVOS
SPIn-NeRF
Feed-forward
SAM2-Video
88.7
86.6
MV-SAM
92.1
92.5
TrianguLang
93.5
91.4
Per-scene optimization
SPIn-NeRF
−
90.7
SA3D
91.1
92.4
SAGA
92.6
93.7
Appendix
Table 12: Comparison on NVOS and SPIn-NeRF benchmarks (mIoU). Per-scene optimization methods require calibrated poses and 30 to 60 minutes of training per scene. TrianguLang and MV-SAM are feed-forward methods requiring no per-scene computation.
Method
ScanNet++
uCo3D
mIoU ( ↑ )
mAcc ( ↑ )
mIoU ( ↑ )
mAcc ( ↑ )
Prompt Projection †
29.2
59.2
78.2
83.3
SAM2-Video [ 35 ]
46.1
61.4
81.9
91.3
SAM2-Long [ 9 ]
41.5
61.4
72.9
86.4
SAM3 [ 2 ]
48.6
63.4
82.4
91.4
MV-SAM
48.9
63.5
87.7
95.0
Appendix
Table 13: Comparison of mIoU and mAcc on ScanNet++ and uCo3D. All baselines use their respective SA-1B-trained weights. Prompt Projection † unprojects prompts into 3D and re-projects per view to run SAM2 independently. SAM3 alone (no cross-view reasoning) underperforms MV-SAM, confirming that TrianguLang’s gains come from GASA, not the backbone.
Method
fern
flower
fortress
horns_center
horns_left
leaves
trex
Avg.
SA3D [ 4 ]
82.90
94.60
98.30
96.20
90.20
93.20
81.99
91.06
94.39
98.74
99.68
99.33
99.36
99.57
97.41
98.35
SAGA [ 3 ]
83.53
96.62
98.16
98.06
93.59
93.51
80.81
92.57
94.60
99.17
99.65
99.50
99.51
99.59
97.26
98.55
OmniSeg3D [ 47 ]
82.70
95.30
98.50
97.70
95.60
92.70
87.40
92.84
94.30
98.90
99.70
99.60
99.70
99.50
98.30
98.57
Appendix
Table 14: Per-scene quantitative results on NVOS. The first row of each method corresponds to metric mIoU and the second row to metric mAcc. TrianguLang achieves state-of-the-art average mIoU (93.46%) among feed-forward methods, approaching per-scene optimization methods while using only text prompts and no per-scene training.
Method
room
orchids
horns
fern
fortress
leaves
fork
truck
lego
Avg.
SPIn-NeRF [ 27 ]
95.6
92.7
92.8
94.3
97.7
94.9
87.9
85.2
74.9
90.67
99.4
98.8
98.7
99.2
99.7
99.7
99.5
95.1
99.2
98.81
SA3D [ 4 ]
88.22
83.55
94.49
97.05
98.33
97.18
89.41
90.82
92.15
92.36
98.33
96.86
99.02
99.59
99.75
99.85
99.55
96.66
99.75
98.82
SAGA [ 3 ]
96.91
90.55
92.96
96.49
96.16
95.52
85.84
95.71
93.17
93.70
99.59
98.29
98.71
99.51
99.41
99.75
99.42
98.53
99.79
99.22
Appendix
Table 15: Per-scene quantitative results on SPIn-NeRF. The first row of each method corresponds to metric mIoU and the second row to metric mAcc. TrianguLang achieves competitive results (91.35% mIoU) using only text prompts without any per-scene optimization, approaching per-scene methods like SA3D (92.36%) while being orders of magnitude faster.
Figure 5: Spatial disambiguation on the NVOS T-Rex scene. Top: The query “dino” (97.6% IoU) segments the dominant triceratops skull in the scene. Bottom: The query “leftmost dino” (95.8% IoU) leverages spatial reasoning to disambiguate between the two skulls, correctly selecting only the left specimen. The depth map (second column) provides the geometric context that enables this: TrianguLang computes 3D centroids for each candidate mask and selects the one satisfying the spatial constraint ( argminixi for “leftmost”), resolving ambiguity that would be impossible with object names alone.
Figure 6: Segmentation results on the SPIn-NeRF room scene (90.7% mean IoU). Each row shows a different viewpoint: RGB input, DA3 depth estimate, ground truth mask, predicted mask, and overlay. When queried for “table,” TrianguLang produces a clean segmentation of the table surface without including the conference equipment (microphones) present in the ground truth annotation, demonstrating learned semantic boundaries rather than memorized annotation artifacts.
Figure 7: TSDF mesh reconstructions from TrianguLang segmentations on ScanNet++ scenes. Left: “sofa chair,” Center: “coffee table,” Right: “TV.” Each mesh is extracted by fusing masked metric depth maps across 8 views using TSDF integration. The clean geometry demonstrates that TrianguLang produces view-consistent segmentations suitable for 3D reconstruction.
Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm
3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly), both preventing camera information from being deterministically propagated into the prediction. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation X^=(uc−cx)Zˉ/fx explicitly in Chain-of-Thought (CoT), and substitutes tool outputs into the formula before regressing the final 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from 0.5× to 1.5×, our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.
Xueying Jiang, Wenhao Li, Quanhao Qian +4
1Nanyang Technological University · 2DAMO Academy, Alibaba Group · 3HuPan Lab +1
Standard visual localization methods typically require offline pre-processing of scenes to obtain 3D structural information for better performance. This inevitably introduces additional computational and time costs, as well as the overhead of storing scene representations. Can we visually localize in a wild scene without any off-line preprocessing step? In this paper, we leverage the online inference capabilities of feed-forward 3D reconstruction networks to propose a novel map-free visual localization framework L3. Specifically, by performing direct online 3D reconstruction on RGB images, followed by two-stage metric scale recovery and pose refinement based on 2D-3D correspondences, L3 achieves high accuracy without the need to pre-build or store any offline scene representations. Extensive experiments demonstrate L3 not only that the performance is comparable to state-of-the-art solutions on various benchmarks, but also that it exhibits significantly superior robustness in sparse scenes (fewer reference images per scene).