Current multi-view indoor 3D object detectors rely on sensor geometry that is costly to obtain (i.e., precisely calibrated multi-view camera poses) to fuse multi-view information into a global scene representation, limiting deployment in real-world scenes. We target a more practical setting: Sensor-Geometry-Free (SG-Free) multi-view indoor 3D object detection, where there are no sensor-provided geometric inputs (multi-view poses or depth). Recent Visual Geometry Grounded Transformer (VGGT) shows that strong 3D cues can be inferred directly from images. Building on this insight, we present VGGT-Det, the first framework tailored for SG-Free multi-view indoor 3D object detection. Rather than merely consuming VGGT predictions, our method integrates VGGT encoder into a transformer-based pipeline. To effectively leverage both the semantic and geometric priors from inside VGGT, we introduce two novel key components: (i) Attention-Guided Query Generation (AG): exploits VGGT attention maps as semantic priors to initialize object queries, improving localization by focusing on object regions while preserving global spatial structure; (ii) Query-Driven Feature Aggregation (QD): a learnable See-Query interacts with object queries to 'see' what they need, and then dynamically aggregates multi-level geometric features across VGGT layers that progressively lift 2D features into 3D. Experiments show that VGGT-Det significantly surpasses the best-performing method in the SG-Free setting by 4.4 and 8.6 [email protected] on ScanNet and ARKitScenes, respectively. Ablation study shows that VGGT's internally learned semantic and geometric priors can be effectively leveraged by our AG and QD.
Figures & tables
Figure 1: To achieve and improve Sensor-Geometry-Free (SG-Free) multi-view indoor 3D object detection, VGGT-Det effectively leverages the internal semantic and geometric priors from VGGT [ 41 ] , rather than merely consuming its predictions. VGGT-Det significantly surpasses competitive methods in the SG-Free setting.
Figure 2: Overview of the proposed Sensor-Geometry-Free (SG-Free) multi-view indoor 3D object detection framework, VGGT-Det . It is built upon the VGGT encoder [ 41 ] , which extracts 3D-aware features from multi-view images. The decoder processes a set of object queries that cross-attend to the extracted features, and iteratively updates the queries for final detection. To effectively leverage both the semantic and geometric priors from inside VGGT, we carefully design two key components: Attention-Guided Query Generation (AG) and Query-Driven Feature Aggregation (QD) . AG utilizes the semantic priors from the VGGT encoder’s attention to generate object queries, enabling these queries to focus on object regions while preserving the global spatial structure. Besides, QD introduces a learnable See-Query, which interacts with object queries via self-attention to ‘see’ their needs and dynamically aggregates multi-level geometric features accordingly.
Figure 3: Computation flow of Attention-Guided Query Generation.
Figure 4: The proposed Attention-Guided Query Generation (AG) is inspired by the interesting observation: attention maps from the VGGT encoder [ 41 ] exhibit a strong correlation with semantic content, even though VGGT is not explicitly trained for semantic tasks. For example, in the left column, object regions tend to receive higher attention weights. In the middle column, AG samples from VGGT-predicted point clouds under the guidance of attention weights and point distribution information. In the right column, compared to farthest point sampling without guidance (red points), the points sampled by AG (green points) are more concentrated in object regions (labeled by green boxes), resulting in more green points than red points in those areas. For clarity, we recommend viewing the figure in color and zooming in.
Methods
cab
bed
chair
sofa
tabl
door
wind
bkshf
pic
cntr
ImVoxelNet [ 34 ]
20.2
76.3
45.8
60.5
36.0
16.3
09.6
17.9
00.3
23.3
FCAF3D [ 33 ]
28.1
81.0
47.2
70.7
40.9
17.1
13.2
22.0
00.7
45.6
NeRF-Det [ 46 ]
24.9
85.3
54.4
64.3
42.2
24.3
12.7
31.0
01.0
39.3
MVSDet [ 47 ]
28.0
80.6
55.5
65.3
43.2
25.2
09.9
29.4
01.1
43.4
VGGT-Det (Basic backbone)
20.1
85.5
48.4
78.5
37.0
13.2
09.4
22.1
00.8
40.4
VGGT-Det (Basic backbone+AG)
25.9
84.7
53.7
73.8
46.3
17.6
13.7
26.4
02.2
40.5
Table 1: Comparison of [email protected] across different methods on ScanNet. To achieve Sensor-Geometry-Free (SG-Free) setting and ensure a fair comparison, FCAF3D is retrained with point clouds predicted by VGGT. ImVoxelNet, NeRF-Det and MVSDet are retrained with multi-view camera poses predicted by VGGT. Our method significantly outperforms the competitive methods.
Table 2: Ablation studies: (a) Effectiveness of the proposed designs. ‘BB’ denotes our basic backbone; (b) Performance across different numbers of input frames; (c) Comparison of different feature aggregation methods; (d) Time and memory analysis. VGGT computation is in the first row, while the additional time and memory consumption of each method are shown in subsequent rows.
Figure 5: Visualization of validation losses. In the left subfigure, after applying AG, the GIoU loss is significantly lower than that of the baseline backbone, indicating that AG effectively improves object localization during training. In the right subfigure, as See-Query progressively learns, within a few epochs, to interact with object queries and to aggregate encoded geometric features effectively, the loss for ‘AG+QD’ becomes significantly lower than that for ‘AG’, highlighting the effectiveness of the proposed QD strategy.
Figure 6: Qualitative comparison. Compared with MVSDet [ 47 ] , our VGGT-Det detects more objects with higher precision.
Table 3: Comparison of [email protected] on ARKitScenes. To achieve Sensor-Geometry-Free (SG-Free) setting and ensure a fair comparison, ImVoxelNet [ 34 ] , NeRF-Det [ 46 ] and MVSDet [ 47 ] are retrained with multi-view camera poses predicted by VGGT. Methods lacking both results and code support for ARKitScenes (e.g., FCAF3D) are not included. Our method significantly outperforms the competitive methods.
Table 4: Performance under different λdist in Eqn. ( 8 ).
Noise Level
FCAF3D
VGGT-Det (Ours)
0.30
0.0
34.1
0.20
0.0
39.4
0.10
0.0
44.5
0.01
18.7
47.0
0.005
33.4
47.0
0.001
39.7
46.9
Table 5: Robustness under different noise levels: comparison between FCAF3D [ 33 ] and our VGGT-Det.
# of Frames
20
40
60
80
100
MVSDet [ 47 ]
35.9
39.7
40.9
42.5
42.5
Ours
42.6
45.3
46.2
46.9
47.3
Table 6: Performance across different numbers of input frames
Method
M1
M2
M3
Ours
mAP
35.2
41.2
42.5
46.9
Method
M1+SG
M2+SG
M3+SG
Ours+SG
mAP
48.1
49.5
56.2
58.8
Table 7: Performance gap to methods that leverage sensor geometry.
Method
VGGT
+M1
+M2
+M3
+Ours
T (s)
0.68
0.09
0.07
0.21
0.23
M (GB)
11.71
12.95
6.76
13.81
3.57
Table 8: Efficiency comparison with different mothods in SG-Free setting with 40 input frames.
# of Frames
20
40
60
80
100
T (s)
0.49
0.91
1.48
2.20
3.06
M (GB)
14.46
15.28
17.50
19.48
21.66
Table 9: Efficiency comparison under different input numbers.
Figure 7: Visualization of attention and generated object query positions. Compared to farthest point sampling without guidance (red points), the points sampled by AG (green points) are more concentrated in object regions (labeled by green boxes), resulting in more green points than red points in those areas. For clarity, we recommend viewing the figure in color and zooming in.
Figure 8: Qualitative comparison with MVSDet [ 47 ] . To achieve the Sensor-Geometry-Free (SG-Free) setting and ensure a fair comparison, MVSDet is trained with multi-view poses predicted by VGGT [ 41 ] . The mesh here is not utilized in the methods and is only for visualization.
Feed-forward geometry models such as VGGT recover camera parameters, depth maps, and point maps from one to hundreds of views in a single forward pass, making them powerful backbones for multi-view 3D perception. However, VGGT's global attention is quadratic in the number of frames and dominates end-to-end runtime on long sequences, which has motivated a range of training-free efficiency methods such as token merging. These methods, however, come with a drop in accuracy that makes them hard to use in place of VGGT. We propose GPA-VGGT, a training-free acceleration method that replaces global attention in early blocks with frame attention augmented by a shared cross-frame K/V set. In the remaining blocks, we merge only keys and values while preserving all queries, restore the attention mass of the tokens each merged key represents, and randomly project the matching metric to 128 dimensions to reduce matching cost. Across diverse multi-view geometry tasks, GPA-VGGT provides training-free acceleration at speedups comparable to prior work while preserving the information required for precise geometric estimation, retaining over 99% of VGGT's pose accuracy under AUC@3.
Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame attention limits its scalability. Existing training-free accelerators reduce computation uniformly along one axis, missing layer heterogeneity. Our spectral, probing, and causal analyses reveal three regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry yet their cross-frame attention remains essential for pose. RegimeVGGT applies layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. Training-free, RegimeVGGT achieves a 6.7x speedup over VGGT* at matched reconstruction quality.
Jinhao You, Shuo Lyu, Zhuohang Lyu +5
University of Pennsylvania · University of California, Irvine · Nanyang Technological University +1
Recent advances in 3D reconstruction have progressed from per-scene optimization to feed-forward inference, and semantic scene understanding has followed suit -- yet existing methods remain confined to object-centric perception, neglecting spatial relations between objects. We formulate 3D spatial relation segmentation in a feed-forward, pose-free multi-view setting: given a visually specified subject and a relational text query, the model segments the target across views without receiving its category name. To this end, we propose RelationVGGT, a novel feed-forward framework that integrates semantic features from a visual foundation model with geometry-aware representations from a 3D geometry foundation model and leverages a relation transformer for subject-conditioned, cross-view relation prediction -- requiring neither per-scene optimization nor known camera poses. We additionally provide a fully automated annotation pipeline built on ScanNet++ with VLMs and LLMs, enabling scalable training data generation for this new task.