Current multi-view indoor 3D object detectors rely on sensor geometry that is costly to obtain (i.e., precisely calibrated multi-view camera poses) to fuse multi-view information into a global scene representation, limiting deployment in real-world scenes. We target a more practical setting: Sensor-Geometry-Free (SG-Free) multi-view indoor 3D object detection, where there are no sensor-provided geometric inputs (multi-view poses or depth). Recent Visual Geometry Grounded Transformer (VGGT) shows that strong 3D cues can be inferred directly from images. Building on this insight, we present VGGT-Det, the first framework tailored for SG-Free multi-view indoor 3D object detection. Rather than merely consuming VGGT predictions, our method integrates VGGT encoder into a transformer-based pipeline. To effectively leverage both the semantic and geometric priors from inside VGGT, we introduce two novel key components: (i) Attention-Guided Query Generation (AG): exploits VGGT attention maps as semantic priors to initialize object queries, improving localization by focusing on object regions while preserving global spatial structure; (ii) Query-Driven Feature Aggregation (QD): a learnable See-Query interacts with object queries to 'see' what they need, and then dynamically aggregates multi-level geometric features across VGGT layers that progressively lift 2D features into 3D. Experiments show that VGGT-Det significantly surpasses the best-performing method in the SG-Free setting by 4.4 and 8.6 [email protected] on ScanNet and ARKitScenes, respectively. Ablation study shows that VGGT's internally learned semantic and geometric priors can be effectively leveraged by our AG and QD.
Figures & tables
Figure 1: To achieve and improve Sensor-Geometry-Free (SG-Free) multi-view indoor 3D object detection, VGGT-Det effectively leverages the internal semantic and geometric priors from VGGT [ 41 ] , rather than merely consuming its predictions. VGGT-Det significantly surpasses competitive methods in the SG-Free setting.
Figure 2: Overview of the proposed Sensor-Geometry-Free (SG-Free) multi-view indoor 3D object detection framework, VGGT-Det . It is built upon the VGGT encoder [ 41 ] , which extracts 3D-aware features from multi-view images. The decoder processes a set of object queries that cross-attend to the extracted features, and iteratively updates the queries for final detection. To effectively leverage both the semantic and geometric priors from inside VGGT, we carefully design two key components: Attention-Guided Query Generation (AG) and Query-Driven Feature Aggregation (QD) . AG utilizes the semantic priors from the VGGT encoder’s attention to generate object queries, enabling these queries to focus on object regions while preserving the global spatial structure. Besides, QD introduces a learnable See-Query, which interacts with object queries via self-attention to ‘see’ their needs and dynamically aggregates multi-level geometric features accordingly.
Figure 3: Computation flow of Attention-Guided Query Generation.
Figure 4: The proposed Attention-Guided Query Generation (AG) is inspired by the interesting observation: attention maps from the VGGT encoder [ 41 ] exhibit a strong correlation with semantic content, even though VGGT is not explicitly trained for semantic tasks. For example, in the left column, object regions tend to receive higher attention weights. In the middle column, AG samples from VGGT-predicted point clouds under the guidance of attention weights and point distribution information. In the right column, compared to farthest point sampling without guidance (red points), the points sampled by AG (green points) are more concentrated in object regions (labeled by green boxes), resulting in more green points than red points in those areas. For clarity, we recommend viewing the figure in color and zooming in.
Methods
cab
bed
chair
sofa
tabl
door
wind
bkshf
pic
cntr
ImVoxelNet [ 34 ]
20.2
76.3
45.8
60.5
36.0
16.3
09.6
17.9
00.3
23.3
FCAF3D [ 33 ]
28.1
81.0
47.2
70.7
40.9
17.1
13.2
22.0
00.7
45.6
NeRF-Det [ 46 ]
24.9
85.3
54.4
64.3
42.2
24.3
12.7
31.0
01.0
39.3
MVSDet [ 47 ]
28.0
80.6
55.5
65.3
43.2
25.2
09.9
29.4
01.1
43.4
VGGT-Det (Basic backbone)
20.1
85.5
48.4
78.5
37.0
13.2
09.4
22.1
00.8
40.4
VGGT-Det (Basic backbone+AG)
25.9
84.7
53.7
73.8
46.3
17.6
13.7
26.4
02.2
40.5
Table 1: Comparison of [email protected] across different methods on ScanNet. To achieve Sensor-Geometry-Free (SG-Free) setting and ensure a fair comparison, FCAF3D is retrained with point clouds predicted by VGGT. ImVoxelNet, NeRF-Det and MVSDet are retrained with multi-view camera poses predicted by VGGT. Our method significantly outperforms the competitive methods.
Table 2: Ablation studies: (a) Effectiveness of the proposed designs. ‘BB’ denotes our basic backbone; (b) Performance across different numbers of input frames; (c) Comparison of different feature aggregation methods; (d) Time and memory analysis. VGGT computation is in the first row, while the additional time and memory consumption of each method are shown in subsequent rows.
Figure 5: Visualization of validation losses. In the left subfigure, after applying AG, the GIoU loss is significantly lower than that of the baseline backbone, indicating that AG effectively improves object localization during training. In the right subfigure, as See-Query progressively learns, within a few epochs, to interact with object queries and to aggregate encoded geometric features effectively, the loss for ‘AG+QD’ becomes significantly lower than that for ‘AG’, highlighting the effectiveness of the proposed QD strategy.
Figure 6: Qualitative comparison. Compared with MVSDet [ 47 ] , our VGGT-Det detects more objects with higher precision.
Table 3: Comparison of [email protected] on ARKitScenes. To achieve Sensor-Geometry-Free (SG-Free) setting and ensure a fair comparison, ImVoxelNet [ 34 ] , NeRF-Det [ 46 ] and MVSDet [ 47 ] are retrained with multi-view camera poses predicted by VGGT. Methods lacking both results and code support for ARKitScenes (e.g., FCAF3D) are not included. Our method significantly outperforms the competitive methods.
Table 4: Performance under different λdist in Eqn. ( 8 ).
Noise Level
FCAF3D
VGGT-Det (Ours)
0.30
0.0
34.1
0.20
0.0
39.4
0.10
0.0
44.5
0.01
18.7
47.0
0.005
33.4
47.0
0.001
39.7
46.9
Table 5: Robustness under different noise levels: comparison between FCAF3D [ 33 ] and our VGGT-Det.
# of Frames
20
40
60
80
100
MVSDet [ 47 ]
35.9
39.7
40.9
42.5
42.5
Ours
42.6
45.3
46.2
46.9
47.3
Table 6: Performance across different numbers of input frames
Method
M1
M2
M3
Ours
mAP
35.2
41.2
42.5
46.9
Method
M1+SG
M2+SG
M3+SG
Ours+SG
mAP
48.1
49.5
56.2
58.8
Table 7: Performance gap to methods that leverage sensor geometry.
Method
VGGT
+M1
+M2
+M3
+Ours
T (s)
0.68
0.09
0.07
0.21
0.23
M (GB)
11.71
12.95
6.76
13.81
3.57
Table 8: Efficiency comparison with different mothods in SG-Free setting with 40 input frames.
# of Frames
20
40
60
80
100
T (s)
0.49
0.91
1.48
2.20
3.06
M (GB)
14.46
15.28
17.50
19.48
21.66
Table 9: Efficiency comparison under different input numbers.
Figure 7: Visualization of attention and generated object query positions. Compared to farthest point sampling without guidance (red points), the points sampled by AG (green points) are more concentrated in object regions (labeled by green boxes), resulting in more green points than red points in those areas. For clarity, we recommend viewing the figure in color and zooming in.
Figure 8: Qualitative comparison with MVSDet [ 47 ] . To achieve the Sensor-Geometry-Free (SG-Free) setting and ensure a fair comparison, MVSDet is trained with multi-view poses predicted by VGGT [ 41 ] . The mesh here is not utilized in the methods and is only for visualization.