RGB-D 3D instance detectors benefit from visual semantics, but the task-specific Faster R-CNN/ResNet branch used by IIFNet3D couples feature extraction to a separately trained 2D detector and its image-domain labels. Replacing that branch with a frozen vision foundation model removes this task-specific dependency, but may introduce occlusion noise and a mismatch between patch features and geometry-aware detection features. In this work, we investigate this replacement through an adaptation of DINOv3 to the instance-level fusion pipeline of IIFNet3D. At the core of our approach is a depth-consistent feature pipeline that projects scene points into calibrated RGB-D frames, applies a metric depth-residual check, averages the accepted DINOv3 features into an offline point cache, and aggregates the cached features inside proposal-aligned RoI grids. The geometric and bidirectional instance-fusion paths are preserved, while Conservative VAID is evaluated as a low-strength, support-weighted semantic distillation recipe applied only to positive RoIs. We conduct extensive evaluations on ScanNetV2 to assess the proposed transfer recipes. On ScanNetV2, our DINOv3 control achieves mAP scores of 76.15 and 60.93 at IoU thresholds of 0.25 and 0.50, respectively. The Conservative VAID setting achieves mAP scores of 76.59 and 62.16, corresponding to numerical gains of 0.44 and 1.23 points over the control, respectively, in this checkpoint-level recipe comparison. The reported IIFNet3D result of 75.7/63.8 is used only as an external reference because the visual branch and processing protocol differ. Accordingly, we interpret these results as evidence for a controlled transfer recipe rather than as a causal estimate of the individual contributions of VAID or depth filtering.
Figures & tables
Fig. 1: Overview of DIFTA-3D. The geometric path generates 3D proposals, while projected DINOv3 features are filtered by a depth-residual check before RoI aggregation and bidirectional instance fusion. Conservative VAID uses detached teacher features, visibility weighting, and positive-RoI supervision.
Fig. 2: Depth-consistent point-feature construction. Each 3D point is projected independently into calibrated RGB and depth cameras. Frozen DINOv3 patches provide the color feature, while the metric depth sample checks d>0.05m and the residual ∣z−d∣≤0.25m ; foreground observations are retained, and occluded background observations are rejected. The comparison shows how depth filtering prevents contaminated features from being aggregated. Accepted observations are averaged into a 384-dimensional offline point cache, with an explicit zero feature assigned when no view is valid. A proposal-aligned 7×7×7 grid queries this cache, and RoI-Conv pooling produces the instance descriptor qi .
Fig. 3: Qualitative Detection Results on SUN-RGBD. Representative validation scenes are arranged by column, with rows showing the input image, Ground Truth, and our DINOv3 predictions. Colors identify object instances/classes, and the visualization is qualitative.
Fig. 4: Qualitative Results on the ScanNet V2 Validation Set. Representative scenes are arranged by column, with rows showing the input point cloud, Ground Truth, and our DINOv3 predictions. The same camera convention is used for the Ground Truth and prediction rows; these views are qualitative and are not used as quantitative evidence.
Fig. 5: Fast-LIVO2 acquisition [ 15 ] and input. Fast-LIVO2-fused indoor point cloud obtained from the Livox Avia–camera–Jetson Orin NX sensing platform for the real-scene diagnostic.
Fig. 6: Qualitative 3D Detection with Fast-LIVO2 Pointcloud Inputs [ 15 ] . Qualitative 3D box predictions from point-cloud inputs obtained from Fast-LIVO2 in an indoor environment.
3D instance segmentation methods typically rely on high-quality point clouds or posed RGB-D scans, requiring complex multi-stage processing pipelines, and are highly sensitive to reconstruction noise. While recent feed-forward transformers have revolutionized multi-view 3D reconstruction, they remain decoupled from high-level semantic understanding. In this work, we present SegVGGT, a unified end-to-end framework that simultaneously performs feed-forward 3D reconstruction and instance segmentation directly from multi-view RGB images. By introducing object queries that interact with multi-level geometric features, our method deeply integrates instance identification into the visual geometry grounded transformer. To address the severe attention dispersion problem caused by the massive number of global image tokens, we propose the Frame-level Attention Distribution Alignment (FADA) strategy. FADA explicitly guides object queries to attend to instance-relevant frames during training, providing structured supervision without extra inference overhead. Extensive experiments demonstrate that SegVGGT achieves the state-of-the-art performance on ScanNetv2 and ScanNet200, outperforming both recent joint models and RGB-D-based approaches, while exhibiting strong generalization capabilities on ScanNet++.
Jinyuan Qu, Hongyang Li, Lei Zhang
Tsinghua University · International Digital Economy Academy (IDEA) · South China University of Technology
Monocular depth estimation (MDE) is a fundamental yet inherently ill-posed task. Recent vision foundation models (VFMs), particularly DINO-based transformers, have significantly improved accuracy and generalization for dense prediction. Prior works generally follow a unified paradigm: sampling a fixed set of intermediate transformer layers at uniform intervals to build multi-scale features. This common practice implicitly assumes that geometric information is uniformly distributed across layers, which may underutilize the structural 3D cues encoded in VFMs. In this study, we present a systematic layer-wise analysis of DINOv3, revealing that 3D information is distributed non-uniformly: deeper layers exhibit stronger depth predictability and better capture inter-sample geometric variation. Motivated by this, we introduce a Last-Layer-Centric Feature Recombination (LFR) module to enhance geometric expressiveness. LFR treats the final layer as a geometric anchor and adaptively selects complementary intermediate layers according to a minimal-similarity criterion. Selected features are fused with the last-layer representation via compact linear adapters.Extensive experiments show that LFR module consistently improves MDE accuracy and achieves state-of-the-art performance. Our analysis sheds light on how geometric knowledge is organized within VFMs and offers an efficient strategy for unlocking their potential in dense 3D tasks.
Gongshu Wang, Zhirui Wang, Kan Yang
Aerospace Information Research Institute Chinese Academy of Sciences Beijing, China
3D instance segmentation requires representations that are both geometrically structured and semantically discriminative. Pure 3D encoders model spatial structure well but often lack strong appearance semantics, while directly lifting 2D priors into 3D by concatenation injects rich semantics without ensuring that they align well with the underlying 3D structure. In this paper, we address this problem by formulating 2D-to-3D semantic transfer as a masked semantic reconstruction task. Specifically, we mask a subset of background lifted 2D priors and train the 3D encoder to recover the missing semantic priors, providing explicit supervision for distilling rich 2D semantics into the 3D backbone. We further propose geometry-modulated semantic injection, which uses raw 3D geometry to modulate lifted 2D priors before sparse encoding, so that geometry explicitly controls semantic injection and alleviates the distribution gap between 2D semantic features and 3D geometric features. Experiments on two baseline frameworks demonstrate that the proposed method achieves state-of-the-art results in both full-data and few-shot settings on ScanNetV2 and ScanNet200. Code is publicly available at https://github.com/yharoldsu0627/Geometry_Guided_Semantic_Reconstruction
Yuanhao Su, Shaofeng Zhang
University of Science and Technology of China · Fuzhou University