Domain Adaptation of SAM 2.1 for Building-Component Segmentation in Image-Based 3D Reconstruction

Ehsan Agha Ebrahimi; Yong-Cheol Lee; Keith G. Mills

cs.CVOct 8, 2026

Domain Adaptation of SAM 2.1 for Building-Component Segmentation in Image-Based 3D Reconstruction

Authors: Ehsan Agha Ebrahimi, Yong-Cheol Lee, Keith G. Mills

Organizations: LSU Dept. Const. Mgmt. · LSU Div. CS&E

Abstract

3D reconstruction of building components using image-based scanning requires appropriate segmentation of the objects in the 2D building images. In order to benefit from consistent grouping of the 2D masks in 3D environments, vision foundation models such as the Segment Anything Model have been utilized by Group Anything with Radiance Fields (GARField) to perform 3D segmentation. This study fine-tuned SAM 2.1 on the HBD dataset, consisting of 2,679 annotated images from 42 buildings, to enhance multi-class segmentation of 2D building images, covering 11 classes (Wall, Window, Ceiling, Door, Column, Beam, Floor, Opening, Curtain Wall, Roof, and Lift). The model was evaluated using per-class IoU, Dice, and mIoU metrics, and the results show an improvement in mIoU from 49.9% to 69.6%, indicating a meaningful improvement in class segmentation and providing a more accurate model for distilling the masks in 3D. SegFormer was then used as a baseline, generating per-class masks for comparison with SAM 2.1, which indicates improvement in some object classes with larger surface area. The masks in the building images were grouped in the neural radiance field, and the pretrained SAM 2.1 was compared with the fine-tuned SAM 2.1 in the newly rendered images. This comparison shows that fine-tuning improves the grouping of building components at a user-selected grouping scale. This advances the application of Group Anything with Radiance Fields (GARField) in 3D, enabling better segmentation of building structures for 3D auto-reconstruction.

Figures & tables

Panel of Figure 1 — Figure 1: Patrick F. Taylor Hall reconstructed in 3D from drone images. Top: rendered photo view. Bottom: the same view segmented into building components with our fine-tuned SAM 2.1 and grouped hierarchically in 3D by Group Anything with Radiance Fields (GARField) [ 7 ] , with each 3D group shown in a distinct color.

Figure 2: The SAM 2 architecture (figure adopted from Ravi et al. [13] ).

Figure 3: Center-click prompt selection. For each ground-truth component mask we compute the Euclidean distance of every interior pixel to the mask boundary (b) and take the pixel of maximum distance, $p^{\ast}$ (red star) as a single positive click for SAM 2. This deepest point always lies inside the object, even for ring-shaped components where the centroid ( $\times$ ) falls outside the mask (a).

Panel of Figure 4 — Figure 4: Training loss convergence during SAM 2 Hiera-Large fine-tuning on the HBD building facade dataset over 40 epochs. The total loss decreased by 55.9% from 1.864 to a minimum of 0.823 at epoch 31, demonstrating successful adaptation to the architectural domain. The model was trained on 2,679 images using 4 $\times$ NVIDIA A100 GPUs for 18 hours.

Panel of Figure 5 — Figure 5: Comparison of automatic mask generation on Patrick F. Taylor Hall. The fine-tuned model (a) produces more semantically meaningful segmentation of architectural elements, with clearer separation of the building facade, windows, and ground plane compared to the pre-trained model (b).

Panel of Figure 6 — Figure 6: SegFormer validation loss per epoch during fine-tuning. The minimum is at epoch 6 (marked); validation loss rises afterwards, so the epoch-6 checkpoint is retained. The training loss (not shown) continues to fall to ${\sim}0.03$ , indicating overfitting.

		IoU (%)			Dice (%)
Class	#Obj	Pre.	FT	Seg. †	Pre.	FT	Seg. †
Wall	1679	35.3	53.0	83.2	44.6	62.4	90.8
Window	1331	54.3	71.3	70.6	63.0	78.8	82.8
Ceiling	673	44.6	65.2	79.9	54.5	74.0	88.8
Door	474	51.8	86.5	83.1	61.3	91.4	90.7
Column	383	62.6	71.7	49.1	71.0	80.6	65.9

Table 1: Per-class segmentation accuracy on the HBD test set (349 images, 5,541 annotated instances). Pre-trained SAM 2.1 (Pre.) and our fine-tuned SAM 2.1 (FT) are both prompted by a single center click derived from the ground-truth mask and scored per object. SegFormer (Seg.) is a fully-automatic semantic segmenter scored per pixel, reported for reference. Bold marks the better of the two matched SAM variants; SegFormer uses a different protocol and is not a strict head-to-head. Scores are macro-averaged over classes.

Panel of Figure 7 — Figure 7: Qualitative 3D grouping on the Patrick F. Taylor Hall drone scene, from a fixed viewpoint (camera 15), at two grouping scales. Each row shows, left to right, the input photograph, the reconstructed RGB, and the GARField grouping field visualized by projecting its per-point features to RGB with PCA. Rows (a,b) use grouping scale 0.15 and rows (c,d) use scale 0.05; (a,c) use pretrained SAM 2.1 masks and (b,d) use our fine-tuned SAM 2.1 masks. Both models are trained with an identical configuration and differ only in the SAM 2.1 checkpoint. Because the PCA projection is computed independently per image, absolute colors are not comparable across rows; we compare the spatial coherence of the groups.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

SAMRI-3D: Adapting SAM2 for 3D MRI Segmentation with Global Volume Tokens
Jul 20, 2026Zhao Wang, Wei Dai, Hongfu Sun +2Medical Imaging Foundation Models 3D Medical Image Segmentation
School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, Australia · School of Engineering, College of Engineering, Science and Environment, University of Newcastle, Australia · School of Human Movement and Nutrition Sciences, The University of Queensland, Australia
TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision
Oct 10, 2026Hao Zhu, Heyu Sun, Shuo Jin +2Promptable Image Segmentation Open-Vocabulary Semantic Segmentation
Automatic Segmentation of 3D CT scans with SAM2 using a zero-shot approach
Mar 24, 2026Miquel Lopez Escoriza, Pau Amargant Alvarez3D Medical Image Segmentation Zero-Shot Medical Image Segmentation
Department of Computer Science, EPFL, Switzerland

Jul 20, 2026J/K move · Enter open · S save

SAMRI-3D: Adapting SAM2 for 3D MRI Segmentation with Global Volume Tokens

Zhao Wang, Wei Dai, Hongfu Sun, Craig Engstrom+1

School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, Australia · School of Engineering, College of Engineering, Science and Environment, University of Newcastle, Australia · School of Human Movement and Nutrition Sciences, The University of Queensland, Australia

Medical Imaging Foundation Models 3D Medical Image Segmentation Medical Image Benchmarks

Open paper AskPDF arXiv

Foundation models such as Segment Anything Model 2 (SAM2) have transformed natural-image and video segmentation, and recent work has begun adapting them to medical imaging. These adaptations, however, are largely general-purpose models that treat MRI as one modality among many; large-scale, MRI-specific modelling and benchmarking remain limited, even though MRI's low soft-tissue contrast leaves many boundaries effectively invisible on individual slices. We present SAMRI-3D, a benchmark and method for 3D MRI segmentation with SAM2. The SAMRI-3D benchmark is the largest MRI-only evaluation to date - 10,392 volumes from 34 datasets (27 public, 7 in-house) spanning 12 anatomical domains and 10+ sequences, with explicit seen/unseen splits. Freezing the image encoder and fine-tuning only the lightweight decoder and memory modules raises mean Dice from 0.58 (zero-shot SAM2) to 0.76, surpassing recent SAM-based medical models (SAMed-2 0.69, Medical-SAM2 0.49, SAM-Med3D 0.37) with strong statistical significance. To target invisible boundaries, we introduce Global Volume Tokens (GVT): persistent memory tokens trained with a Truncated Signed Distance Field (TSDF) reconstruction objective that is discarded at inference (zero added cost). This full model, SAMRI-3D, attains the best accuracy (0.78) and lowest variance across all 34 datasets and, uniquely, shows no drop on 8 held-out datasets (0.79 unseen vs. 0.78 seen); per-sequence analysis confirms the TSDF objective helps most where per-slice contrast is weakest. We will release the benchmark, code, and models in this paper.

Abstract

Semantic Segmentation

Explore similar work

SAMRI-3D: Adapting SAM2 for 3D MRI Segmentation with Global Volume Tokens

TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision

Automatic Segmentation of 3D CT scans with SAM2 using a zero-shot approach