Boxes2Pixels: Learning Defect Segmentation from Noisy SAM Masks

Camile Lendering; Erkut Akdag; Egor Bondarev

cs.CVApr 13, 2026

Boxes2Pixels: Learning Defect Segmentation from Noisy SAM Masks

Authors: Camile Lendering, Erkut Akdag, Egor Bondarev

Organizations: AIMS Group, Department of Electrical Engineering, Eindhoven University of Technology

Abstract

Accurate defect segmentation is critical for industrial inspection, yet dense pixel-level annotations are rarely available. A common workaround is to convert inexpensive bounding boxes into pseudo-masks using foundation segmentation models such as the Segment Anything Model (SAM). However, these pseudo-labels are systematically noisy on industrial surfaces, often hallucinating background structure while missing sparse defects. To address this limitation, a noise-robust box-to-pixel distillation framework, Boxes2Pixels, is proposed that treats SAM as a noisy teacher rather than a source of ground-truth supervision. Bounding boxes are converted into pseudo-masks offline by SAM, and a compact student is trained with (i) a hierarchical decoder over frozen DINOv2 features for semantic stability, (ii) an auxiliary binary localization head to decouple sparse foreground discovery from class prediction, and (iii) a one-sided online self-correction mechanism that relaxes background supervision when the student is confident, targeting teacher false negatives. On a manually annotated wind turbine inspection benchmark, the proposed Boxes2Pixels improves anomaly mIoU by +6.97 and binary IoU by +9.71 over the strongest baseline trained under identical weak supervision. Moreover, online self-correction increases the binary recall by +18.56, while the model employs 80% fewer trainable parameters. Code is available at https://github.com/CLendering/Boxes2Pixels.

Figures & tables

Figure 1: Overview of the proposed framework, Boxes2Pixels, for sparse defect segmentation. From left to right, top to bottom: (i) an input wind turbine image, (ii) legacy YOLO bounding boxes used as coarse supervision, (iii) SAM-generated pseudo-masks exhibiting both spurious regions and missed defects, and (iv) prediction from Boxes2Pixels. Colors indicate correct detections (green), defects recovered by the student but missed by the teacher (blue), and omitted or erroneous regions (red).

Figure 2: Overview of the hierarchical student architecture. The global semantic branch employs DINOv2 ViT-S/14 with BitFit adaptation to extract intermediate features from layers $\{7,4,2,1\}$ . Projected features $F^{(l)}$ are fused top-down through residual fusion blocks to produce $F_{\text{global}}$ . In parallel, a lightweight CNN detail branch extracts local features at the same resolution, yielding $F_{\text{local}}$ . Both branches are concatenated and processed by a feature mixer to obtain $F_{\text{fusion}}$ , followed by a Binary Defect Head and a Fine-grained Semantic Head for $(K{+}1)$ -class segmentation. Training relies on offline SAM pseudo-masks generated from bounding boxes with objectives designed to mitigate systematic teacher errors.

Model	mIoU	mIoU ${}_{\text{anom}}$	F1 ${}_{\text{anom}}$	IoU ${}_{\text{bin}}$
U-Net	0.7057	0.5629	0.7036	0.5427
DeepLabV3-B2	0.6867	0.5342	0.6955	0.5331
SegFormer-B2	0.7231	0.5881	0.6939	0.5312
Boxes2Pixels (ours)	0.7661	0.6523	0.7674	0.6226

Table 1: Quantitative comparison on the manually annotated test split (evaluation against pixel-level ground truth). All models are trained with identical box-only supervision via SAM pseudo-masks; higher is better for all metrics.

Configuration	mIoU	mIoU ${}_{\text{anom}}$	F1 ${}_{\text{anom}}$	IoU ${}_{\text{bin}}$
Boxes2Pixels	0.6709	0.5093	0.6761	0.5107
w/o Local Detail Branch	0.6655	0.5013	0.6631	0.4960
w/o Binary Head	0.5868	0.3851	0.5693	0.3979
w/o Self-Correction	0.6679	0.5049	0.6622	0.4949

Table 2: Ablation study on the validation split (evaluation against SAM pseudo-labels). Metrics reflect agreement with the noisy teacher and are used for comparison and checkpoint selection.

Method	mIoU ${}_{\text{anom}}$	F1 ${}_{\text{anom}}$	IoU ${}_{\text{bin}}$	Recall ${}_{\text{bin}}$
w/o Self-Correction	0.5826	0.6889	0.5255	0.6195
w/ Self-Correction (Ours)	0.6523	0.7674	0.6226	0.8051
Improvement	+0.0697	+0.0785	+0.0971	+0.1856

Table 3: Impact of online self-correction on the manually annotated test split (evaluation against pixel-level ground truth). Two models differ only by enabling correction; Recall

{}_{\text{bin}}

highlights missed-defect recovery.

Model	Damage	Dirt	$\Delta_{\text{Damage}}$
DeepLabV3-B2	53.89	52.95	-17.76
U-Net	61.01	51.57	-10.64
SegFormer-B2	71.48	46.13	-0.17
Boxes2Pixels (ours)	71.65	58.81	–

Table 4: Per-class IoU on the manually annotated test set.

Model	Total	Trainable	FLOPs	Latency	FPS
	Params (M)	Params (M)	(G)	(ms)
DeepLabV3-B2	20.5	20.5	18.92	7.99	125.1
SegFormer-B2	27.3	27.3	58.97	13.33	75.0
Boxes2Pixels (ours)	27.6	5.6	43.37	6.20	161.4

Table 5: Efficiency comparison. Boxes2Pixels achieves real-time throughput (

>160

FPS) while requiring 80% fewer trainable parameters for adaptation.

Figure 3: Qualitative comparison on wind turbine blade inspection images. From left to right: YOLO bounding boxes used for supervision, SAM-generated pseudo-masks, and predictions of the proposed Boxes2Pixels

Weakly Supervised Image Segmentation

Weakly supervised image segmentation learns pixel-level regions from cheaper, incomplete labels such as image tags, boxes, points, scribbles, or reports.

See all 46 papers

SAM3 Self-Distillation for Fine-Grained GOOSE 2D Semantic Segmentation
Jun 18, 2026Xuesong WangVision Foundation Models Self-Distillation
Wayne State University, Detroit, MI, USA
OSAGEN: Object-Aware Mask Priors and Multistage Decoupled Diffusion for Industrial Anomaly Generation
Jul 31, 2026Jinyi Xu, Peng Chen, Yunkang Cao +3Industrial Anomaly Detection Anomaly Localization
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University · School of Artificial Intelligence and Robotics, Hunan University · Department of Computer and Information Science, University of Macau +1
UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity
Nov 17, 2025Junwei Yu, Trevor Darrell, XuDong WangImage Segmentation Promptable Image Segmentation
UC Berkeley

Jun 18, 2026J/K move · Enter open · S save

SAM3 Self-Distillation for Fine-Grained GOOSE 2D Semantic Segmentation

Xuesong Wang

Wayne State University, Detroit, MI, USA

Vision Foundation Models Self-Distillation Test-Time Augmentation Fine-Grained Semantic Segmentation Semantic Segmentation

Open paper AskPDF arXiv

We describe our 4th-place entry to the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge, which reached a composite mean Intersection-over-Union (mIoU) of 69.73% on the official 1,815-image test set. Our model adapts the image encoder of a recent visual foundation model, Segment Anything Model 3 (SAM3), with a lightweight decoder. Beyond this, we contribute two techniques and one empirical finding: (i) a self-distillation scheme that re-uses SAM3 itself, prompted with ground-truth boxes, as a teacher on the classes where it outperforms our own model; (ii) an image-level multi-scale test-time augmentation scheme that restores multi-scale inference for a fixed-input-size model by rescaling the image rather than the model input; and (iii) the finding that an aggressive photometric distortion from a winning 2025 GOOSE 2D entry, transplanted onto our pipeline, is its single largest source of improvement.

Abstract

Weakly Supervised Image Segmentation

Explore similar work

SAM3 Self-Distillation for Fine-Grained GOOSE 2D Semantic Segmentation

OSAGEN: Object-Aware Mask Priors and Multistage Decoupled Diffusion for Industrial Anomaly Generation

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity