Automatic weld seam segmentation for industrial quality control: a comparison of RGB and polarimetric imaging with CNN and transformer architectures

Simone Garbin; Leonardo Venturoso; Marco Todescato

cs.CVAug 26, 2026

Automatic weld seam segmentation for industrial quality control: a comparison of RGB and polarimetric imaging with CNN and transformer architectures

Authors: Simone Garbin, Leonardo Venturoso, Marco Todescato

Organizations: Fraunhofer Italia Research, Via A. Volta 13/A, 39100 Bolzano, Italy

Abstract

Visual inspection of welded assemblies remains one of the least automated stages in many industrial production processes, still depending largely on the experience of human operators and thus subject to inter-operator variability; the manufacturing of special-purpose machinery cabins, the setting of this study, is one representative case. This work evaluates the feasibility of automatic weld seam segmentation from RGB and polarimetric imagery, comparing controlled laboratory acquisitions with images captured under real, uncontrolled conditions. Convolutional neural network (CNN) architectures and transformer-based architectures are benchmarked under a unified, threshold-independent protocol, training each CNN with three random seeds to separate genuine effects from seed noise. In controlled RGB conditions, CNN models reach a mean mask mAP50 of up to 0.87, but drop to 0.22-0.48 under uncontrolled acquisition, showing that the acquisition setup is a first-order component of the inspection system. Polarimetric imaging with alignment-preserving geometric augmentation localizes previously unseen welds with a mean mask mAP50 up to 0.93: on par with, rather than ahead of, the best controlled-RGB result, but reaching that accuracy on uncontrolled RGB without requiring acquisition control. The clearest architectural finding concerns viewpoint robustness. In-distribution, transformers and CNNs are broadly comparable; but under a test-time viewpoint shift, the transformer models, and RF-DETR in particular, retain high accuracy while every CNN collapses. The gap holds across three seeds and a resolution-matched control, pointing to architecture rather than training resolution. Within the CNN family, capacity brings no reliable in-distribution gain once seed variance is accounted for: small CNNs suffice for fixed viewpoints, transformers for variable ones.

Figures & tables

Figure 1: End-to-end validation pipeline. Data partitions are split at the physical-weld level before any augmentation (Sec. 3.5 ); all six models are evaluated with the same pycocotools protocol at a fixed confidence threshold (Sec. 3.5 ), both in-distribution and, zero-shot, on the close-range set (Sec. 5 ).

Dataset	Camera	Resolution	Train (aug.)	Val	Test
RGB controlled	OAK-D	$1920{\times}1080$	51 (510)	15	12
RGB industrial	OAK-D	$1920{\times}1080$	35 (175)	11	8
PolarSens multi-map	PolarSens	$2448{\times}2048$	47 (1384)	15	5
PolarSens Intensity	PolarSens	$1224{\times}1024$	45 (450)	14	10
Close-range test	OAK-D	$1920{\times}1080$	—	—	62

Table 1: Datasets used in this study. “Train” counts physically distinct source acquisitions, with augmented training images in parentheses; augmented variants are generated from training sources only (Sec. 3.5 ). Acquisition conditions are detailed in the text. The close-range set is evaluation-only: by design it has no train or validation split (Sec. 3.1 , “Close-range test set”), and all 62 images are used exclusively for zero-shot cross-evaluation of models trained on the other scenarios.

Figure 2: The six PolarSens per-pixel maps derived from one acquisition (weld 20, test split), ground-truth weld outline in green.

Model	Family	Params (M)	Input res. (px)	Latency (ms/img)
YOLOv11-n-seg	CNN	2.8	2464	24
YOLOv11-s-seg	CNN	10.1	2464	49
YOLOv11-m-seg	CNN	22.4	2464	111
YOLOv8-s-seg	CNN	11.8	2464	42
RF-DETR-Seg-S	Transformer	36.3	1120	132
Mask2Former (Swin-S)	Transformer	69	1024 †	454

Table 2: Evaluated architectures. End-to-end latency (preprocessing, inference, postprocessing) per image on a single NVIDIA RTX 3090, measured on the polarimetric validation split at each model’s input resolution; transformer figures include amortized model loading.

Figure 3: The four polarimetric strategies of Sec. 3.4 . (a)–(b) use an unmodified YOLO11-seg architecture, single 3-channel input; they differ only in what is fed as training data. (c)–(d) use custom Split / ConvMultiInput layers to fuse all six maps inside the network, either through one shared backbone on the stacked 18-channel input (c) or through six parallel backbones concatenated at three feature scales before a shared neck (d). Verified against config/yolo11-seg-custom-polarsens-{singlebackbone,multibackbone}.yaml .

Figure 4: Qualitative segmentation results (ground truth in solid green, prediction in dashed red) across the three main acquisition scenarios. Industrial RGB shows a representative CNN failure mode: a false-positive mask on cluttered background (bottom) alongside a missed weld (top right, green outline with no matching prediction).

	Test		Validation
Model	mask mAP 50	mAP 50–95	mask mAP 50	mAP 50–95
RF-DETR-Seg-S	0.900	0.471	0.755	0.395
YOLOv11-n	0.819	0.452	0.792	0.436
YOLOv11-s	0.797	0.475	0.814	0.433
YOLOv11-m	0.795	0.437	0.783	0.433
YOLOv8-s	0.791	0.454	0.723	0.422

Table 3: Controlled RGB scenario, uniform pycocotools protocol, seed-0 checkpoints (test: 12 images; validation: 15 images).

Model	mask mAP 50	mAP 50–95
RF-DETR-Seg-S	$\mathbf{0.842\pm 0.015}$	$\mathbf{0.537\pm 0.027}$
Mask2Former (Swin-S)	$0.406\pm 0.050$	$0.243\pm 0.042$
YOLOv11-s	$0.168\pm 0.037$	$0.091\pm 0.019$
YOLOv11-n	$0.116\pm 0.005$	$0.066\pm 0.004$
YOLOv11-m	$0.102\pm 0.011$	$0.048\pm 0.008$
YOLOv8-s	$0.081\pm 0.005$	$0.045\pm 0.004$

Table 4: Zero-shot cross-evaluation on the close-range test set (62 images) of the six models trained on controlled RGB (Table 3 ), uniform pycocotools protocol; no fine-tuning on close-range data. Mean

\pm

population standard deviation over three training seeds for every model. Even the worst individual Mask2Former seed (0.357) exceeds the best individual CNN seed (YOLOv11-s, 0.215), so the transformer advantage is not an artifact of seed selection.

	Test		Validation
Model	mask mAP 50	mAP 50–95	mask mAP 50	mAP 50–95
RF-DETR-Seg-S	0.619	0.243	0.643	0.311
Mask2Former (Swin-S)	0.523	0.198	0.435	0.212
YOLOv8-s	0.471	0.244	0.500	0.224
YOLOv11-s	0.414	0.167	0.477	0.209
YOLOv11-n	0.408	0.130	0.507	0.251

Table 5: Industrial RGB scenario, uniform pycocotools protocol, seed-0 checkpoints (test: 8 images; validation: 11 images).

Model	mask mAP 50	mAP 50–95
RF-DETR-Seg-S	$\mathbf{0.810\pm 0.007}$	$\mathbf{0.537\pm 0.014}$
Mask2Former (Swin-S)	$0.461\pm 0.164$	$0.300\pm 0.124$
YOLOv11-n	$0.189\pm 0.008$	$0.089\pm 0.005$
YOLOv11-s	$0.102\pm 0.037$	$0.037\pm 0.011$
YOLOv8-s	$0.061\pm 0.016$	$0.025\pm 0.007$
YOLOv11-m	$0.043\pm 0.015$	$0.018\pm 0.010$

Table 6: Zero-shot cross-evaluation on the close-range test set of the six models trained on industrial RGB (Table 5 ), uniform pycocotools protocol; no fine-tuning on close-range data. Mean

\pm

population standard deviation over three training seeds for every model. Mask2Former’s variance here is exceptional (seeds 0.352, 0.338, 0.693): one seed nearly doubles the other two. Even so, its worst individual seed (0.338) exceeds the best individual CNN seed (YOLOv11-n, 0.200), so the ranking is unaffected; this high transformer variance is itself discussed in Sec. 5 .

	Test		Validation
Model	mask mAP 50	mAP 50–95	mask mAP 50	mAP 50–95
YOLOv11-n	0.914	0.501	0.756	0.350
RF-DETR-Seg-S	0.904	0.421	0.701	0.295
YOLOv8-s	0.862	0.500	0.700	0.336
YOLOv11-s	0.807	0.479	0.679	0.323
YOLOv11-m	0.743	0.382	0.644	0.289

Table 7: PolarSens multi-map with geometric augmentation, uniform pycocotools protocol, seed-0 checkpoints: leakage-free sample-level splits (test: 5 unseen welds / 30 images; validation: 15 unseen welds / 90 images).

Controlled RGB
	Test		Validation
Model	mask mAP 50	mAP 50–95	mask mAP 50	mAP 50–95
YOLOv11-n	$0.858\pm 0.009$	$0.503\pm 0.020$	$0.869\pm 0.013$	$0.517\pm 0.025$
YOLOv11-s	$0.860\pm 0.018$	$0.504\pm 0.013$	$0.849\pm 0.007$	$0.521\pm 0.007$
YOLOv11-m	$0.865\pm 0.012$	$0.510\pm 0.014$	$0.844\pm 0.014$	$0.514\pm 0.005$
YOLOv8-s	$0.779\pm 0.034$	$0.463\pm 0.026$	$0.855\pm 0.006$	$0.549\pm 0.009$

Table 8: CNN seed variance across the three in-distribution scenarios: mean

\pm

population standard deviation over 3 seeds, Ultralytics-native validation protocol. This protocol differs from the uniform pycocotools protocol used in Tables 3 , 5 and 7 , so the seed-0 entries here are different numbers from the seed-0 rows in those tables for the same checkpoints; the ranges reported per scenario in the text apply to that scenario’s rows only, not to the table as a whole. These figures characterize variance within the CNN family and are not comparable across families; see Tables 3 , 5 and 7 for the protocol-matched comparison against the transformers, whose in-distribution results are single-seed (Sec. 3.3 ). Nothing is bolded, since no architecture is reliably distinguishable from the others at these standard deviations.

	Test		Validation
Model	mask mAP 50	mAP 50–95	mask mAP 50	mAP 50–95
Single-backbone (18ch, split layer)	0.117	0.040	0.178	0.068
Multi-backbone (6 parallel)	0.216	0.029	0.083	0.039

Table 9: Polarimetric channel fusion (YOLOv11-n backbone, single class), alignment-preserving augmentation, leakage-free splits (test: 5 unseen welds / 30 images; validation: 15 unseen welds / 90 images). Multi-backbone leads only on test mask mAP 50 and trails on every other column, including its own validation split; with only 5 test and 15 validation welds, single high-IoU-threshold hits or misses swing these metrics disproportionately, so the two variants should be read as comparably poor rather than one reliably beating the other (see the small-test-set caveat in Sec. 5 ).

Figure 5: Domain-shift comparison on a single close-range test image (ground truth in solid green, prediction in dashed red), both models trained on controlled RGB only and evaluated zero-shot: YOLOv11-n produces no prediction at all, while RF-DETR-Seg-S localizes the weld closely.

Configuration	Train (px)	Test (px)	mask mAP 50	mAP 50–95
YOLOv11-n, native (baseline)	1920	1920	0.119	0.068
YOLOv11-n, inference-only	1920	1120	0.611	0.330
YOLOv11-n, retrained matched	1120	1120	0.174	0.081
RF-DETR-Seg-S (reference)	1120	1120	0.842	0.537

Table 10: Resolution ablation: YOLOv11-n trained on controlled RGB, evaluated zero-shot on the close-range test set (62 images), uniform pycocotools protocol. RF-DETR-Seg-S is included as the transformer reference point at the same test resolution. The native-train/test and RF-DETR rows reproduce values already reported in Table 4 .

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 399 papers

Enhanced Seam Segmentation for Automated Welding Robot in Construction Through Transfer Learning: Addressing Limitations of Bilateral Segmentation Network
Jul 7, 2026Keonvin Park, Yong Ann Voeurn, Hyeokjun Kweon +1Image Segmentation Robotics
Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, 08826, Republic of Korea · Department of Civil Engineering and Construction, Georgia Southern University, Statesboro, GA 30460, USA · The Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University, Seoul, 06974, Republic of Korea
Automated Weld Seam Recognition and 3D Mapping for Robotic Post Processing Using Photogrammetry and Semantic Segmentation
Sep 3, 2026Augustin Raju, Abilash Madavath, Chandra Yuvesh Aubeeluck +3
TH Köln - University of Applied Sciences, Germany
AOI-SSL: Self-Supervised Framework for Efficient Segmentation of Wire-bonded Semiconductors In Optical Inspection
May 12, 2026Joaquín Figueira, Rob Van Gastel, Giacomo D'Amicantonio +4Self-Supervised Pre-Training Semiconductor Manufacturing
AIMS Lab, Eindhoven University of Technology · CoC, ASMPT · iCIS, Radboud University

Jul 7, 2026J/K move · Enter open · S save

Enhanced Seam Segmentation for Automated Welding Robot in Construction Through Transfer Learning: Addressing Limitations of Bilateral Segmentation Network

Keonvin Park, Yong Ann Voeurn, Hyeokjun Kweon, Doyun Lee

Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, 08826, Republic of Korea · Department of Civil Engineering and Construction, Georgia Southern University, Statesboro, GA 30460, USA · The Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University, Seoul, 06974, Republic of Korea

Image Segmentation Robotics Semantic Segmentation

Open paper AskPDF arXiv

Reliable seam segmentation is essential for autonomous robotic welding in construction, where harsh illumination, specular reflections, and thin weld geometries often degrade segmentation performance. This study proposes a reflection-robust seam segmentation framework that enhances a BiSeNetV2 backbone through transfer learning and a hybrid Cross-Entropy--Lovász loss. Rather than increasing architectural complexity, the proposed framework improves reflection robustness through learning-stability-oriented optimization. Experimental results show that the proposed method achieves 81.76% Joint IoU and 90.73% mIoU, improving Joint IoU by +22.36 percentage points over the OHEM-based baseline while maintaining identical FLOPs, parameter count, and inference speed. The proposed approach also recovers 96.33% of severe zero-IoU failure cases under reflective conditions. Comparative experiments across BiSeNetV2, DeepLabV3+, UNet, and SegFormer further demonstrate that the proposed optimization strategy is particularly effective for lightweight real-time segmentation architectures. Qualitative analyses additionally show improved seam continuity and reflection robustness in challenging welding environments. These findings suggest that the proposed framework provides a practical and lightweight perception solution for robotic welding applications involving reflective metallic surfaces.

Abstract

Image Segmentation

Explore similar work

Enhanced Seam Segmentation for Automated Welding Robot in Construction Through Transfer Learning: Addressing Limitations of Bilateral Segmentation Network

Automated Weld Seam Recognition and 3D Mapping for Robotic Post Processing Using Photogrammetry and Semantic Segmentation

AOI-SSL: Self-Supervised Framework for Efficient Segmentation of Wire-bonded Semiconductors In Optical Inspection