SteelDefectX: A Multi-Form Vision-Language Dataset and Benchmark for Steel Surface Defect Analysis

Shuxian Zhao; Jie Gui; Baosheng Yu; Dacheng Tao

cs.CVMar 23, 2026

SteelDefectX: A Multi-Form Vision-Language Dataset and Benchmark for Steel Surface Defect Analysis

Authors: Shuxian Zhao, Jie Gui, Baosheng Yu, Dacheng Tao

Organizations: Southeast University · Purple Mountain Laboratories · Nanyang Technological University

Abstract

Steel surface defect analysis is critical for industrial quality control, yet existing benchmarks rely primarily on label-only annotations, limiting fine-grained semantic understanding and systematic evaluation of vision-language models. To address this gap, we introduce SteelDefectX, a vision-language dataset with multi-form textual annotations for steel surface defect analysis, comprising 7,778 images across 25 defect categories. At the class level, the dataset provides defect names, representative visual attributes, and industrial causes. At the sample level, each image is annotated with three forms of textual representations: (1) free-form natural language descriptions, (2) structured attribute annotations, and (3) template-based sentences. These annotations provide flexible textual supervision with varying levels of expressiveness and controllability. We further establish a comprehensive benchmark covering vision-language classification, segmentation, and cross-dataset transfer, along with additional evaluations such as retrieval and text-guided localization. Experimental results reveal a trade-off between structure and flexibility in textual representations. Structured attributes provide more stable semantic alignment, while natural language descriptions improve transferability and fine-grained spatial grounding. These findings highlight the critical role of textual design in industrial vision-language learning. SteelDefectX provides a new benchmark for studying semantic alignment and generalization in industrial vision-language learning. The code and dataset are available at https://github.com/Zhaosxian/SteelDefectX.

Figures & tables

Figure 1: Illustration of multi-form textual annotations in SteelDefectX. Given a defect image and its segmentation mask, we present text forms from class-level descriptions (T1) to sample-level annotations, including natural language descriptions (T2), structured attributes (T3), and template-based sentences (T4). These forms vary in expressiveness and structure while sharing consistent attribute-based representations, enabling diverse forms of supervision for vision-language learning.

Dataset	Domain	#Images	Sample-level Text	Structured Attributes	Segmentation Mask
NEU [ 17 ]	Steel	1.8K	✗	✗	✗
GC10 [ 18 ]	Steel	2.3K	✗	✗	✗
X-SDD [ 19 ]	Steel	1.3K	✗	✗	✗
S3D [ 20 ]	Steel	0.8K	✗	✗	✓
MVTec-AD [ 29 ]	Multi-object	5.3K	✗	✗	✓
VisA [ 30 ]	Multi-object	10.8K	✗	✗	✓

Table 1: Comparison of SteelDefectX with representative industrial defect and multimodal datasets.

Figure 2: Sample-level annotation pipeline. The framework produces three text forms for each defect sample: natural language descriptions (T2), structured attributes (T3), and template-based sentences (T4). (a) “T2” is generated through candidate generation, refinement, supplementation, and manual correction. (b) “T3” combines rule-based measurements from images and masks with constrained semantic inference to produce a fixed nine-field attribute representation. (c) “T4” linearizes structured attributes into a standardized sentence for controllable vision-language learning.

Model	Backbone	T2-T0		T2-T1		T3-T0		T3-T1		T4-T0		T4-T1
Model	Backbone	Acc	mAcc	Acc	mAcc	Acc	mAcc	Acc	mAcc	Acc	mAcc	Acc	mAcc
CLIP [ 10 ]	ViT-B/16	82.96	82.42	76.94	73.13	90.75	88.08	77.62	79.21	85.80	85.69	74.96	75.66
OpenCLIP [ 22 ]	ViT-B/16	87.13	83.99	78.06	75.24	85.20	83.40	74.70	68.07	86.19	86.01	78.23	77.03
EVA-CLIP [ 23 ]	ViT-B/16	85.54	80.53	78.79	73.48	83.69	85.79	78.31	78.84	86.19	85.42	73.49	70.67
Long-CLIP [ 24 ]	ViT-B/16	88.25	85.41	82.83	79.11	90.23	91.27	82.92	81.96	89.50	89.35	78.23	77.32
FG-CLIP [ 25 ]	ViT-B/16	84.77	81.66	80.29	77.24	89.37	88.07	84.55	84.60	87.31	86.50	77.67	77.92

Table 2: Vision-language classification results under different training text representations, with evaluation performed using “T0” and “T1”. Best results are in bold.

Dataset	Backbone	Zero-shot	T0	T1	T2	T3	T4
Aluminum [ 44 ]	ViT-B/16	10.22	15.59	18.01	22.58	19.35	23.12
	ViT-L/14	8.60	12.90	20.43	29.03	12.37	15.59
Seamless Steel Tubes [ 45 ]	ViT-B/16	31.05	31.51	33.33	43.38	37.90	31.05
	ViT-L/14	25.11	28.31	33.79	40.18	28.77	27.40

Table 3: Cross-dataset transfer results (% Acc) of Long-CLIP Adapter trained on SteelDefectX and evaluated on two unseen defect datasets. Best results are in bold.

Table 7

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Representative samples of each defect category in the SteelDefectX. For each class, four images are shown to illustrate intra-class variations in appearance, texture, and morphology. The figure highlights substantial diversity in defect scale, shape, illumination, and surface conditions, illustrating the visual complexity of real-world steel defect patterns.

Figure 6: Class distribution of SteelDefectX. The dataset exhibits an imbalanced distribution across 25 defect categories, with sample counts following a log-normal trend. The average number of samples in the dataset is 311. Common defects such as inclusion and water spot dominate the dataset, whereas rare defects (e.g., crease and rolled pit ) are underrepresented, reflecting real-world variability in steel surface inspection scenarios.

Figure 7: t-SNE visualization of image-level features in SteelDefectX. The distribution reveals substantial intra-class variation and partial inter-class overlap across defect categories, highlighting the visual complexity of industrial defect data.

Metric	Avg. Coverage	Coverage $\geq$ 4	Shape	Size	Depth	Position	Contrast
Value	4.39 / 5	94.5%	100.0%	92.9%	51.3%	95.4%	99.3%

Appendix

Table 7: Dimension coverage of “T2”descriptions over five semantic dimensions. “Coverage

\geq

4” denotes the proportion of descriptions that cover at least four dimensions.

Figure 8: Text statistics of “T2” in SteelDefectX. (a) Distribution of description lengths in tokens. The annotations are concentrated around a mean of 59.59 tokens with mild positive skewness, indicating a consistent balance between detail and conciseness. (b) Sample-wise vocabulary diversity measured as the number of non-zero TF-IDF terms after stop-word removal. The relatively stable fluctuation across samples reflects controlled but meaningful lexical variation in the textual annotations.

Figure 9: Distribution of rule-based attributes in “T3” for the SteelDefectX dataset. Vertical dashed lines indicate discretization thresholds in the structured-attribute framework. (a) Scale ratio, defined as the defect-mask area divided by the full image area. (b) Number of connected components, plotted on a log-scaled horizontal axis. (c) Polarity, measured as the mean intensity difference between the defect region and the background. (d) Saliency, defined as the absolute mean intensity difference.

Type	Attribute	Candidate set (ratio)
Rule-based	Scale	tiny (5.3%); small (26.1%); medium (38.5%); large (20.2%); extensive (9.9%)
	Polarity	dark (79.6%); neutral (9.8%); bright (10.6%)
	Saliency	low (24.6%); medium (50.2%); high (25.2%)
	Number of defects	one (36.1%); two (12.4%); three (7.5%); four (5.0%); five (3.5%); multiple ( $>5$ , 35.6%)
	Position	3 $\times$ 3 grid; defect-area ratio within each cell
Semantic	Shape	{linear, curved, elongated, circular, irregular, fragmented, network-like, spot-like, patch-like, diffuse, unclear}

Appendix

Table 8: Definition of structured attributes in “T3”. Rule-based attributes are computed from image-mask statistics, while semantic attributes are inferred from “T2” descriptions within constrained candidate sets. Percentages indicate the empirical distribution across all 7,778 images.

Text	AUROC	F1-max	IoU	ImgTxtSim
DT	52.71	18.20	9.89	0.1719
DT+Shape	58.29	20.13	11.02	0.1685
DT+Direction	53.81	18.22	9.75	0.1857
DT+Spatial Distribution	52.27	18.04	9.92	0.1971
DT+Number of Defects	54.02	18.49	9.84	0.1662
DT+Position	49.59	18.04	9.92	0.1872

Appendix

Table 9: Attribute-level localization ablation with frozen FG-CLIP (ViT-L/14) on SteelDefectX. “DT” denotes the defect-type-only prompt, while “DT+attribute” refers to adding a single structured attribute to the defect-type prompt. Best results are in bold.

Setting	ViT-B/16		ViT-L/14
	Acc	mAcc	Acc	mAcc
T2-T0	$0.8759\pm 0.0068$	$0.8605\pm 0.0121$	$0.9227\pm 0.0084$	$0.9215\pm 0.0093$
T2-T1	$0.8279\pm 0.0141$	$0.7963\pm 0.0124$	$0.9096\pm 0.0074$	$0.8729\pm 0.0057$
T3-T0	$0.9002\pm 0.0058$	$0.8967\pm 0.0043$	$0.9314\pm 0.0123$	$0.9309\pm 0.0048$
T3-T1	$0.8411\pm 0.0037$	$0.8375\pm 0.0069$	$0.9316\pm 0.0061$	$0.9105\pm 0.0113$
T4-T0	$0.8931\pm 0.0066$	$0.8961\pm 0.0073$	$0.9234\pm 0.0067$	$0.9185\pm 0.0031$

Appendix

Table 10: Statistical significance analysis (

mean\pm std

over 3 runs) using Long-CLIP-Adapter with different backbones.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines
Jun 6, 2026Zekai Zhang, Jinglin Zhang, Qinghui Chen +8Open-Vocabulary Object Detection VLM Adaptation
School of Control Science and Engineering, Shandong University, Jinan 250061, China · Shandong Computer Science Center, Qilu University of Technology, Jinan, China · CEREMADE, University Paris Dauphine, PSL Research University, CNRS, UMR 7534, 75775 Paris, France +4
SynSur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection
Apr 29, 2026Paul Julius Kühn, Mika Pommeranz, Arjan Kuijper +1Synthetic Data Augmentation Industrial Anomaly Detection
Fraunhofer IGD, 64283 Darmstadt, Germany
SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments
Jul 6, 2026Suryanarayana Reddy Yarrabothula, Manisha Chawla, Kunal Sinha +5VLM Evaluation Video Understanding
Indian Institute of Technology Bhilai · Steel Authority of India Limited · VIT Vellore

Jun 6, 2026J/K move · Enter open · S save

Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines

Zekai Zhang, Jinglin Zhang, Qinghui Chen, Gang Li+7

School of Control Science and Engineering, Shandong University, Jinan 250061, China · Shandong Computer Science Center, Qilu University of Technology, Jinan, China · CEREMADE, University Paris Dauphine, PSL Research University, CNRS, UMR 7534, 75775 Paris, France+4

Open-Vocabulary Object Detection VLM Adaptation Industrial Anomaly Detection Visual Prompt Learning

Open paper AskPDF arXiv

Large-scale Visual-Language Models (LVLMs) have achieved remarkable success in natural visual tasks, yet their application to industrial defect detection remains challenging due to two fundamental limitations: (i) the scarcity of large-scale industrial datasets that cover diverse defect categories across multiple domains, and (ii) the reliance on manual prompts (points, boxes, masks) that introduce subjective noise and lack text-visual interaction for fine-grained understanding. To address these challenges, we introduce a Large-Scale Multi-Modal Industrial Open-Closed benchmark (MMIOC-1M) containing over one million samples across

14

super-categories,

29

industrial scenes, and

351

defect subcategories. To our knowledge, MMIOC-1M is the first unified largest benchmark supporting both open-vocabulary and closed-set industrial detection, providing valuable pre-training data for LVLMs in industrial scenarios. Furthermore, we propose a Refined Text-Visual Prompt Network (RTVPNet) that incorporates three key innovations: (1) an expert-assisted domain projection mechanism that enables rapid adaptation of general vision models to industrial domains, (2) an energy-based sparse sampling strategy that automatically generates refined visual prompts without manual intervention, and (3) a bidirectional text-visual interaction module that enhances cross-modal semantic alignment and understanding. Extensive experiments demonstrate that RTVPNet achieves state-of-the-art performance on MMIOC-1M, LVIS, and COCO benchmarks while maintaining computational efficiency. The dataset and code are available at https://github.com/hellozzk/MMIO.

Abstract

VLM Evaluation

Explore similar work

Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines

SynSur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection

SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments