Defect grading of power transmission equipment (DGPTE) is crucial to the stability of electric energy transmission. Although existing machine learning methods exhibit strong capabilities in defect detection, they are plagued by difficulties in integrating expert experience and facing class imbalance in more refined defect grading field. To address this issue, this paper introduces a novel defect grading framework based on multimodal large language model (MLLM). Specifically, this approach maximizes the commercial MLLMs' potential of DGPTE through in-context learning and obtains the state-of-te-art (SOTA) model. By sending a secondary request to this model, a small number of chain of thought-based question-answer pairs (Q&As) are generated, which effectively reduces the cost of manual annotation. In this way, these high-quality interpretable Q&As are used to train Qwen3-VL-8B via Low-Rank Adaption-based supervised fine-tuning (SFT). Experimental results on three DGPTE tasks demonstrate that fine-tuning only the language model layer yields the SOTA performance. Furthermore, multi-task joint fine-tuning verifies the feasibility of handling multiple grading tasks within only a single lightweight MLLM.
Figures & tables
Acronym
Description
DGPTE
Defect grading of power transmission equipment
MLLM
Multimodal Large Language Model
SFT
Supervised Fine-tuning
LoRA
Low-Rank Adaptation
ICL
In-context learning
FSL
Few-shot learning
TABLE I: Commonly Used Symbols
Fig. 1: The three PTEDG tasks used in the experiments of this paper
Fig. 2: The framework of proposed fine-tuning strategy
Execute the checks in steps 1 → 4 . If a No or Not is returned, the grading outcome shall be determined directly, and the subsequent check items are exempted from further evaluation.
1. Having insulator: Does the image contain at least one insulator string?
⊢ Not Exists → grade result is ”Else”
⊢ Exists → 2
2. Nest location: Is the bird’s nest located on the crossarm, away from the main body of the vertical tangent tower or concrete pole?
TABLE II: Decision Tree-based Chain of Thought with Task 3
Fig. 3: An example of Q&A for MLLM fine-tuning in Task 1
Stage
Parameter
Range / Value
Fine-tuning
Fine-tuning Model
Qwen3-VL-8B
Fine-tuning Type
SFT
Fine-tuning Module
{VE, MMA, LLM}
Fine-tuning Method
LoRA
LoRA Rank
32
LoRA Alpha
128
TABLE III: Fine-tuning and Inference Parameters
Case
Only result
DT-based CoT result
Reference images
Case 1
✓
Case 2
✓
Case 3
✓
✓
Case 4
✓
✓
TABLE IV: Four prompt settings for testing the commercial MLLMs
Model
Task 1
Task 2
Task 3
Case 1
Case 2
Case 3
Case 4
Case 1
Case 2
Case 3
Case 4
Case 1
Case 2
Case 3
Case 4
GPT-4.1-2025-04-14
53.24%
54.64%
74.85%
89.82%
43.49%
44.80%
46.25%
50.66%
43.86%
48.26%
50.50%
52.61%
GPT-4o-2024-11-20
54.37%
55.61%
74.47%
88.64%
30.18%
41.20%
46.98%
48.03%
42.50%
39.50%
48.95%
51.60%
GPT-5-chat
51.84%
51.24%
80.68%
90.26%
30.18%
45.72%
52.64%
56.58 %
42.12%
39.50%
46.50%
45.00%
Qwen-VL-plus
53.99%
55.81%
73.53%
84.92%
48.64%
46.90%
50.08%
54.56%
40.23%
42.54%
32.86%
33.45%
Qwen3-VL-235B-a22b
61.18%
69.38%
81.53%
88.22%
38.55%
52.64%
48.64%
52.64%
43.50%
42.65%
45.79%
48.26%
TABLE V: Accuracy comparison of different MLLMs in DGPTE
Fig. 4: Accuracy comparison of different fine-tuning modules
Fig. 5: Accuracy comparison of different fine-tuning strategies
Fig. 6: two generated defect grading examples for Task 3
Image Location
Task 1
Task 2
Task 3
Before Fine-tuning
After Fine-tuning
Before Fine-tuning
After Fine-tuning
Before Fine-tuning
After Fine-tuning
ACC
MF1
ACC
MF1
ACC
MF1
ACC
MF1
ACC
MF1
ACC
MF1
At the Front
42.11%
56.83%
50.23%
53.17%
32.80%
46.08%
34.66%
36.52%
23.14%
38.94%
27.50%
29.36%
At Corresponding Position
59.06%
71.03%
92.63%
92.50%
34.79%
49.51%
85.85%
88.01%
30.61%
46.02%
77.04%
77.32%
At the End
43.86%
58.02%
50.43%
53.30%
30.06%
44.40%
35.33%
37.20%
25.88%
42.12%
27.50%
30.58%
TABLE VI: Performance Comparison by Image Location
Size
ACC
MF1
Training Time (h)
Training VRAM (GB)
Inference Time (s)
Inference VRAM (GB)
p224
68.55%
67.76%
2.246
255.996
5.046
12.969
p448
72.76%
73.88%
2.956
282.395
5.140
22.495
p768
77.42%
80.02%
3.907
325.293
6.975
31.850
p1280
85.17%
85.94%
9.469
489.999
9.236
42.285
p2560
-
-
-
-
10.564
74.592
TABLE VII: Performance Overhead by Different Input Sizes
Utility companies increasingly rely on drone imagery for post-event and routine inspection, but training accurate defect-type classifiers remains difficult because defect examples are rare and inspection datasets are often limited or proprietary. We address this data-scarcity setting by using an off-the-shelf multimodal large language model (MLLM) as a training-free image generator to synthesize defect images from visual references and text prompts. Our pipeline increases diversity via dual-reference conditioning, improves label fidelity with lightweight human verification and prompt refinement, and filters the resulting synthetic pool using an embedding-based selection rule based on distances to class centroids computed from the real training split. We evaluate on ceramic insulator defect-type classification (shell vs. glaze) using a public dataset with a realistic low training-data regime (104 real training images; 152 validation; 308 test). Augmenting the 10% real training set with embedding-selected synthetic images improves test F1 score (harmonic mean of precision and recall) from 0.615 to 0.739 (20% relative), corresponding to an estimated 4--5x data-efficiency gain, and the gains persist with stronger backbone models and frozen-feature linear-probe baselines. These results suggest a practical, low-barrier path for improving defect recognition when collecting additional real defects is slow or infeasible.
Xuesong Wang, Caisheng Wang
Department of Electrical and Computer Engineering, Wayne State University, 42 W. Warren Ave., Detroit, 48201, MI, USA
Multimodal large language models have demonstrated strong defect recognition capability in industrial anomaly detection. However, in lithography review, merely determining whether an image contains a defect is insufficient for engineering inspection; models must also understand defect morphology, spatial location, and the potential causes supported by visible evidence. To this end, this paper proposes LDU-Bench, a multi-task multimodal benchmark for lithography defect understanding. Constructed from real lithography and integrated-circuit review images, LDU-Bench decomposes the review workflow into four independent tasks: defect triage, morphology recognition, coarse localization, and image-conditioned cause analysis. It systematically evaluates models using task-level metrics, diagnostic readouts, and the Lithography Closure Score (LCS). Experimental results show that although existing MLLMs can perform defect triage relatively reliably, this ability does not stably transfer to downstream review stages. Morphology alignment, effective localization, and evidence-to-cause mapping remain the major bottlenecks. Further diagnostics indicate that this capability break is not a fluctuation of a single metric, but reflects insufficient structured understanding across semantic levels. Overall, LDU-Bench provides a quantifiable and diagnostic unified platform for evaluating the usability, failure points, and capability boundaries of industrial MLLMs in lithography review chains.
Multimodal large language models (MLLMs) can enrich industrial anomaly detection with semantic descriptions and anomaly reasoning, but they still lag specialist anomaly detectors in binary detection accuracy. Existing approaches address this gap by fine-tuning MLLMs or training bridging modules to align expert outputs with MLLM inputs, limiting flexibility across backbones. We propose EAGLE, a tuning-free framework that integrates expert anomaly detectors with frozen MLLMs. EAGLE consists of Threshold-Guided Prompt Selection (TGPS), which estimates a decision threshold from expert model statistics and selects textual and visual prompts, and Confidence-Aware Attention Sharpening (CAAS), which shifts MLLM attention toward visual evidence when expert confidence is low. Beyond improving accuracy, we analyze MLLM attention and find that correct anomaly predictions are associated with stronger focus on ground-truth defect regions; EAGLE consistently strengthens this alignment. On MVTec-AD and VisA, EAGLE improves five MLLM backbones without parameter updates, reaching up to 94.4% and 88.1% in anomaly discrimination accuracy, respectively, and achieving performance competitive with fine-tuning-based methods while largely preserving MLLM semantic reasoning ability.