Organizations: Department of Engineering, University of Ferrara, Via Saragat 1, 44122 Ferrara, Italy · Bonfiglioli Engineering, Via Amerigo Vespucci 20, 44124 Ferrara, Italy
Anomaly detection is nowadays increasingly used in industrial applications and processes. One of the main fields of the appliance is the visual inspection for surface anomaly detection, which aims to spot regions that deviate from regularity and consequently identify abnormal products. Defect localization is a key task, that usually is achieved using a basic comparison between generated image and the original one, implementing some blob-analysis or image-editing algorithms, in the post-processing step, which is very biased towards the source dataset, and they are unable to generalize. Furthermore, in industrial applications, the totality of the image is not always interesting but could be one or some regions of interest (ROIs), where only in those areas there are relevant anomalies to be spotted. For these reasons, we propose a new architecture composed by two blocks. The first block is a Generative Adversarial Network (GAN), based on a residual autoencoder (ResAE), to perform reconstruction and denoising processes, while the second block produces image segmentation, spotting defects. This method learns from a dataset composed of good products and generated synthetic defects. The discriminative network is trained using a ROI for each image contained in the training dataset. The network will learn in which area anomalies are relevant. This approach guarantees the reduction of using pre-processing algorithms, formerly developed with blob-analysis and image-editing procedures. To test our model we used challenging MVTec anomaly detection datasets and an industrial large dataset of pharmaceutical BFS strips of vials. This set constitutes a more realistic use case of the aforementioned network.
Figures & tables
Figure 1: Simulated anomaly generation process. In (a) there is an example of Perlin noise. (b) represent the merging of the anomaly map and a random RGB pixels. (c) represent and example of a image with generated fake anomalies.
Figure 2: The architecture of DRÆM GAN. The architecture is quite similar to vanilla DRÆM, but we can see the implementation of GANomaly instead of the AE which acted as the Reconstructive network.
Figure 3: Train step flowchart: input image X is transformed in Xn , that is the image with the Perlin noise superimposed. M is the mask image of the noise areas.
Figure 4: Inference step flowchart.
Figure 5: Two consecutive residual blocks of one stage of the encoder network. The introduction of a residual architecture in the encoder-decoder-encoder GAN [ 49 ] revealed to be more stable during training phase, by giving better results with equal epochs.
AUROC per image (pixel) at 10 epochs
DRÆM
GRD-Net
73.1
96.7
hazelnut
(55.7)
(91.0)
58.3
96.4
metal nut
(49.0)
(69.3)
74.2
77.5
Table 1: AUROC score after 10 epochs of training per image and (pixel).
AUROC per image (pixel) at 35 epochs
DRÆM
GRD-NET
85.3
99.5
hazelnut
(82.4)
(95.5)
61.8
99.3
metal nut
(49.0)
(69.3)
75.7
89.8
Table 2: AUROC score after 35 epochs of training per image and (pixel).
AUROC per image (pixel) at 100 epochs
DRÆM
GRD-Net
98.8
100.0
hazelnut
(94.8)
(97.3)
99.7
99.8
metal nut
(86.7)
(70.4)
93.8
98.2
Table 3: AUROC score after 100 epochs of training per image and (pixel).
AUROC per image (pixel)
GANomaly
DRÆM
PaDiM
PatchCore
GRD-Net
78.5
100.0
-
100.0
100.0
hazelnut
(-)
(95.0)
97.7
98.6
(97.4)
70.0
98.7
-
99.7
100.0
metal nut
(-)
(86.7)
96.7
98.4
(96.2)
74.3
97.9
-
97.0
98.5
Table 4: Final comparative table with AUROC score between GANomaly (200 epochs), DRÆM (200 epochs), PaDiM (ResNet18 pre-train), PatchCore (ResNet50 pre-train) and GRD-Net (200 epochs). The results of GANomaly and DRÆM. are obtained by us adjusting the number of training epochs to the number of training epochs used to train GRD-Net, for this reason the final result may vary a little from the reference paper.
Figure 6: Visual representation of how the network with vanilla autoencoder (magenta) is not only less effective, but also noisier in some losses, such as adversarial loss, compared to the residual architecture (cyan).
Loss value for training (validation) phase
Epoch 250 vanilla
Epoch 250 residual
Epoch 500 vanilla
Epoch 500 residual
2.8811×10−3
3.2498×10−4
7.4750×10−4
1.3797×10−4
Adversarial Loss
( 5.1180×10−2 )
( 3.8928×10−3 )
( 1.3720×10−3 )
( 4.5011×10−4 )
0.03502
0.01853
0.02912
0.01460
Contextual loss
(0.04136)
(0.02782)
(0.03747)
(0.02488)
1.7221×10−4
6.8905×10−5
1.1745×10−4
4.6982×10−5
Table 5: Comparison between vanilla and residual architecture used for generative part of the GAN architecture in the GRD-Net.
Figure 7: Validation losses for generative (a) and discriminative (b) sub-networks. Red curve is obtained during training of our model, the orange one is obtained with the vanilla model. It is evident that the learning curve is much better in our case, for both nets.
Figure 8: Especially significant example from zipper dataset in which we could spot 3 anomalies: one in the zipper, one in the middle of fabric part and another, the last, on the border of the fabric zone. As we can see, the only spotted is the one in the zipper region, perfectly inside ROI, and almost perfectly aligned with the ground truth defect region.
Figure 9: 3 examples of real cases where algorithmic analysis is difficult, if not almost impossible.
Figure 10: Visual results on real case experiment. First 3 images represent a defect, the last a regular product really difficult to spot.
Best results on 30 epochs training
Best AUROC per image
Best AUROC per pixel
Best accuracy
Vials on meniscus area
0.981
0.996
0.932
Table 6: Real-case experiment statistics after 30 epochs of training.
Figure 11: Pill in picture (a) is the original one from train set, pill in (b) is the original with Perlin noise superimposed. Last two images are the output from the generative subnetwork. (c) is from the full-convolutional residual autoencoder (CRAE), (d) is from the dense-bottleneck residual autoencoder (DRAE). It is clear that the capability to rebuild the original image is much higher in fully-convolutional residual autoencoder (CRAE) networks, especially for the little details on the texture.
Figure 12: Visual comparison between 4 losses, for the discriminative network. Second case is the more clear, that segment better only the anomalous areas.
AUROC per image (pixel) at 100 epochs
AUROC
AUROX pixel
Accuracy
Case 1.
99.4
94.3
94.6
Case 2.
100.0
95.3
100.0
Case 3.
99.9
91.6
99.1
Case 4.
100.0
93.5
100.0
Table 7: AUROC score after 200 epochs of training per image and pixel.
Department of Engineering, University of Ferrara, Via Saragat 1, Ferrara, 44122, Italy. · Bonfiglioli Engineering, Via Amerigo Vespucci 20, Ferrara, 44124, Italy.
State Key Laboratory of Precision Measurement Technology and Instruments, Tsinghua University, Beijing, 100084, China · State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, 430074, China