AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning

Jiaming Su; Tengchao Yang; Ruikang Zhang; Zhengan Yan; Haoyu Sun; Linfeng Zhang

cs.CVApr 9, 2026

AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning

Authors: Jiaming Su, Tengchao Yang, Ruikang Zhang, Zhengan Yan, Haoyu Sun, Linfeng Zhang

Organizations: Shanghai Jiao Tong University Shanghai, China · Tongji University Shanghai, China · Fudan University Shanghai, China

Abstract

Industrial anomaly generation is a crucial method for alleviating the data scarcity problem in anomaly detection tasks. Most existing anomaly synthesis methods rely on single-step generation mechanisms, lacking complex reasoning and iterative optimization capabilities, making it difficult to generate anomaly samples with high semantic realism. We propose AnomalyAgent, an anomaly synthesis agent with self-reflection, knowledge retrieval, and iterative refinement capabilities, aiming to generate realistic and diverse anomalies. Specifically, AnomalyAgent is equipped with five tools: Prompt Generation (PG), Image Generation (IG), Quality Evaluation (QE), Knowledge Retrieval (KR), and Mask Generation (MG), enabling closed-loop optimization. To improve decision-making and self-reflection, we construct structured trajectories from real anomaly images and design a two-stage training framework: supervised fine-tuning followed by reinforcement learning. This process is driven by a three-part reward mechanism: (1) task rewards to supervise the quality and location rationality of generated anomalies; (2) reflection rewards to train the model's ability to improve anomaly synthesis prompt; (3) behavioral rewards to ensure adherence to the trajectory. On the MVTec-AD dataset, AnomalyAgent achieves IS/IC-L of 2.10/0.33 for anomaly generation, 57.0% classification accuracy using ResNet34, and 99.3%/74.2% AP at the image/pixel level using a simple UNet, surpassing all zero-shot SOTA methods. The code is publicly available at https://github.com/sssjmmm/AnomalyAgent.

Figures & tables

Figure 1. Motivation of AnomalyAgent. Few-shot methods generalize poorly with limited defect data, while zero-shot methods often generate semantically inconsistent anomalies. AnomalyAgent resolves this via a closed-loop agentic framework for realistic, generalizable synthesis.

Figure 2. Overview of AnomalyAgent. Given a normal image, the agent iteratively invokes tools (PG, IG, QE, KR, MG) through a thought–action–observation loop. Feedback from each turn guides refinement, producing high-quality anomaly images and corresponding masks. Tool selection is adaptively determined by the model instead of a predefined sequence. Overview of AnomalyAgent. A normal industrial image and an anomaly prompt are input to the agent planner, which selects appropriate tools from a tool library to synthesize anomaly images and corresponding masks.

Figure 3. Pipeline of trajectory construction and taxonomy. Given an anomaly image, we generate multi-turn trajectories categorized into three types based on the number of IG calls.

Method	IS $\uparrow$	IC-L $\uparrow$
Traditional Zero-Shot Methods
CutPaste ( Li et al., 2021 )	1.76	0.22
DRAEM ( Zavrtanik et al., 2021 )	1.76	0.25
NSA ( Schlüter et al., 2022 )	1.44	0.26
RealNet ( Zhang et al., 2024 )	1.64	0.22
AnomalyAny ( Sun et al., 2025 )	2.02	0.33

Table 1. Comparison of IS and IC-L on the MVTec dataset. Our method achieves the best IS and IC-L scores. Bold and underline represent optimal and sub-optimal results.

Method	Accuracy $\uparrow$
Traditional Zero-Shot Methods
AnoStyler ( So and Kang, 2025 )	32.2
AnoHybrid ( Zhao, 2025 )	52.6
Image Generation and Editing Models
Gemini 3.1 Flash Image Preview	44.7
GPT Image 1.5	40.5

Table 2. Comparison of average anomaly classification accuracy on MVTec-AD. Our method achieves the highest mean performance.

Category	AnoHybrid ( Zhao, 2025 )			Gemini-img			GPT-img			AnomalyAgent			AnoHybrid ( Zhao, 2025 )			Gemini-img			GPT-img			AnomalyAgent
	Pixel-Level Performance												Image-Level Performance
	AUC	AP	$F_{1}$	AUC	AP	$F_{1}$	AUC	AP	$F_{1}$	AUC	AP	$F_{1}$	AUC	AP	$F_{1}$	AUC	AP	$F_{1}$	AUC	AP	$F_{1}$	AUC	AP	$F_{1}$
bottle	98.3	77.2	72.5	96.1	74.5	68.1	86.0	52.7	51.5	97.2	73.5	69.7	99.2	99.8	98.7	98.7	99.5	97.7	99.5	99.8	97.7	98.7	99.6	98.1
cable	94.1	76.0	73.4	93.3	68.5	60.2	90.2	55.5	50.4	97.4	76.5	69.2	96.1	97.7	91.9	98.8	98.9	94.7	98.3	98.8	97.4	99.1	99.3	96.1
capsule	98.4	51.7	54.6	96.0	44.0	47.3	88.3	33.2	34.5	96.6	46.0	49.7	94.9	98.8	95.2	90.8	96.7	92.0	90.2	91.3	88.7	91.7	97.6	91.8
carpet	98.6	82.8	75.6	98.8	79.0	72.3	91.3	69.0	60.4	99.4	85.1	76.4	96.3	98.9	94.0	97.9	99.2	96.0	95.6	96.1	95.9	99.4	99.8	98.4

Table 3. Comprehensive performance comparison on MVTec-AD dataset. The table is structured to compare Pixel-Level (left) and Image-Level (right) performance across four methods. Bold indicates the best results. (Gemini-img: Gemini 3.1 Flash Image Preview; GPT-img: GPT Image 1.5)

Figure 4. Training dynamics of AnomalyAgent. Left: SFT loss rapidly decreases and converges. Right: RL reward steadily improves throughout GRPO optimization.

Figure 5. Visualization of anomaly synthesis results on MVTec-AD. AnomalyAgent achieves higher semantic consistency and more accurate defect localization than traditional generative models (Gemini-img: Gemini 3.1 Flash Image Preview; GPT-img: GPT Image 1.5; Grok-img: Grok Imagine Image). visual Comparison.

#	Method					Generation		Classification
#	PG	QE	KR	SFT	RL	IS $\uparrow$	IC-L $\uparrow$	Accuracy $\uparrow$
(a)						1.91	0.29	44.7
(b)		✓	✓			1.90	0.29	48.2
(c)	✓	✓				2.03	0.32	47.3
(d)	✓					1.99	0.32	45.1
(e)	✓	✓	✓			2.03	0.32	49.5

Table 4. Ablation study of proposed components with different task categories.

#	Rewards			Generation		Classification
#	$R_{\text{task}}$	$R_{\text{ref}}$	$R_{\text{beh}}$	IS $\uparrow$	IC-L $\uparrow$	Accuracy $\uparrow$
(a)				2.08	0.33	52.4
(b)	✓			2.09	0.33	53.6
(c)	✓	✓		2.09	0.33	55.3
(d)	✓	✓	✓	2.10	0.33	57.0

Table 5. Ablation study of different reward components.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7. Case Study 1. Satisfactory anomaly images are obtained with a single IG call. Case Study 1.

Figure 8. Case Study 2. After an initial low-quality generation, the prompt is refined based on QE feedback to produce satisfactory anomaly image and mask. Case Study 2.

Figure 9. Case Study 3. After an initial low-quality generation, the prompt is refined using both KG retrieval and QE feedback, resulting in satisfactory anomaly image and mask. Case Study 3.

Figure 10. Generates anomaly images for various defect types in MVTec-AD. Each image is displayed along with its corresponding anomaly mask.

Explore similar work

One-to-More: High-Fidelity Training-Free Anomaly Generation with Attention Control
Mar 18, 2026Haoxiang Rao, Zhao Wang, Chenyang Si +4Synthetic Anomaly Generation Industrial Anomaly Detection
Nanjing University · China Mobile Zijin Innovation Institute · Shandong University of Science and Technology
CORE: Contrast-and-Reflect for Self-Supervised Prompt Optimization in Anomaly Synthesis
Oct 14, 2026Haoxiang Rao, Fang Zhao, Zhao Wang +3Synthetic Anomaly Generation Synthetic Data Augmentation
School of Intelligence Science and Technology, Nanjing University · China Mobile Zijin Innovation Institute
UniScale: Arbitrary-Scale Industrial Anomaly Generation
Aug 8, 2026Shilei Zeng, Linxin Guan, Xurui Li +2Industrial Anomaly Detection Synthetic Anomaly Generation
School of Electronic Information and Communications, Huazhong University of Science and Technology

Mar 18, 2026J/K move · Enter open · S save

One-to-More: High-Fidelity Training-Free Anomaly Generation with Attention Control

Haoxiang Rao, Zhao Wang, Chenyang Si, Yan Lyu+3

Nanjing University · China Mobile Zijin Innovation Institute · Shandong University of Science and Technology

Synthetic Anomaly Generation Industrial Anomaly Detection Diffusion-Based Image Generation Attention Control in Diffusion Models Few-Shot Anomaly Detection

Open paper AskPDF arXiv

Industrial anomaly detection (AD) is characterized by an abundance of normal images but a scarcity of anomalous ones. Although numerous few-shot anomaly synthesis methods have been proposed to augment anomalous data for downstream AD tasks, most existing approaches require time-consuming training and struggle to learn distributions that are faithful to real anomalies, thereby restricting the efficacy of AD models trained on such data. To address these limitations, we propose a training-free few-shot anomaly generation method, namely O2MAG, which leverages the self-attention in One reference anomalous image to synthesize More realistic anomalies, supporting effective downstream anomaly detection. Specifically, O2MAG manipulates three parallel diffusion processes via self-attention grafting and incorporates the anomaly mask to mitigate foreground-background query confusion, synthesizing text-guided anomalies that closely adhere to real anomalous distributions. To bridge the semantic gap between the encoded anomaly text prompts and the true anomaly semantics, Anomaly-Guided Optimization is further introduced to align the synthesis process with the target anomalous distribution, steering the generation toward realistic and text-consistent anomalies. Moreover, to mitigate faint anomaly synthesis inside anomaly masks, Dual-Attention Enhancement is adopted during generation to reinforce both self- and cross-attention on masked regions. Extensive experiments validate the effectiveness of O2MAG, demonstrating its superior performance over prior state-of-the-art methods on downstream AD tasks.

Abstract

Explore similar work

One-to-More: High-Fidelity Training-Free Anomaly Generation with Attention Control

CORE: Contrast-and-Reflect for Self-Supervised Prompt Optimization in Anomaly Synthesis

UniScale: Arbitrary-Scale Industrial Anomaly Generation