Copy What Is Seen, Generate What Is Not: Training-Free Anomaly-Aware Video Restoration
Authors: Zhida Qu, Shengchao Chen
Organizations: Department of Computer Science and Engineering, Tandon School of Engineering, New York University · Australian AI Institute, University of Technology Sydney
A surveillance system that detects an anomaly often has to repair the footage as well, yet the two tasks are studied in isolation: training-free anomaly detectors stop at a score or a label, while training-free video editing answers to a user prompt rather than to a detector. This paper proposes AVR (Anomaly-aware Video Restoration), which closes that gap with frozen pretrained models alone and generates content only where the clip offers no evidence to copy. Motion evidence first gates open-vocabulary proposals into spatio-temporal masks. A background prior computed from the clip then fills every pixel the anomaly ever uncovers, leaving diffusion to synthesize only what no frame showed, and a frozen verifier decides per clip whether to trust a classical, a prior-anchored, or a background-conditioned restorer. Extensive experiments on three surveillance datasets, under both full-reference anomaly injection and real anomalies, show that AVR leads full-frame fidelity under oracle masks, matches three trained video inpainters inside the edited region, and outperforms a detect-then-generate pipeline on the masks it produces itself, while suppressing both the residual anomaly and the flicker of free diffusion.
Figures & tables
Figure 1: Comparison with existing paradigms. (a) Detection localizes and stops, (b) editing needs a person to supply the region, and (c) chaining the two fills from a generative prior alone. (d) AVR copies every pixel seen elsewhere and generates only what no frame observed.
Method
Region
Content
Train
Temporal model
Detect
LAVAD [ 42 ]
Detector
—
✗
—
SphereVAD [ 14 ]
Detector
—
✗
—
Edit
T2V-Zero [ 16 ]
User
✗
Attention
TokenFlow [ 9 ]
User
✗
Feature prop.
Inpaint
E2FGVI [ 22 ]
Given
✓
Learned prop.
ProPainter [ 45 ]
Given
✓
Learned prop.
Table 1: Comparison of method families along the two inputs a detect-and-restore system requires. Content follows the colour code of Fig. 1 : pixels copied from other frames of the same clip, pixels produced by a generative prior rather than observed.
Figure 2: The AVR pipeline. Step 1 turns a detection into the mask M , keeping open-vocabulary boxes only where motion measured on the clip corroborates them. Step 2 turns M back into video, running three restorers ordered from copying observed pixels to synthesising never-observed ones and letting a frozen verifier pick one per clip. Pixels outside M are untouched, and no component is trained.
CUHK Avenue
UCSD Ped2
ShanghaiTech
Method
PSNR ↑
PSNR reg↑
LPIPS ↓
FVD ↓
PSNR ↑
PSNR reg↑
LPIPS ↓
FVD ↓
PSNR ↑
PSNR reg↑
LPIPS ↓
FVD ↓
Classical (Spatial)
39.76
18.79
0.008
5.86
48.84
27.96
0.004
2.78
52.09
31.22
0.004
1.97
Classical (Flow)
40.36
19.50
0.007
6.12
49.31
28.61
0.004
2.51
51.88
31.26
0.004
2.30
ProPainter [ 45 ]
43.73
25.34
0.004
0.64
50.56
32.08
0.004
0.49
53.88
35.21
0.003
0.64
E2FGVI [ 22 ]
45.97
27.34
0.004
0.57
51.90
33.33
0.004
0.53
55.26
36.41
0.004
0.67
FloED [ 12 ]
45.39
26.19
0.005
0.77
51.00
32.47
0.004
0.51
54.35
35.95
0.004
0.76
Table 2: Restoration under oracle masks. PSNR reg is measured inside the anomaly region, bold for best and underlined for second best.
Benchmark
Method
PSNR ↑
PSNR reg↑
SSIM ↑
LPIPS ↓
tLP ↓
FVD ↓
Warp ↓
RD ↑
IoU ↑
CUHK Avenue
Grounded-SD (base)
29.00
15.78
0.972
0.048
0.036
46.13
1.93
− 1.202
0.097
ProPainter (SAM2 masks)
28.63
13.38
0.974
0.057
0.017
51.32
1.18
− 0.117
0.095
ProPainter (gated masks)
32.52
14.63
0.986
0.035
0.012
28.33
1.09
− 0.201
0.205
E2FGVI (gated masks)
33.34
15.05
0.987
0.033
0.014
28.16
1.13
− 0.238
0.205
FloED (gated masks)
32.72
14.89
0.986
0.038
0.016
29.97
1.15
− 0.271
0.205
AVR (Ours)
33.79
15.00
0.988
0.031
0.013
27.07
1.10
+ 0.123
0.205
Table 3: End-to-end comparison. The base row chains a per-frame detector with unconstrained inpainting, while ProPainter, E2FGVI and FloED restore the same motion-gated masks as AVR, hence identical Mask IoU. Best per column in bold , second best underlined .
Figure 3: End-to-end restoration under detected masks. All restorers share the same motion-gated masks, one median clip per row.
CUHK Avenue
UCSD Ped2
ShanghaiTech
Detector
Restorer
PSNR ↑
RD ↑
PSNR ↑
RD ↑
PSNR ↑
RD ↑
SAM2
Prior-anchored
27.84
− 0.255
34.86
− 2.550
23.90
− 1.413
SAM2
Selection (2 cand.)
28.59
− 0.071
37.25
− 1.220
24.79
− 0.870
Motion-gated
BG-diffusion
33.19
+ 0.145
43.23
− 4.282
39.74
− 0.072
Motion-gated
Selection (3 cand.)
33.79
+ 0.123
44.00
− 0.970
40.08
+ 0.255
Table 4: Ablation study. The SAM2 rows predate the background prior, so their pool holds two restorers instead of three. Bold : the best result, and underlined : the second best.
Figure 4: PSNR that each mechanism adds to the preceding stage under oracle masks, from per-frame diffusion onward.
Figure 5: FVD sensitivity on Avenue to blend λ , detector threshold τ and diffusion steps. Panel (b) is measured end to end while (a) and (c) use oracle masks, which puts its FVD an order of magnitude higher.
CUHK Avenue
ShanghaiTech
Detector
PSNR ↑
LPIPS ↓
FVD ↓
IoU ↑
PSNR ↑
LPIPS ↓
FVD ↓
IoU ↑
Heuristic
31.07
0.035
20.67
0.177
32.86
0.058
34.63
0.334
Grounded-SAM
30.60
0.045
24.79
0.097
29.15
0.251
95.35
0.119
Grounded-SAM2
30.14
0.050
45.36
0.095
26.84
0.326
121.62
0.128
TAO [ 15 ]
32.12
0.038
23.78
0.117
35.59
0.103
25.23
0.257
Motion-gated (ours)
33.19
0.032
20.17
0.205
39.74
0.047
20.47
0.343
Table 5: Detector study with the restoration module held fixed at BG-diffusion. Best per column in bold , second best underlined .
Figure 6: Restorer chosen by the frozen verifier, as a share of the 60 oracle-mask clips available in each benchmark.
Sec/clip
Sec/frame
Peak mem.
Detection (Grounding DINO + SAM)
6.81
0.43
9.2
Motion gate
0.05
< 0.01
Classical
0.21
0.01
Prior-anchored
19.11
1.19
BG-diffusion
20.74
1.30
Verifier
3.72
0.23
Table 6: Inference cost on CUHK Avenue per 16-frame clip on an RTX A5500, under the same motion-gated masks. Clip medians, warm-up excluded, peak memory in GB.
Flicker ↓
Warp Err. ↓
Pipeline
before
after
before
after
RD ↑
impr.
Grounded-SD (base)
12.77
27.71
0.95
3.22
− 0.513
7/20
AVR (Ours)
21.12
3.32
0.95
0.84
+ 0.721
20/20
Table 7: 20 real ShanghaiTech test anomalies, evaluated without a reference. Flicker is measured inside each pipeline’s own detected region, and “impr.” counts clips whose flicker decreased.
Figure 7: Residual drop by anomaly type, end to end on all three benchmarks, with the axis truncated at −1.15 .
Figure 8: Case study on injected anomalies, with ProPainter and AVR restoring identical detected masks. Numbers give the region PSNR in decibels obtained on that one particular clip.
Figure 9: Real anomalies under detected masks. Rows 1–2 show a cyclist with its flicker curve, and row 3 a parked car.
Anomaly generation is often framed as few-shot fine-tuning with anomalous samples, which contradicts the scarcity that motivates generation and tends to overfit category priors. We tackle the setting where no real anomaly samples or training are available. We propose Delta-Denoising (DeltaDeno), a training-free zero-shot anomaly generation method that localizes and edits defects by contrasting two diffusion branches driven by a minimal prompt pair under a shared schedule. By accumulating per-step denoising deltas into an image-specific localization map, we obtain a mask to guide the latent inpainting during later diffusion steps and preserve the surrounding context while generating realistic local defects. To improve stability and control, DeltaDeno performs token-level prompt refinement that aligns shared content and strengthens anomaly tokens, and applies a spatial attention bias restricted to anomaly tokens in the predicted region. Experiments on public datasets show that DeltaDeno achieves great generation, realism and consistent gains in downstream detection performance. Code will be made publicly available at https://github.com/CROVO1026/DeltaDeno.
Chaoran Xu, Chengkan Lv, Qiyu Chen +3
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · CASI Vision Technology CO., LTD., Luoyang, China +1
Progress in video anomaly understanding (VAU) has long been limited by inherent deficiencies of real-world anomaly videos, which are hard to collect and offer little control over their content. Synthetic anomaly approaches partially alleviate data scarcity, yet their generation remains largely controlled at the category or prompt level. They also lack component-level verification of video-text consistency and provide insufficient hard normal samples near the normal-anomaly boundary. To address this, we present AnomalyCraft-700K, a component-controllable synthetic anomaly dataset for fine-grained VAU, containing over 40K videos and over 700K task-level textual annotations. From fine-grained semantic components and a progressive three-stage pipeline, we craft anomaly events that are richly detailed, semantically controlled, and temporally structured, and additionally construct per-category hard normal samples to prompt the model to discriminate based on anomaly semantics rather than surface visual cues. Moreover, using the components as verification units, AnomalyCraft-700K further performs component-wise correction of video-text discrepancies introduced during generation, providing reliable annotations with verified cross-modal alignment for six tasks that progress from anomaly detection, through anomaly retrieval and captioning, to fine-grained anomaly reasoning. Evaluations of widely used methods under both traditional and MLLM-based protocols demonstrate that AnomalyCraft-700K serves as an effective source of supervision, from anomaly detection to fine-grained anomaly understanding.
Yuzhou Long, Haodong Zhang, Yunpeng Yang +2
Northwestern Polytechnical University · Singapore Management University
Existing Video Anomaly Detection (VAD) methods typically rely on task-specific training, leading to strong domain dependency and high training costs. Moreover, most existing methods output only scalar anomaly scores, providing limited insight into why specific events are considered abnormal. Recent advances in Vision-Language Models (VLMs) have enabled both anomaly detection and human-interpretable reasoning. However, many VLM-based approaches still require additional training steps (e.g., instruction tuning or verbalized learning) or external Large Language Models (LLMs), incurring further training costs and inference overhead. To address these challenges, we propose CoReVAD, a contextual reasoning framework for training-free video anomaly detection that operates with a single frozen VLM. CoReVAD directly generates anomaly scores and temporal descriptions from the VLM. To mitigate noise in generative outputs, we introduce a Local Response Cleaning (LRC) module based on local vision-text alignment. Furthermore, global temporal context and progression are incorporated through softmax-based refinement, Gaussian smoothing, and position weighting. Experiments on UCF-Crime and XD-Violence demonstrate that CoReVAD achieves competitive performance among training-free methods while providing reliable and interpretable explanations. Our official code is available at: https://github.com/Muk-00/CoReVAD
Hyeongmuk Lim, Youngbum Hur
Department of Industrial Engineering, Inha University, Incheon, Republic of Korea