Copy What Is Seen, Generate What Is Not: Training-Free Anomaly-Aware Video Restoration

Zhida Qu; Shengchao Chen

cs.CVSep 16, 2026

Copy What Is Seen, Generate What Is Not: Training-Free Anomaly-Aware Video Restoration

Authors: Zhida Qu, Shengchao Chen

Organizations: Department of Computer Science and Engineering, Tandon School of Engineering, New York University · Australian AI Institute, University of Technology Sydney

Abstract

A surveillance system that detects an anomaly often has to repair the footage as well, yet the two tasks are studied in isolation: training-free anomaly detectors stop at a score or a label, while training-free video editing answers to a user prompt rather than to a detector. This paper proposes AVR (Anomaly-aware Video Restoration), which closes that gap with frozen pretrained models alone and generates content only where the clip offers no evidence to copy. Motion evidence first gates open-vocabulary proposals into spatio-temporal masks. A background prior computed from the clip then fills every pixel the anomaly ever uncovers, leaving diffusion to synthesize only what no frame showed, and a frozen verifier decides per clip whether to trust a classical, a prior-anchored, or a background-conditioned restorer. Extensive experiments on three surveillance datasets, under both full-reference anomaly injection and real anomalies, show that AVR leads full-frame fidelity under oracle masks, matches three trained video inpainters inside the edited region, and outperforms a detect-then-generate pipeline on the masks it produces itself, while suppressing both the residual anomaly and the flicker of free diffusion.

Figures & tables

Figure 1: Comparison with existing paradigms. (a) Detection localizes and stops, (b) editing needs a person to supply the region, and (c) chaining the two fills from a generative prior alone. (d) AVR copies every pixel seen elsewhere and generates only what no frame observed.

	Method	Region	Content	Train	Temporal model
Detect	LAVAD [ 42 ]	Detector	—	✗	—
Detect	SphereVAD [ 14 ]	Detector	—	✗	—
Edit	T2V-Zero [ 16 ]	User		✗	Attention
Edit	TokenFlow [ 9 ]	User		✗	Feature prop.
Inpaint	E2FGVI [ 22 ]	Given		✓	Learned prop.
	ProPainter [ 45 ]	Given		✓	Learned prop.

Table 1: Comparison of method families along the two inputs a detect-and-restore system requires. Content follows the colour code of Fig. 1 : pixels copied from other frames of the same clip, pixels produced by a generative prior rather than observed.

Figure 2: The AVR pipeline. Step 1 turns a detection into the mask $M$ , keeping open-vocabulary boxes only where motion measured on the clip corroborates them. Step 2 turns $M$ back into video, running three restorers ordered from copying observed pixels to synthesising never-observed ones and letting a frozen verifier pick one per clip. Pixels outside $M$ are untouched, and no component is trained.

	CUHK Avenue				UCSD Ped2				ShanghaiTech
Method	PSNR $\uparrow$	PSNR ${}_{\text{reg}}\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	PSNR ${}_{\text{reg}}\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	PSNR ${}_{\text{reg}}\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
Classical (Spatial)	39.76	18.79	0.008	5.86	48.84	27.96	0.004	2.78	52.09	31.22	0.004	1.97
Classical (Flow)	40.36	19.50	0.007	6.12	49.31	28.61	0.004	2.51	51.88	31.26	0.004	2.30
ProPainter [ 45 ]	43.73	25.34	0.004	0.64	50.56	32.08	0.004	0.49	53.88	35.21	0.003	0.64
E2FGVI [ 22 ]	45.97	27.34	0.004	0.57	51.90	33.33	0.004	0.53	55.26	36.41	0.004	0.67
FloED [ 12 ]	45.39	26.19	0.005	0.77	51.00	32.47	0.004	0.51	54.35	35.95	0.004	0.76

Table 2: Restoration under oracle masks. PSNR

{}_{\text{reg}}

is measured inside the anomaly region, bold for best and underlined for second best.

Benchmark	Method	PSNR $\uparrow$	PSNR ${}_{\text{reg}}\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	tLP $\downarrow$	FVD $\downarrow$	Warp $\downarrow$	RD $\uparrow$	IoU $\uparrow$
CUHK Avenue	Grounded-SD (base)	29.00	15.78	0.972	0.048	0.036	46.13	1.93	$-$ 1.202	0.097
	ProPainter (SAM2 masks)	28.63	13.38	0.974	0.057	0.017	51.32	1.18	$-$ 0.117	0.095
	ProPainter (gated masks)	32.52	14.63	0.986	0.035	0.012	28.33	1.09	$-$ 0.201	0.205
	E2FGVI (gated masks)	33.34	15.05	0.987	0.033	0.014	28.16	1.13	$-$ 0.238	0.205
	FloED (gated masks)	32.72	14.89	0.986	0.038	0.016	29.97	1.15	$-$ 0.271	0.205
	AVR (Ours)	33.79	15.00	0.988	0.031	0.013	27.07	1.10	$\bm{+}$ 0.123	0.205

Table 3: End-to-end comparison. The base row chains a per-frame detector with unconstrained inpainting, while ProPainter, E2FGVI and FloED restore the same motion-gated masks as AVR, hence identical Mask IoU. Best per column in bold , second best underlined .

Figure 3: End-to-end restoration under detected masks. All restorers share the same motion-gated masks, one median clip per row.

		CUHK Avenue		UCSD Ped2		ShanghaiTech
Detector	Restorer	PSNR $\uparrow$	RD $\uparrow$	PSNR $\uparrow$	RD $\uparrow$	PSNR $\uparrow$	RD $\uparrow$
SAM2	Prior-anchored	27.84	$-$ 0.255	34.86	$-$ 2.550	23.90	$-$ 1.413
SAM2	Selection (2 cand.)	28.59	$-$ 0.071	37.25	$-$ 1.220	24.79	$-$ 0.870
Motion-gated	BG-diffusion	33.19	$\bm{+}$ 0.145	43.23	$-$ 4.282	39.74	$-$ 0.072
Motion-gated	Selection (3 cand.)	33.79	$+$ 0.123	44.00	$\bm{-}$ 0.970	40.08	$\bm{+}$ 0.255

Table 4: Ablation study. The SAM2 rows predate the background prior, so their pool holds two restorers instead of three. Bold : the best result, and underlined : the second best.

Figure 4: PSNR that each mechanism adds to the preceding stage under oracle masks, from per-frame diffusion onward.

Figure 5: FVD sensitivity on Avenue to blend

\lambda

, detector threshold

\tau

and diffusion steps. Panel (b) is measured end to end while (a) and (c) use oracle masks, which puts its FVD an order of magnitude higher.

	CUHK Avenue				ShanghaiTech
Detector	PSNR $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	IoU $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	IoU $\uparrow$
Heuristic	31.07	0.035	20.67	0.177	32.86	0.058	34.63	0.334
Grounded-SAM	30.60	0.045	24.79	0.097	29.15	0.251	95.35	0.119
Grounded-SAM2	30.14	0.050	45.36	0.095	26.84	0.326	121.62	0.128
TAO [ 15 ]	32.12	0.038	23.78	0.117	35.59	0.103	25.23	0.257
Motion-gated (ours)	33.19	0.032	20.17	0.205	39.74	0.047	20.47	0.343

Table 5: Detector study with the restoration module held fixed at BG-diffusion. Best per column in bold , second best underlined .

Figure 6: Restorer chosen by the frozen verifier, as a share of the 60 oracle-mask clips available in each benchmark.

	Sec/clip	Sec/frame	Peak mem.
Detection (Grounding DINO $+$ SAM)	6.81	0.43	9.2
Motion gate	0.05	$<$ 0.01
Classical	0.21	0.01
Prior-anchored	19.11	1.19
BG-diffusion	20.74	1.30
Verifier	3.72	0.23

Table 6: Inference cost on CUHK Avenue per 16-frame clip on an RTX A5500, under the same motion-gated masks. Clip medians, warm-up excluded, peak memory in GB.

Pipeline	before	after	before	after	RD $\uparrow$	impr.
	Flicker $\downarrow$		Warp Err. $\downarrow$
Grounded-SD (base)	12.77	27.71	0.95	3.22	$-$ 0.513	7/20
AVR (Ours)	21.12	3.32	0.95	0.84	$\bm{+}$ 0.721	20/20

Table 7: 20 real ShanghaiTech test anomalies, evaluated without a reference. Flicker is measured inside each pipeline’s own detected region, and “impr.” counts clips whose flicker decreased.

Figure 7: Residual drop by anomaly type, end to end on all three benchmarks, with the axis truncated at

-1.15

Figure 8: Case study on injected anomalies, with ProPainter and AVR restoring identical detected masks. Numbers give the region PSNR in decibels obtained on that one particular clip.

Figure 9: Real anomalies under detected masks. Rows 1–2 show a cyclist with its flicker curve, and row 3 a parked car.

Video Restoration

Video restoration reconstructs clearer, more faithful, temporally consistent footage from blur, noise, compression, weather, and other degradations.

See all 59 papers

DeltaDeno: Zero-Shot Anomaly Generation via Delta-Denoising Attribution
Nov 21, 2025Chaoran Xu, Chengkan Lv, Qiyu Chen +3Diffusion-Based Image Generation Synthetic Anomaly Generation
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · CASI Vision Technology CO., LTD., Luoyang, China +1
AnomalyCraft-700K: Component-Level Controllable and Verifiable Synthetic Anomalies for Fine-Grained Video Anomaly Understanding
Sep 7, 2026Yuzhou Long, Haodong Zhang, Yunpeng Yang +2Synthetic Anomaly Generation Video Anomaly Detection
Northwestern Polytechnical University · Singapore Management University
CoReVAD: A Contextual Reasoning Framework for Training-Free Video Anomaly Detection
May 22, 2026Hyeongmuk Lim, Youngbum HurInterpretable Anomaly Detection VLM Reasoning
Department of Industrial Engineering, Inha University, Incheon, Republic of Korea

Nov 21, 2025J/K move · Enter open · S save

DeltaDeno: Zero-Shot Anomaly Generation via Delta-Denoising Attribution

Chaoran Xu, Chengkan Lv, Qiyu Chen, Yunkang Cao+2

Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · CASI Vision Technology CO., LTD., Luoyang, China+1

Diffusion-Based Image Generation Synthetic Anomaly Generation Diffusion Models

Open paper AskPDF arXiv

Anomaly generation is often framed as few-shot fine-tuning with anomalous samples, which contradicts the scarcity that motivates generation and tends to overfit category priors. We tackle the setting where no real anomaly samples or training are available. We propose Delta-Denoising (DeltaDeno), a training-free zero-shot anomaly generation method that localizes and edits defects by contrasting two diffusion branches driven by a minimal prompt pair under a shared schedule. By accumulating per-step denoising deltas into an image-specific localization map, we obtain a mask to guide the latent inpainting during later diffusion steps and preserve the surrounding context while generating realistic local defects. To improve stability and control, DeltaDeno performs token-level prompt refinement that aligns shared content and strengthens anomaly tokens, and applies a spatial attention bias restricted to anomaly tokens in the predicted region. Experiments on public datasets show that DeltaDeno achieves great generation, realism and consistent gains in downstream detection performance. Code will be made publicly available at https://github.com/CROVO1026/DeltaDeno.

Abstract

Video Restoration

Explore similar work

DeltaDeno: Zero-Shot Anomaly Generation via Delta-Denoising Attribution

AnomalyCraft-700K: Component-Level Controllable and Verifiable Synthetic Anomalies for Fine-Grained Video Anomaly Understanding

CoReVAD: A Contextual Reasoning Framework for Training-Free Video Anomaly Detection