Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Dissemination

Shuo Liang; Yixing Ma; Pengfei Zhou; Zhenglin Wan; Xingyan Chen; Zihan Mei; Manting Li; Feihan Chen

cs.CVAug 14, 2026

Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Dissemination

Authors: Shuo Liang, Yixing Ma, Pengfei Zhou, Zhenglin Wan, Xingyan Chen, Zihan Mei, Manting Li, Feihan Chen,

Organizations: National University of Singapore · Xidian University · University of California, Berkeley · Arizona State University · Renmin University of China · Monash University · InfRec, Cardinal AI Lab · University of Wisconsin–Madison · Shanghai Jiao Tong University · Stanford University · ETH Zurich · University of Illinois Urbana–Champaign · Xi’an Jiaotong University · Carnegie Mellon University · Alaya Lab · Independent Researcher · Zhejiang University · The Hong Kong University of Science and Technology · GigaAI

Abstract

Recent video generators can fabricate realistic depictions of wars, disasters, public emergencies, and other real-world crises, creating substantial risks of misinformation. Existing benchmarks, however, provide limited evidence on detector and generator behavior in such settings, including how detectability varies with generation conditions, how people perceive generated videos, and whether detectors remain reliable during social dissemination. To address this gap, we introduce RA-Bench, a benchmark for AI-generated video detection that uses Real videos as Anchors. RA-Bench contains 17,886 videos, comprising 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from four open-source and five closed-source generators. Based on RA-Bench, we organize our evaluation along three dimensions. We first assess detector generalization across seven traditional detectors, ten zero-shot multimodal models under three review settings, and two MLLMs specifically fine-tuned on AI-generated video detection. Across these methods, none of the three detector families generalizes consistently across RA-Bench instances. We then examine how detectability varies with generation quality, conditioning information, and sampling seeds. These analyses show that generation properties affect detector families differently, while source-level detection patterns remain stable across seeds. Finally, we study human authenticity judgments and detector reliability during social dissemination. We find that videos that mislead people are also difficult for current detectors, and that social dissemination makes detection harder. Together, these findings show that current methods struggle to detect realistic AI-generated videos, highlighting the need for detectors robust to evolving video generators.

Figures & tables

Benchmark	Real-event grounding	Social-risk taxonomy	Real-event- conditioned I2V	Human-deceptive challenge set	Social dissemination simulation
GVF ( Ma et al., 2025 )	✗	✗	✗	✗	✗
GenVideo ( Chen et al., 2024 )	✗	✗	✗	✗	$\bigcirc$
GenVidBench ( Ni et al., 2026 )	✗	✗	$\bigcirc$	✗	✗
GenBuster-Bench ( Wen et al., 2025a )	✗	✗	✗	✗	$\bigcirc$
GenWorld ( Chen et al., 2025a )	✗	✗	$\bigcirc$	✗	✗
AIGVDBench ( Ma et al., 2026 )	✗	✗	✗	✗	✗

Table 1: Comparison of benchmark designs for AI-generated video detection. We compare how existing benchmarks handle real-event sources, social-risk categories, real-event-conditioned generation, human deception, and social dissemination simulations during dataset construction and evaluation. RA-Bench also includes recent open- and closed-source video generators and representative detector families.

Figure 2: Overview of the five-stage RA-Bench construction pipeline. We collect public videos from real-world crises and other socially consequential events, segment them into scene-level clips and screen near-duplicates, conduct two-round manual review, and apply release-oriented postprocessing to form the real anchor set. Each retained anchor is then paired with image-to-video (I2V) generated counterparts using first-frame conditioning and a shared prompting pipeline.

Figure 4: A real anchor and its generated counterpart for a representative RA-Bench crisis event. The top filmstrip shows the real space-launch anchor, and the bottom shows the generated video, each represented by six uniformly sampled frames ( f0 – f5 ). The red dashed box marks the shared first frame used to condition the generator. The generated video continues from this frame, preserving much of the scene layout, capture style, and lighting while synthesizing new motion as the rocket leaves the pad. This example uses the dynamic-duration Wan2.2 setting. Additional examples across crisis categories and generators are provided in Figures 12 and 13 .

		Public	Open-source settings					Open	Closed-source generators					Closed
Detector	Metric	High ref.	Wan2.2 dyn.	Wan2.2 fixed ∗	Wan2.2 Light.	LTX	Omni Weav.	avg.	Happy Horse	Runway	Kling	Seedance 2.0	Hailuo	avg.
Image / frame-level detectors
CNNSpot	AUC	81.4	35.5	34.6	43.8	64.0	45.8	47.3	39.2	31.9	45.4	40.4	53.9	42.2
	T@5%	–	2.3	1.7	3.0	11.7	3.5	5.1	2.8	1.6	3.9	2.8	5.8	3.4
NPR	AUC	67.6	50.6	49.9	55.2	58.6	50.9	53.8	54.6	52.3	44.3	48.7	54.6	50.9
	T@5%	–	3.8	4.0	4.6	4.5	5.6	4.6	4.5	3.4	3.2	3.9	5.7	4.2

Table 2: Traditional detector performance across the nine RA-Bench generation sources We report paired AUC and TPR at 5% FPR (T@5%) in %. T@5% is generated-video recall at an operating point with 5% FPR on the matched real videos. The public-reference column reports AIGVDBench LTX-I2V AUC for six detectors and VidProM AUC for ReStraV; these are contextual references rather than matched-domain baselines. Wan2.2 fixed ∗ is an auxiliary control and is excluded from the Open average. Spearman correlations use only the six detectors that share the AIGVDBench reference.

Figure 5: Source-specific AUC profiles for the seven traditional detectors. Each spoke represents one detector, and each polygon represents one generation setting. The outer dashed contour shows the public-reference result for each detector (AIGVDBench LTX-I2V for six detectors and VidProM for ReStraV), while the faint inner ring marks 50% AUC. (a) The four open-source generators in RA-Bench and the fixed-duration Wan2.2 control. (b) The five closed-source generators in RA-Bench.

Figure 6: Effect of temporal allocation under a fixed eight-frame budget. (a) AUC change from Uniform-8 to Global–Local-8 for each of five sparse-frame detectors, averaged over eight fixed-length RA-Bench sources and the fixed-duration Wan2.2 control; the diamond and interval show the five-detector mean and source-block bootstrap 95% confidence interval. (b) The corresponding five-detector mean for each evaluated setting. The ablation excludes dynamic-duration Wan2.2. Positive values favor Global–Local-8.

		Open-source settings					Open	Closed-source generators					Closed
Model	Prompt	Wan2.2 dyn.	Wan2.2 fixed ∗	Wan2.2 Light.	LTX	Omni Weav.	avg.	Happy Horse	Runway	Kling	Seedance 2.0	Hailuo	avg.
Discrete classification prompts (BAcc/macro-F1)
Qwen3.5-27B	Binary	51.3 43.8	51.2 43.6	55.3 49.9	59.7 56.2	54.8 49.2	55.3 49.8	51.6 44.2	55.3 50.0	50.0 41.4	49.8 41.7	51.4 43.9	51.6 44.2
	Diagnostic	50.9 36.5	50.7 36.1	51.8 38.4	53.3 41.2	51.7 38.2	51.9 38.6	50.8 36.4	50.9 36.5	50.5 35.6	50.2 35.0	51.1 36.9	50.7 36.1
Qwen3.5-122B -A10B	Binary	51.6 43.5	50.7 41.9	52.7 45.3	58.3 53.8	54.1 47.5	54.2 47.5	51.8 43.8	50.2 41.1	53.4 46.4	52.4 44.9	52.3 44.6	52.0 44.2
	Diagnostic	54.1 50.2	53.9 49.8	55.7 52.3	65.6 64.7	60.2 58.2	58.9 56.3	51.6 46.5	51.7 46.6	52.1 47.1	49.7 44.6	52.2 47.5	51.5 46.4

Table 3: Source-specific zero-shot multimodal model results on the nine RA-Bench generation sources. Binary and Diagnostic cells report BAcc/macro-F1, while Rating cells report paired AUC/verdict macro-F1 (top/bottom). Values are in %. Open and Closed are source-equal averages over the four open-source and five closed-source RA-Bench generators, respectively; Wan2.2 fixed ∗ is excluded from the Open average.

Figure 7: Qwen3.5 scaling across prompts and generation sources. The four columns correspond to representative RA-Bench generators. The top row reports BAcc for Binary and Diagnostic and paired AUC for Rating; the bottom row reports macro-F1 from each prompt’s explicit Overall Verdict. Horizontal dashed lines in the top row mark the 50% reference level. The vertical dotted line separates dense and MoE variants, whose trajectories are drawn separately. Solid and dashed lines denote open-source and closed-source generators, respectively.

		Open-source settings					Open	Closed-source generators					Closed
Prompt	Metric	Wan2.2 dyn.	Wan2.2 fixed ∗	Wan2.2 Light.	LTX	Omni Weav.	avg.	Happy Horse	Runway	Kling	Seed ance2.0	Hailuo	avg.
Skyra-SFT (RealR: 87.4 / 55.4)
Official timestamp	BAcc	60.6	91.9	59.9	74.7	51.9	61.8	80.2	80.3	80.1	73.5	54.9	73.8
	FakeR	33.8	96.4	32.5	61.9	16.4	36.1	73.3	73.2	72.8	59.6	22.5	60.3
	macro-F1	57.5	91.9	56.7	74.2	45.0	58.4	80.1	80.2	80.0	73.0	49.6	72.6
Frame index	BAcc	60.2	59.1	58.7	31.1	50.9	50.2	59.4	61.0	59.1	52.8	56.0	57.7

Table 4: Fine-tuned MLLMs across the nine RA-Bench generation sources. Skyra prompts receive the same 16 frames; BusterX++ ( Wen et al., 2025b ) uses its released pipeline. Each source is evaluated on matched real–generated pairs, with BAcc, FakeR, and macro-F1 reported in %. The RealR values beside each model name are measured on the full set of 1,830 real anchors and are listed in official-timestamp/frame-index order for Skyra. Open and Closed are source-equal averages over the four open-source and five closed-source RA-Bench generators, respectively; Wan2.2 fixed ∗ is excluded from Open.

Figure 8: Skyra performance changes with temporal-tag representation. The official-timestamp and frame-index prompts receive the same 16 frames. Each radar reports per-source BAcc across the nine RA-Bench generators and Wan2.2 fixed ∗ , an auxiliary control excluded from benchmark-level averages; the dashed ring marks 50%.

Figure 9: Generation quality and fake-side detectability on RA-Bench. (a) Associations of the VBench-I2V Quality Score, Condition Fidelity, and Combined Quality with three Gemini-3.1-Pro-Preview outputs and the mean fake-score percentile of seven traditional detectors. (b) Dimension-level associations for Gemini Diagnostic and the traditional-detector mean; complete prompt-wise results are reported in Appendix C.1 . (c) Dynamic-minus-static contrasts after adjustment for generation source and matched real-video anchor. Panels (a) and (b) report adjusted changes associated with an interquartile increase within each source and Dynamic Degree group. Negative values indicate weaker fake evidence at higher quality, while positive values in panel (c) indicate stronger fake evidence for dynamic clips. Traditional scores are percentile-normalized within detector and source before averaging. Error bars denote 95% confidence intervals based on standard errors clustered by real-video anchor.

Detector / configuration	Metric	T2V	First frame (I2V; RA-Bench)	First+last frames (I2V)
Traditional detectors
CNNSpot	AUC T@5%	15.9 0.2	35.5 2.4	40.5 3.1
NPR	AUC T@5%	30.7 0.9	50.6 3.8	44.4 3.0
UnivFD	AUC T@5%	16.3 0.1	32.7 0.4	34.5 0.6
ForgeLens	AUC T@5%	24.2 1.0	60.0 19.0	44.8 6.8
DeCoF	AUC T@5%	66.6 1.5	62.5 3.9	68.5 6.8

Table 5: Detection under three Wan2.2 generation settings. We compare T2V, first-frame I2V (the RA-Bench setting), and first+last-frame I2V on the same 1,830 anchor-derived prompts at seed 0. Traditional detectors report AUC with T@5% (TPR at 5% FPR) in gray below; Skyra-SFT, Skyra-RL, and BusterX++ report FakeR. All values are percentages. BusterX++ abstentions are counted as incorrect.

Figure 10: Changes in detectability across Wan2.2 generation settings. Each panel reports the second setting minus the first setting named in its title. Panels (a) and (c) show paired AUC for the seven traditional detectors, while panels (b) and (d) show FakeR for the five fine-tuned MLLM settings. Positive values indicate higher AUC or FakeR in the second setting; negative values indicate lower values. Diamonds denote detector-family means.

Configuration	RA-Bench-HumanProof				Source-matched RA-Bench BAcc/FakeR or AUC/T@5%
	Discrete output		Continuous score
	BAcc	FakeR	AUC	T@5%
Traditional detectors
CNNSpot	–	–	40.5	3.6	43.5 / 3.8
NPR	–	–	50.3	3.8	50.4 / 4.0
UnivFD	–	–	39.4	0.6	39.8 / 1.0

Table 8: Human-deceptive videos remain difficult for current detectors. RA-Bench-HumanProof contains 633 generated videos, each paired with its matched real anchor. Discrete-output methods report BAcc and FakeR, while continuous-score methods report paired AUC and T@5%. The final column reports BAcc/FakeR for discrete outputs and AUC/T@5% for continuous scores on full RA-Bench after weighting its source-specific results by the RA-Bench-HumanProof source proportions. All values are percentages. Skyra official-timestamp results follow the released protocol; frame index is the timestamp-free control.

Figure 11: Detection under the RA-Bench-LastMile social dissemination simulation. Original retains the standardized clips; T1 denotes VP9-to-H.264 transcoding, T2 denotes

0.5\times

spatial downsampling, T3 denotes conversion to 8 fps, and T4 denotes a synthetic news badge. T1+T2, T1+T3, and T1+T4 isolate each added operation, while Full combines T1–T4. (a–b) Mean AUC and T@5% over the seven traditional detectors. (c–d) BAcc and FakeR for the fine-tuned MLLM detectors. Gray extensions show the loss from each configuration’s Original result, hatched bars denote frame-index prompts, and the red dashed line gives the mean over the five fine-tuned configurations. Results are equal-source means over the nine RA-Bench generators.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

L2	Event scope	#
L1-01 Weather and natural disasters (75)
L2-01a	Windstorms, hurricanes, and tornadoes	17
L2-01b	Wildfire spread	15
L2-01c	Earthquakes, building damage, and post-disaster street scenes	23
L2-01d	Flooding	20
L1-02 War and armed conflict (60)

Appendix

Table 9: L2 taxonomy and source distribution of RA-Bench. Counts refer to the 675 collected source videos before scene segmentation and manual review. L1 subtotals are shown in parentheses.

Source	#	Rights basis
Government & public institutions (348)
DVIDS	240	US public domain
The White House	22	Open / free license
NTSB	16	Platform terms
EU institutions (Council, Parliament, Commission)	38	Official reuse terms
NASA (incl. SVS)	14	Public domain / official reuse

Appendix

Table 10: Source inventory and recorded rights basis. Counts refer to source videos before scene segmentation. Combined entries (e.g., “public domain / official reuse”) indicate that the aggregated row contains sources with different recorded rights bases.

Field	Content
Core visual fields
short_caption	one-sentence summary
main_object_caption	foreground subject(s)
background_caption	scene / setting
camera_caption	shot type and camera motion
style_caption	capture / visual style

Appendix

Table 11: Stage A structured caption schema. The six core fields describe visible content; the auxiliary fields support prompt construction, uncertainty tracking, and downstream claim-level analysis.

Source	Execution string / request setting	Output res.	fps	Temporal setting	$n$
Open-source benchmark generators
Wan2.2 dynamic	Wan2.2-I2V-A14B	832 $\times$ 480	16	$4n{+}1$ frames ( $\geq$ 33)	1,830
Wan2.2-Lightning	Wan2.2-Lightning- I2V-A14B-NFE4-V1	1280 $\times$ 720	16	$4n{+}1$ frames ( $\geq$ 33)	1,830
LTX	ltx-2.3-22b-dev	1536 $\times$ 1024	24	$8n{+}1$ frames ( $\geq$ 33)	1,830
OmniWeaving	HY-OmniWeaving	848 $\times$ 480	16	$4n{+}1$ frames ( $\geq$ 33)	1,830
Closed-source benchmark generators

Appendix

Table 12: Generation sources and realized outputs. Execution strings and request settings are taken from the recorded generation manifests.

n

counts clips in the primary paired benchmark set; additional seed repeats and the Wan2.2 fixed-duration control are excluded from the 16,056 benchmark clips.

Source	Conditioning size	Steps	Sampler / inference setting
Wan2.2 dynamic/fixed	output resolution	40	UniPC; shift 5.0; guidance (3.5, 3.5)
Wan2.2-Lightning	output resolution	4	Euler + 4-step LoRA; guidance (1.0, 1.0)
LTX	768 $\times$ 512	30	distilled LoRA 0.8; cfg/STG/rescale 3.0/1.0/0.7
OmniWeaving	output resolution	30	480p, 16:9

Appendix

Table 13: Internal inference settings for locally executed sources. Provider APIs do not expose corresponding sampler-level controls. The dynamic and fixed Wan2.2 settings differ only in their temporal policy.

Figure 12: Qualitative coverage across all ten social-risk domains. Each panel pairs a real anchor (top) with one generated counterpart (bottom); each continuation starts from the highlighted conditioning frame, retains the event-specific scene appearance, and introduces new motion and content.

Figure 13: Cross-generator comparison under matched conditioning. The top row is a real space-launch anchor; subsequent rows show Wan2.2 dynamic, its auxiliary fixed-duration control, Wan2.2-Lightning, LTX, and OmniWeaving under the same prompt and first-frame reference. The outputs preserve the initial launch-pad layout but differ in subsequent motion and visual drift.

Method	Visual input / temporal sampling	Evaluation output
Traditional detectors
CNNSpot, NPR, UnivFD, ForgeLens	Eight uniformly spaced full-video frames; official ImageNet or CLIP normalization	Per-frame fake probabilities averaged into one clip score
DeCoF	Eight frames within the first 32 decoded frames; center crop and $224\!\times\!224$ resize	Softmax fake probability from the eight-frame stack
D3	One 3 s window sampled at 8 fps; 16 frames when available; 10% center crop and $224\!\times\!224$ resize	Continuous dis_2nd_std score
ReStraV	Center 2 s window; 24 frames resized to $224\!\times\!224$ ; DINOv2 ViT-S/14 features	Temporal-geometry MLP score, $\texttt{prob\_fake}=1-\texttt{prob\_real}$
Zero-shot multimodal models

Appendix

Table 14: Detector inputs and evaluated outputs. Unless a control is explicitly noted, visual preprocessing and temporal sampling follow the released implementations. The table reports the visual evidence presented to each method and the output used to compute the common metrics; prompt-specific fields and interventions are detailed in the corresponding zero-shot and fine-tuned appendices.

Detector	Public AUC	Public rank	RA-Bench AUC	RA-Bench rank	$\bm{\Delta}$ Rank
ForgeLens	92.9	1	61.1	1	0
UnivFD	89.9	2	40.7	6	$-4$
DeCoF	81.5	3	60.9	2	$+1$
CNNSpot	81.4	4	44.4	4	0
D3	77.7	5	43.5	5	0
NPR	67.6	6	52.2	3	$+3$

Appendix

Table 15: Public detector rankings do not transfer to RA-Bench. Public AUC and rank use the shared AIGVDBench LTX-I2V reference. RA-Bench AUC is the source-equal mean over the nine benchmark generation sources; the fixed-duration Wan2.2 control is excluded.

\Delta

Rank is public rank minus RA-Bench rank, so a positive value denotes an improved relative rank on RA-Bench. AUC values are in %.

	Open-source generators			Closed-source generators
Detector	T@1% $\uparrow$	T@5% $\uparrow$	F@95% $\downarrow$	T@1% $\uparrow$	T@5% $\uparrow$	F@95% $\downarrow$
Image / frame-level detectors
CNNSpot	0.5	5.1	91.1	0.2	3.4	92.4
NPR	0.7	4.6	91.1	0.6	4.2	93.3
UnivFD	0.2	1.0	95.8	0.2	1.1	96.6
ForgeLens	5.3	18.3	84.7	2.5	9.7	85.3

Appendix

Table 16: Traditional detectors remain weak at both low-FPR and high-recall operating points. Cells are source-equal means over the four open-source and five closed-source generators in RA-Bench, in %. Higher is better for T@1% and T@5%; lower is better for F@95%. The Wan2.2 fixed-duration control is excluded. Bold marks the best value within each source group.

Figure 14: Operating-point variability across generation sources. Each filled point is one detector–source pair; horizontal segments span the four open-source or five closed-source generators in RA-Bench, and hollow markers denote source-equal means. The Wan2.2 fixed-duration control is excluded. (a) TPR at 5% FPR. (b) FPR at 95% TPR. Dashed lines mark random-ranking values, and shaded regions indicate worse-than-random operating points.

Prompt	Requested output	Primary metric	Additional analysis
Binary	Overall Verdict	BAcc / macro-F1	FakeR and RealR
Rating	0–100 rating and verdict	Paired rating AUC	Verdict macro-F1, FakeR, and RealR
Diagnostic	Five 0–100 aspect scores and verdict	Verdict BAcc / macro-F1	Per-aspect paired AUC and exact-five collapse

Appendix

Table 17: Output fields and metrics for the three zero-shot prompts. Binary and Diagnostic use the explicit Overall Verdict for their primary classification metrics. Rating uses the continuous score for paired AUC; its verdict and the Diagnostic aspect scores are analyzed separately.

Figure 16: Model-specific use of the 0–100 Rating scale. Gray-blue and coral markers show source-equal means for matched real and generated videos; the right column reports paired AUC. Large differences in score location across models do not imply corresponding differences in separability.

Figure 17: Separability and collapse of the five Diagnostic scores. Panel (a) reports source-equal paired AUC for each requested dimension. Panel (b) reports the fraction of complete outputs that assign exactly the same value to all five dimensions. Panel (c) reports BAcc from the explicit Overall Verdict.

Model	Temporal label	Full RealR	FakeR	BAcc	$\bm{\Delta}_{\mathbf{5}}$
Skyra-SFT	official timestamp	87.4	49.6	68.5	$+48.6$
	frame index	55.4	53.6	54.4	$+1.5$
Skyra-RL	official timestamp	84.0	55.1	69.5	$+43.4$
	frame index	49.5	60.6	54.9	$+1.4$

Appendix

Table 19: Temporal-label controls for Skyra. FakeR and BAcc are source-equal results over the nine RA-Bench generation sources, whereas Full RealR is measured on all 1,830 real anchors; Wan2.2 fixed ∗ is excluded from the benchmark averages.

\Delta_{5}

is the source-equal fake-recall difference between exact-5-second and other clips over the eight RA-Bench generation sources containing both strata. Values are in %, and

\Delta_{5}

is in percentage points.

	Count		Skyra-SFT				Skyra-RL
			Timestamp		Frame index		Timestamp		Frame index
Source	5	other	5	other	5	other	5	other	5	other
Open-source generators
Wan2.2 dynamic	74	1,756	97.3	31.1	63.5	65.1	98.6	35.6	70.3	70.6
Wan2.2 fixed ∗	1,830	0	96.4	–	62.8	–	97.7	–	68.4	–
Wan2.2-Lightning	74	1,756	94.6	29.8	66.2	61.8	97.3	34.8	70.3	67.6

Appendix

Table 20: Fake recall by final temporal tag. Results cover the nine benchmark sources and the auxiliary Wan2.2 fixed-duration control. 5 denotes clips whose final official timestamp token is exactly

[\mathrm{T}{=}5.00\,\mathrm{s}]

; other denotes all remaining clips. Wan2.2 fixed ∗ contains only the 5-second stratum, whereas Hailuo contains no exact-5-second clip. Replacing timestamps with frame indices removes the large 5-versus-other gap on all eight RA-Bench generation sources containing both strata.

Figure 18: Exact-5-second fake-recall gap under the Skyra prompt ablation. For each of the eight RA-Bench generation sources containing both strata, points show

\Delta_{5}=R_{5\mathrm{s}}-R_{\mathrm{other}}

. Replacing timestamps with frame indices reduces the source-equal mean gap from 48.6 to 1.5 points for Skyra-SFT and from 43.4 to 1.4 points for Skyra-RL. Both prompts receive the same sampled frames. The auxiliary Wan2.2 fixed ∗ control and Hailuo are excluded because each contains only one stratum.

Generation source	$N$	Quality Score	Condition Fidelity	Combined Quality	I2V-S	I2V-B
Open-source models
Wan2.2 dynamic	1,830	76.2	94.0	85.1	94.4	95.9
Wan2.2-Lightning	1,830	76.4	96.4	86.4	96.8	97.4
LTX	1,830	74.9	92.5	83.7	93.5	94.5
OmniWeaving	1,830	72.9	95.3	84.1	95.8	96.6
Closed-source models

Appendix

Table 21: Source-level VBench++ profile on RA-Bench. (a) Aggregate scores and conditioning consistency. The VBench-I2V Quality Score follows the released six-dimension normalization and weighting. Condition Fidelity averages normalized Video–Image Subject and Background Consistency, and Combined Quality gives the two aggregates equal weight. I2V-S and I2V-B are the corresponding raw evaluator outputs. (b) Raw means for the six dimensions of the VBench-I2V Quality Score. Source-level Subject and Background Consistency follow the released frame-level aggregation. All values except

N

are percentages. These are custom-input results on RA-Bench rather than VBench++ Image Suite leaderboard scores.

Quality score or dimension	Gemini Binary	Gemini Diagnostic	Gemini Rating	7-detector mean
Aggregate scores
VBench-I2V Quality Score	$-6.1\,[-8.2,-3.9]$	$-9.6\,[-11.7,-7.5]$	$-6.4\,[-8.1,-4.6]$	$-11.5\,[-12.3,-10.8]$
Condition Fidelity	$-10.6\,[-12.5,-8.6]$	$-14.4\,[-16.3,-12.5]$	$-10.2\,[-11.8,-8.5]$	$-9.5\,[-10.1,-8.8]$
Combined Quality	$-8.2\,[-10.3,-6.1]$	$-12.4\,[-14.4,-10.4]$	$-8.3\,[-10.0,-6.6]$	$-10.9\,[-11.6,-10.2]$
Condition fidelity and temporal consistency
Video–Image subject consistency	$-9.6\,[-11.6,-7.7]$	$-13.2\,[-15.1,-11.3]$	$-9.4\,[-11.0,-7.7]$	$-9.3\,[-10.0,-8.7]$

Appendix

Table 22: Adjusted associations between VBench++ scores and fake-side detector outputs. Except for Dynamic Degree, each cell reports the estimated change associated with an interquartile increase within each source and Dynamic Degree group, followed by its 95% confidence interval. Negative values indicate weaker fake evidence at higher scores. Dynamic Degree reports the adjusted dynamic-minus-static contrast. Gemini Binary and Diagnostic values are percentage points of fake recall, Gemini Rating is expressed on a 0–100 scale, and traditional values are percentile points of the mean normalized fake score across seven detectors.

Configuration	Metric	T2V	First frame (I2V)	First+last frames (I2V)	End-frame effect $\bm{\Delta}$
Skyra-SFT (RealR: 87.4 / 55.4)
Official timestamp	BAcc	75.3	60.6	51.9	$-8.7$
	FakeR	63.2	33.8	16.4	$-17.4$
	macro-F1	74.9	57.5	45.0	$-12.6$
Frame index	BAcc	75.2	60.2	50.6	$-9.6$
	FakeR	94.9	65.1	45.8	$-19.3$

Appendix

Table 23: Complete seed-0 fine-tuned MLLM results across generation settings. The real-video control is fixed within each configuration. RealR is listed in official-timestamp/frame-index order for Skyra. All values are percentages;

\Delta

is first+last minus first frame. BusterX++ abstentions are counted as incorrect.

Configuration	Generation setting	Seed 0	Seed 42	Seed 123	Seed range
Skyra-SFT
Official timestamp	T2V	63.2	64.2	60.4	3.7
	First frame	33.8	32.8	31.4	2.4
	First+last frames	16.4	16.8	16.3	0.4
	End-frame effect	$-17.4$	$-16.1$	$-15.1$	2.3
Skyra-RL

Appendix

Table 24: Fine-tuned MLLM FakeR across generation settings and seeds. Each setting contains the same 1,830 anchors for seeds 0, 42, and 123. Skyra uses the official-timestamp prompt, matching the paper’s cross-seed protocol. End-frame effect is first+last minus first frame; Seed range is the maximum minus the minimum across the three seeds. All values are percentages. BusterX++ abstentions are counted as incorrect.

Figure 19: Cross-seed stability of the generation-setting effects. Panel (a) reports first-frame-I2V-minus-T2V changes in FakeR, and panel (b) reports first+last-frame-I2V-minus-first-frame-I2V changes. Skyra-SFT and Skyra-RL share the same color and are distinguished by solid and dashed lines, while BusterX++ is shown in orange. Negative values indicate lower FakeR in the second setting. The panels use different vertical ranges to show cross-seed variation at the scale of each contrast.

Detector	Seed 0	Seed 42	Seed 123	$\bm{\Delta}$
Wan2.2 dynamic
CNNSpot	35.50	34.78	35.38	0.72
NPR	50.64	49.79	50.38	0.85
UnivFD	32.72	33.23	33.84	1.11
ForgeLens	59.99	60.23	59.89	0.34
DeCoF	62.47	63.18	63.29	0.82

Appendix

Table 25: Detector-specific traditional results across generation seeds. Each cell reports AUC.

\Delta

is the maximum minus the minimum across seeds. All values are percentages; bold marks the largest range in the table.

Generation source	FakeR			$\bm{\Delta}$ FakeR	$\bm{\Delta}$ BAcc
	Seed 0	Seed 42	Seed 123
Skyra-SFT
Wan2.2 dynamic	33.77	32.84	31.42	2.35	1.17
Wan2.2-Lightning	32.46	30.78	30.11	2.35	1.17
LTX	61.91	62.90	63.01	1.09	0.55
OmniWeaving	16.45	17.32	16.07	1.26	0.63

Appendix

Table 26: Source-wise seed sensitivity of the fine-tuned MLLMs. Skyra-SFT and Skyra-RL use the official-timestamp prompt, and BusterX++ uses its released evaluation pipeline. FakeR and BAcc are reported in %;

\Delta

denotes the maximum minus the minimum across seeds. Bold marks the largest range for each metric.

Video source	Videos	Judgments	Judged Real	Uncertain	Judged Generated
Real videos	1,812	5,436	71.9	5.3	22.8
Open-source generators
Wan2.2 dynamic	1,828	5,484	27.0	5.5	67.5
Wan2.2-Lightning	1,828	5,484	28.5	4.9	66.6
LTX	1,828	5,484	30.3	5.1	64.6
OmniWeaving	1,828	5,484	19.5	4.7	75.8

Appendix

Table 27: Stage-1 human recognition by source. The standard analysis stream contains three judgments per video. Response shares are in %; for generated sources, Judged Generated is human FakeR. Open and Closed averages give equal weight to each generation source, whereas Generated total pools all generated-video judgments.

Source	Stage 1 3/3 Real	Final 5/5 Real	Retained
Open-source generators
Wan2.2 dynamic	49	21	42.9
Wan2.2-Lightning	77	40	51.9
LTX	97	49	50.5
OmniWeaving	23	9	39.1
Open total	246	119	48.4

Appendix

Table 28: Two-stage construction and source composition of RA-Bench-HumanProof. (a) Source-wise retention from generated videos labeled Real by all three Stage 1 reviewers to those also labeled Real by both Stage 2 reviewers. Retention rates are in %. (b) Complete Stage 2 response pairs; only the Real/Real cell enters RA-Bench-HumanProof.

Configuration	RA-Bench- HumanProof AUC	T@5%	Matched ref. AUC	Matched ref. T@5%	$\bm{\Delta}$
Traditional detectors
CNNSpot	40.5	3.6	43.5	3.8	-3.0
NPR	50.3	3.8	50.4	4.0	-0.1
UnivFD	39.4	0.6	39.8	1.0	-0.4
ForgeLens	54.9	8.8	58.4	10.4	-3.5
DeCoF	59.0	1.9	59.2	1.7	-0.2

Appendix

Table 29: Complete detector results on RA-Bench-HumanProof. (a) Continuous-score configurations report paired AUC and T@5%. (b) Discrete-output configurations report BAcc, FakeR, and RealR. Matched ref. reports the corresponding AUC/T@5% or BAcc/FakeR pair on source-matched RA-Bench, and

\Delta

is the RA-Bench-HumanProof AUC or BAcc minus its matched reference. All values are in %. Skyra official-timestamp results follow the released protocol; frame index removes the numerical timestamp values.

Condition	Operations, in order
Original	Standardized input clip; no additional encoding.
T1	VP9 encoding followed by H.264 transcoding.
T1+T2	$0.5\times$ spatial downsampling, followed by T1.
T1+T3	Conversion to 8 fps, followed by T1.
T1+T4	Synthetic news badge, followed by T1.
Full	Spatial downsampling, 8-fps conversion, news badge, and T1.

Appendix

Table 30: Social dissemination simulation conditions. Each condition contains 150 real videos and 1,350 matched generated videos. T2–T4 are evaluated as additions to the common T1 transcode.

Detector / configuration	Original	T1	T1+T2	T1+T3	T1+T4	Full
Traditional detectors	AUC / T@5%
CNNSpot	44.5 [-0.2ex] 2.4	48.2 [-0.2ex] 2.4	48.0 [-0.2ex] 2.4	50.1 [-0.2ex] 2.1	47.5 [-0.2ex] 1.9	46.5 [-0.2ex] 2.2
NPR	52.2 [-0.2ex] 3.8	45.6 [-0.2ex] 3.4	44.9 [-0.2ex] 3.0	45.9 [-0.2ex] 2.7	45.5 [-0.2ex] 3.5	43.5 [-0.2ex] 3.0
UnivFD	37.1 [-0.2ex] 2.1	36.6 [-0.2ex] 1.8	36.0 [-0.2ex] 0.3	38.1 [-0.2ex] 1.0	37.2 [-0.2ex] 1.9	37.4 [-0.2ex] 0.2
ForgeLens	61.6 [-0.2ex] 17.3	44.7 [-0.2ex] 6.1	32.6 [-0.2ex] 2.2	46.3 [-0.2ex] 6.5	44.9 [-0.2ex] 4.8	35.6 [-0.2ex] 3.0
DeCoF	59.9 [-0.2ex] 3.2	63.0 [-0.2ex] 3.9	60.1 [-0.2ex] 0.4	65.1 [-0.2ex] 5.0	63.0 [-0.2ex] 3.3	62.3 [-0.2ex] 1.2

Appendix

Table 31: Complete detector results on RA-Bench-LastMile. Each cell reports the primary metric on top and the secondary metric below: AUC/T@5% for traditional detectors and BAcc/FakeR for fine-tuned MLLM detectors. All values are percentages and equal-source means over the nine RA-Bench generation sources.

Method	Metric	Change [95% CI]
Seven-detector mean	AUC	$-4.2\;[-5.6,-2.7]$
Skyra-SFT, timestamp	BAcc	$-18.3\;[-21.1,-15.4]$
Skyra-SFT, frame index	BAcc	$-11.5\;[-15.0,-8.3]$
Skyra-RL, timestamp	BAcc	$-20.0\;[-23.1,-16.9]$
Skyra-RL, frame index	BAcc	$-13.3\;[-16.7,-10.1]$

Appendix

Table 32: Uncertainty of Full-condition changes relative to Original. Values are percentage-point changes with 95% anchor-cluster bootstrap confidence intervals.

Deepfake Detection

Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.

See all 241 papers

Sep 7, 2026

DF26: We Cannot Tell Fake From Real Anymore

We introduce DF26, a novel benchmark for detecting AI-generated videos containing fully synthetic clips produced by recent text-to-video and image-to-video models. The videos capture single-person public-speaking scenarios, spanning direct-to-camera recordings, official statements, and studio interviews - 271 real and 2,420 synthetic videos generated by seven modern video models. The study on DF26 shows that human performance in detecting AI-generated videos, as well as state-of-the-art deepfake detectors, is close to random chance. Our results highlight the limitations of current evaluation protocols and motivate the need for benchmarks that explicitly measure robustness to modern generative model distribution shifts.

Severyn Shykula, Andrii Yermakov, Ivan Samarskyi +3

Ukrainian Catholic University · Faculty of Electrical Engineering, Czech Technical University in Prague · Hover Inc., USA +1

T2V Generation OOD Generalization Deepfake Detection Synthetic Video Detection Benchmarks Video Forensics

Ask PDF

Jun 30, 2026

Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit

AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards, yet most evaluation protocols leave uncontrolled confounds that can inflate reported generalization. As an existence proof, a three-feature clip-length classifier reaches a leave-one-generator-out (LOGO) AUC of 0.998 on GenVidBench under unaudited evaluation, while measuring nothing about motion. A 20-paper survey finds none applying all six standard controls that would catch this, so we combine them into an audited protocol and apply it to six representative feature sources (three published detectors and three repurposed signal sources), re-running it cross-dataset on AIGVDBench. The audit both debunks and certifies: the trivial classifier collapses to near chance (0.529), a CLIP baseline is caught carrying dataset identity, and the 2025 forensic detector WaveRep clears the floor at out-of-distribution LOGO AUC 0.996 with chance-level real-vs-real coherence. At a deployable FPR of 0.1%, multiple high-AUC methods fall to single-digit recall and the leaderboard order changes, so we recommend an audited tuple (AUC, above-floor margin, operating-point recall, and calibration) over a single number. As a white-box positive control, we add TemporalSpec (codec motion vectors); via cross-substrate feature fusion (XSFF), a second substrate adds genuine complementarity that survives the audit. We release VidAudit, to our knowledge the largest unified and audited detector collection for this task, providing 14 detectors behind one plugin API, a leaderboard, and Croissant metadata, available at https://github.com/KurbanIntelligenceLab/vidaudit. Together, the protocol and toolkit move evaluation from leaderboard rank toward whether a result measures what it claims.

Mert Onur Cakiroglu, Zhihe Lu, Mehmet Dalkilic +1

Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar

Benchmark Auditing OOD Generalization Deepfake Detection Generative AI Evaluation Synthetic Video Detection Benchmarks Video Forensics

Ask PDF

Aug 31, 2026

TUE-Detector: A Tool-Using Expert MLLM-Based Detector for AI-Generated Videos

AI-generated video detection, which aims to distinguish AI-generated videos from real ones, has recently received increasing research attention. To perform this task reliably, a key challenge lies in accurately identifying subtle-yet-measurable unnatural artifacts. In this work, we address this challenge from a novel perspective of tool-mediated evidence discovery and propose Tool-Using Expert MLLM-based AI-generated Video Detector (TUE-Detector), a novel framework for AI-generated video detection. TUE-Detector trains a general MLLM into a task-tailored tool-using expert detector that learns to invoke suitable tools, collect concrete evidence of unnaturalness, and reason over the evidence for reliable detection. Meanwhile, TUE-Detector further introduces novel designs to equip the expert detector with high-quality and suitable tools. Extensive experiments demonstrate the effectiveness of our framework.

Yichen Wu, Haoxuan Qu, Yongxing Dai +5

University of Nottingham · Lancaster University · Peking University

Multimodal Large Language Models Deepfake Detection LLM Tool Use

Ask PDF

Abstract

Deepfake Detection

Explore similar work

DF26: We Cannot Tell Fake From Real Anymore

Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit

TUE-Detector: A Tool-Using Expert MLLM-Based Detector for AI-Generated Videos