Human-AI Ensembles Improve Deepfake Detection in Low-to-Medium Quality Videos

Marco Postiglione; Isabel Gortner; V. S. Subrahmanian

cs.CVMar 15, 2026

Human-AI Ensembles Improve Deepfake Detection in Low-to-Medium Quality Videos

Authors: Marco Postiglione, Isabel Gortner, V. S. Subrahmanian

Organizations: Northwestern University

Abstract

Deepfake detection is widely framed as a machine learning problem, yet how humans and AI detectors compare under realistic conditions remains poorly understood. We evaluate 200 human participants and 95 state-of-the-art AI detectors across two datasets: DF40, a standard benchmark, and CharadesDF, a novel dataset of videos of everyday activities. CharadesDF was recorded using mobile phones leading to low/moderate quality videos compared to the more professionally captured DF40. Humans outperform AI detectors on both datasets, with the gap widening in the case of CharadesDF where AI accuracy collapses to near chance (0.537) while humans maintain robust performance (0.784). Human and AI errors are complementary: humans miss high-quality deepfakes while AI detectors flag authentic videos as fake, and hybrid human-AI ensembles reduce high-confidence errors. These findings suggest that effective real-world deepfake detection, especially in non-professionally produced videos, requires human-AI collaboration rather than AI algorithms alone.

Figures & tables

Figure 1: Examples of video recordings from the CharadesDF dataset. Participants recorded videos in their home environments while performing everyday activities such as drinking from cups, holding objects, stretching, and working at desks. These conditions produce substantial variability in lighting, camera angles, face visibility, and image quality—challenges common in authentic user-generated content but underrepresented in existing deepfake detection benchmarks.

Figure 2: Distribution of participant-level accuracy for human participants and AI detectors. ( A ) DF40 dataset. ( B ) CharadesDF dataset. Each data point represents one participant’s overall accuracy across all videos they evaluated.

	DF40				CharadesDF
	Accuracy	F1	AUC	CFR	Accuracy	F1	AUC	CFR
Human (indiv.)	0.743 $\pm$ 0.023	0.759 $\pm$ 0.020	0.823 $\pm$ 0.020	0.190 $\pm$ 0.017	0.784 $\pm$ 0.027	0.783 $\pm$ 0.027	0.846 $\pm$ 0.024	0.171 $\pm$ 0.025
AI detector (indiv.)	0.610 $\pm$ 0.030	0.612 $\pm$ 0.042	0.686 $\pm$ 0.037	0.279 $\pm$ 0.036	0.537 $\pm$ 0.009	0.533 $\pm$ 0.041	0.583 $\pm$ 0.014	0.317 $\pm$ 0.029
Human ensemble	0.890 $\pm$ 0.020	0.888 $\pm$ 0.021	0.952 $\pm$ 0.013	0.019 $\pm$ 0.009	0.928 $\pm$ 0.017	0.926 $\pm$ 0.019	0.979 $\pm$ 0.009	0.010 $\pm$ 0.007
AI ensemble	0.869 $\pm$ 0.020	0.876 $\pm$ 0.021	0.949 $\pm$ 0.012	0.001 $\pm$ 0.002	0.607 $\pm$ 0.033	0.683 $\pm$ 0.034	0.697 $\pm$ 0.035	0.009 $\pm$ 0.006
Hybrid ensemble	0.941 $\pm$ 0.014	0.941 $\pm$ 0.014	0.988 $\pm$ 0.005	0.000 $\pm$ 0.000	0.924 $\pm$ 0.018	0.924 $\pm$ 0.019	0.979 $\pm$ 0.008	0.000 $\pm$ 0.000

Table 1: Comparison of individual versus ensemble performance on deepfake detection. Values shown as point estimate

\pm

error. Error represents half-width of 95% bootstrap CI. CFR = Catastrophic Failure Rate. Bold indicates best ensemble performance.

Figure 3: Complementary failure patterns between human and AI ensembles. Each point represents a video, plotted by human ensemble probability (x-axis) versus AI ensemble probability (y-axis) of being a deepfake. Dashed lines indicate the decision threshold (0.5). Small circles show videos without catastrophic failures (teal: real videos; orange: deepfakes). Squares indicate videos where only the human ensemble failed catastrophically; triangles indicate videos where only the AI ensemble failed. No videos caused catastrophic failures in both groups simultaneously (no X markers). (A) DF40 dataset. (B) CharadesDF dataset.

Figure 4: Catastrophic failure rates across threshold definitions. CFR was computed for threshold values ranging from 0.5 (any probability error

\geq 0.5

is catastrophic) to 1.0 (only complete misclassification with probability error = 1.0). Dashed lines show mean individual performance; solid lines show ensemble performance. Quality-weighted voting was used for all ensembles. (A) DF40 dataset. (B) CharadesDF dataset.

	DF40					CharadesDF
Quality Factor	Human	AI	Human Ens.	AI Ens.	Hybrid Ens.	Human	AI	Human Ens.	AI Ens.	Hybrid Ens.
FR Confidence	0.008*	0.001	0.001	0.011	0.002	0.024*	-0.003	0.014	-0.012	0.035*
Bbox Area (ratio)	0.046***	0.034***	0.028	0.039***	0.017	0.016	0.026**	0.000	0.038	0.003
Face Presence (ratio)	-0.008	0.002	-0.006	-0.011*	-0.002	0.008	0.002	0.010	-0.001	0.008
IOD	-0.029	-0.057***	0.019	-0.029	0.003	-0.022	-0.062***	0.026	-0.106**	-0.004
Left EAR	-0.001	-0.002	-0.007	0.013	0.002	0.013	0.004	0.013	0.026	0.019

Table 2: Quality factors predicting detection accuracy.

Figure 5: Detection accuracy as a function of image quality factors. Smoothed performance curves showing the relationship between quality features and detection accuracy for human individuals (teal, dashed), AI detectors (orange, dashed), human ensembles (teal, solid), AI ensembles (orange, solid), and hybrid ensembles (purple, solid). Shaded regions indicate 95% confidence intervals. We indicate correlations (

\rho

) in each subplot for both humans (

H

) and AI detectors (

M

). Rows correspond to DF40 (top) and CharadesDF (bottom) datasets. ( A ) Bounding box area ratio. ( B ) Inter-ocular distance (IOD). ( C ) Signal-to-noise ratio (SNR). ( D ) Contrast. ( E ) Color balance.

Figure 6: Reliability diagrams for deepfake detection confidence: humans vs. AI detectors. Calibration curves showing the relationship between mean predicted probability and fraction correct for human participants (dark curves) and AI detectors (orange curves). The dashed line indicates perfect calibration; points above the line indicate underconfidence, points below indicate overconfidence. Error bars show 95% confidence intervals. ( A ) DF40, ( B ) CharadesDF.

Figure 7: Overconfidence score by performance quartile: humans vs. AI detectors. Overconfidence computed as normalized confidence minus accuracy; positive values (red) indicate overconfidence, negative values (green) indicate underconfidence. Top row: human participants; bottom row: AI detectors. Poor performers (Q1) are overconfident while top performers (Q4) trend toward underconfidence in both groups, consistent with the Dunning-Kruger effect. The effect is more pronounced for AI detectors. ( A ) DF40 humans:

\rho=-0.405

p<0.001

. ( B ) CharadesDF humans:

\rho=-0.210

p=0.036

. ( C ) DF40 AI detectors:

\rho=-0.431

p<0.001

. ( D ) CharadesDF AI detectors:

\rho=-0.238

p=0.020

Demographic Factor	DF40	CharadesDF
Age	0.021	$-$ 0.025
Education	0.032	$-$ 0.043**
Tech Savviness	$-$ 0.023	0.001
Social Media Usage	0.012	0.002
Deepfake Familiarity	$-$ 0.004	0.013
Prior Deepfake Exposure	0.009	0.020

Table 3: Demographic factors predicting human deepfake detection accuracy.

Variable	Category	DF40	CharadesDF
Age	Under 18	0 (0.0%)	0 (0.0%)
	18–24 years old	5 (5.0%)	2 (2.0%)
	25–34 years old	38 (38.0%)	33 (33.0%)
	35–44 years old	28 (28.0%)	23 (23.0%)
	45–54 years old	14 (14.0%)	24 (24.0%)
	55–64 years old	10 (10.0%)	12 (12.0%)

Table 4: Participant demographics summary.

Feature	Description
Face Detection and Recognition
FR Confidence	Face detection confidence score from DeepFace Serengil and Ozpinar (2024) , reflecting how reliably the face was detected and localized. Higher values indicate clearer, more frontal faces.
Bbox Area (ratio)	Ratio of the face bounding box area to the total frame area, computed as $(w_{\text{bbox}}\times h_{\text{bbox}})/(w_{\text{frame}}\times h_{\text{frame}})$ . Larger values indicate more prominent faces in the frame.
Face Presence (ratio)	Proportion of sampled frames in which at least one face was successfully detected.
Facial Geometry
IOD	Inter-Ocular Distance: Euclidean distance in pixels between the centers of the left and right eyes, computed from 68-point facial landmarks. Serves as a proxy for face scale and distance from camera.

Table 5: Quality features extracted for each video. Features are computed per-frame and aggregated to video-level statistics using the mean across frames with detected faces.

	DF40 ( $N=997$ )			CharadesDF ( $N=779$ )
Quality Feature	Mean	Median	SD	Mean	Median	SD	Direction
Face Detection and Recognition
FR Confidence	1.000	1.000	0.0006	0.9995	1.000	0.0024	DF40 $>$ ∗∗∗†
Face Presence (ratio)	1.000	1.000	0.0005	0.811	0.919	0.235	DF40 $>$ ∗∗∗
Bbox Area (ratio)	0.076	0.054	0.069	0.078	0.052	0.081	DF40 $>$ ∗∗∗
Facial Geometry

Table 6: Distributional comparison of quality features between DF40 and CharadesDF datasets. Mann-Whitney U tests with Benjamini-Hochberg FDR correction.

Panel of Table 7 — Table 7: Sampled frames from example video stimuli and their associated quality factor values.

Model	Trained on FF++		Trained on CDF		Trained on DF40		Avg
Model	DF40	CharadesDF	DF40	CharadesDF	DF40	CharadesDF	Avg
Capsule Nguyen et al. (2019)	0.562	0.519	0.558	0.492	0.884	0.553	0.595
CLIP Radford et al. (2021)	0.500	0.493	0.501	0.493	0.541	0.515	0.507
CORE Ni et al. (2022b)	0.669	0.548	0.596	0.614	0.889	0.555	0.645
EfficientNet-B4 Tan and Le (2019)	0.503	0.508	0.556	0.567	0.837	0.501	0.579
Effort Yan et al. (2025)	0.648	0.577	0.658	0.601	0.780	0.628	0.649

Table 8: Performance (Accuracy) of deepfake detectors trained on FaceForensics++ (FF++), CelebDF-v2 (CDF), or DF40, and evaluated on our DF40 test sample and CharadesDF. Row and column averages are italicized.

Model	Trained on FF++		Trained on CDF		Trained on DF40		Avg
Model	DF40	CharadesDF	DF40	CharadesDF	DF40	CharadesDF	Avg
Capsule Nguyen et al. (2019)	0.706	0.561	0.619	0.545	0.983	0.643	0.676
CLIP Radford et al. (2021)	0.530	0.545	0.521	0.590	0.551	0.527	0.544
CORE Ni et al. (2022b)	0.768	0.668	0.634	0.635	0.982	0.582	0.712
EfficientNet-B4 Tan and Le (2019)	0.605	0.626	0.584	0.583	0.929	0.520	0.641
Effort Yan et al. (2025)	0.821	0.753	0.721	0.674	0.953	0.709	0.772

Table 9: Performance (AUC) of deepfake detectors trained on FaceForensics++ (FF++), CelebDF-v2 (CDF), or DF40, and evaluated on our DF40 test sample and CharadesDF. Row and column averages are italicized.

Model	Trained on FF++		Trained on CDF		Trained on DF40		Avg
Model	DF40	CharadesDF	DF40	CharadesDF	DF40	CharadesDF	Avg
Capsule Nguyen et al. (2019)	0.681	0.665	0.629	0.635	0.895	0.674	0.697
CLIP Radford et al. (2021)	0.667	0.661	0.667	0.661	0.526	0.607	0.631
CORE Ni et al. (2022b)	0.727	0.683	0.553	0.618	0.876	0.597	0.676
EfficientNet-B4 Tan and Le (2019)	0.668	0.666	0.534	0.620	0.848	0.568	0.650
Effort Yan et al. (2025)	0.723	0.688	0.563	0.463	0.719	0.569	0.621

Table 10: Performance (F1 score) of deepfake detectors trained on FaceForensics++ (FF++), CelebDF-v2 (CDF), or DF40, and evaluated on our DF40 test sample and CharadesDF. Row and column averages are italicized.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Distribution of video-level accuracy for human participants and AI detectors. ( A ) DF40 dataset. ( B ) CharadesDF dataset. Each data point represents the proportion of correct classifications for a single video, aggregated across all human participants or AI detectors.

	DF40				CharadesDF
	Accuracy	F1	AUC	CFR	Accuracy	F1	AUC	CFR
Human (indiv.)	0.743 $\pm$ 0.023	0.759 $\pm$ 0.020	0.823 $\pm$ 0.020	0.190 $\pm$ 0.017	0.784 $\pm$ 0.027	0.783 $\pm$ 0.027	0.846 $\pm$ 0.024	0.171 $\pm$ 0.025
AI detector (indiv.)	0.610 $\pm$ 0.030	0.612 $\pm$ 0.042	0.686 $\pm$ 0.037	0.279 $\pm$ 0.036	0.537 $\pm$ 0.009	0.533 $\pm$ 0.041	0.583 $\pm$ 0.014	0.317 $\pm$ 0.029
Human ensemble	0.881 $\pm$ 0.020	0.881 $\pm$ 0.022	0.948 $\pm$ 0.013	0.020 $\pm$ 0.009	0.928 $\pm$ 0.017	0.926 $\pm$ 0.019	0.975 $\pm$ 0.010	0.012 $\pm$ 0.007
AI ensemble	0.796 $\pm$ 0.025	0.818 $\pm$ 0.025	0.912 $\pm$ 0.017	0.003 $\pm$ 0.004	0.603 $\pm$ 0.034	0.681 $\pm$ 0.034	0.692 $\pm$ 0.034	0.008 $\pm$ 0.006
Hybrid ensemble	0.925 $\pm$ 0.016	0.926 $\pm$ 0.016	0.980 $\pm$ 0.007	0.000 $\pm$ 0.000	0.914 $\pm$ 0.018	0.914 $\pm$ 0.020	0.975 $\pm$ 0.009	0.000 $\pm$ 0.000

Appendix

Table 11: Soft Mean Ensemble. Values shown as point estimate

\pm

error. Error represents half-width of 95% bootstrap CI. CFR = Catastrophic Failure Rate. Bold indicates best ensemble performance.

	DF40				CharadesDF
	Accuracy	F1	AUC	CFR	Accuracy	F1	AUC	CFR
Human (indiv.)	0.743 $\pm$ 0.023	0.759 $\pm$ 0.020	0.823 $\pm$ 0.020	0.190 $\pm$ 0.017	0.784 $\pm$ 0.027	0.783 $\pm$ 0.027	0.846 $\pm$ 0.024	0.171 $\pm$ 0.025
AI detector (indiv.)	0.610 $\pm$ 0.030	0.612 $\pm$ 0.042	0.686 $\pm$ 0.037	0.279 $\pm$ 0.036	0.537 $\pm$ 0.009	0.533 $\pm$ 0.041	0.583 $\pm$ 0.014	0.317 $\pm$ 0.029
Human ensemble	0.857 $\pm$ 0.022	0.860 $\pm$ 0.023	0.924 $\pm$ 0.016	0.067 $\pm$ 0.015	0.920 $\pm$ 0.018	0.919 $\pm$ 0.020	0.961 $\pm$ 0.013	0.045 $\pm$ 0.014
AI ensemble	0.767 $\pm$ 0.025	0.803 $\pm$ 0.024	0.907 $\pm$ 0.017	0.038 $\pm$ 0.012	0.579 $\pm$ 0.035	0.682 $\pm$ 0.033	0.696 $\pm$ 0.034	0.083 $\pm$ 0.020
Hybrid ensemble	0.903 $\pm$ 0.019	0.906 $\pm$ 0.019	0.974 $\pm$ 0.008	0.006 $\pm$ 0.005	0.910 $\pm$ 0.020	0.910 $\pm$ 0.021	0.966 $\pm$ 0.012	0.001 $\pm$ 0.003

Appendix

Table 12: Soft Median Ensemble. Values shown as point estimate

\pm

error. Error represents half-width of 95% bootstrap CI. CFR = Catastrophic Failure Rate. Bold indicates best ensemble performance.

	DF40				CharadesDF
	Accuracy	F1	AUC	CFR	Accuracy	F1	AUC	CFR
Human (indiv.)	0.743 $\pm$ 0.023	0.759 $\pm$ 0.020	0.823 $\pm$ 0.020	0.190 $\pm$ 0.017	0.784 $\pm$ 0.027	0.783 $\pm$ 0.027	0.846 $\pm$ 0.024	0.171 $\pm$ 0.025
AI detector (indiv.)	0.610 $\pm$ 0.030	0.612 $\pm$ 0.042	0.686 $\pm$ 0.037	0.279 $\pm$ 0.036	0.537 $\pm$ 0.009	0.533 $\pm$ 0.041	0.583 $\pm$ 0.014	0.317 $\pm$ 0.029
Human ensemble	0.568 $\pm$ 0.032	0.697 $\pm$ 0.027	0.847 $\pm$ 0.022	0.331 $\pm$ 0.028	0.597 $\pm$ 0.033	0.710 $\pm$ 0.031	0.848 $\pm$ 0.023	0.317 $\pm$ 0.033
AI ensemble	0.499 $\pm$ 0.032	0.666 $\pm$ 0.028	0.812 $\pm$ 0.026	0.501 $\pm$ 0.032	0.496 $\pm$ 0.035	0.663 $\pm$ 0.031	0.675 $\pm$ 0.037	0.504 $\pm$ 0.035
Hybrid ensemble	0.502 $\pm$ 0.032	0.667 $\pm$ 0.028	0.936 $\pm$ 0.014	0.430 $\pm$ 0.032	0.496 $\pm$ 0.035	0.663 $\pm$ 0.031	0.902 $\pm$ 0.022	0.402 $\pm$ 0.033

Appendix

Table 13: Soft Max Ensemble. Values shown as point estimate

\pm

error. Error represents half-width of 95% bootstrap CI. CFR = Catastrophic Failure Rate. Bold indicates best ensemble performance.

	DF40				CharadesDF
	Accuracy	F1	AUC	CFR	Accuracy	F1	AUC	CFR
Human (indiv.)	0.743 $\pm$ 0.023	0.759 $\pm$ 0.020	0.823 $\pm$ 0.020	0.190 $\pm$ 0.017	0.784 $\pm$ 0.027	0.783 $\pm$ 0.027	0.846 $\pm$ 0.024	0.171 $\pm$ 0.025
AI detector (indiv.)	0.610 $\pm$ 0.030	0.612 $\pm$ 0.042	0.686 $\pm$ 0.037	0.279 $\pm$ 0.036	0.537 $\pm$ 0.009	0.533 $\pm$ 0.041	0.583 $\pm$ 0.014	0.317 $\pm$ 0.029
Human ensemble	0.821 $\pm$ 0.026	0.835 $\pm$ 0.025	0.822 $\pm$ 0.025	0.179 $\pm$ 0.026	0.897 $\pm$ 0.021	0.899 $\pm$ 0.022	0.898 $\pm$ 0.021	0.103 $\pm$ 0.021
AI ensemble	0.767 $\pm$ 0.025	0.803 $\pm$ 0.024	0.767 $\pm$ 0.023	0.233 $\pm$ 0.025	0.579 $\pm$ 0.035	0.682 $\pm$ 0.033	0.582 $\pm$ 0.025	0.421 $\pm$ 0.035
Hybrid ensemble	0.719 $\pm$ 0.028	0.780 $\pm$ 0.026	0.900 $\pm$ 0.019	0.059 $\pm$ 0.015	0.597 $\pm$ 0.035	0.709 $\pm$ 0.032	0.890 $\pm$ 0.021	0.050 $\pm$ 0.015

Appendix

Table 14: Hard Ensemble. Values shown as point estimate

\pm

error. Error represents half-width of 95% bootstrap CI. CFR = Catastrophic Failure Rate. Bold indicates best ensemble performance.

Figure 9: Detection F1 as a function of quality features. Smoothed performance curves showing the relationship between quality features and F1 for human individuals (teal, dashed), AI detectors (orange, dashed), human ensembles (teal, solid), AI ensembles (orange, solid), and hybrid ensembles (purple, solid). Shaded regions indicate 95% confidence intervals. We indicate correlations (

\rho

) in each subplot for both humans (

H

) and AI detectors (

M

). Rows correspond to DF40 (top) and CharadesDF (bottom) datasets. Dashed vertical lines indicate human-AI crossovers. ( A ) Bounding box area ratio. ( B ) Signal-to-noise ratio (SNR). ( C ) Inter-ocular distance (IOD). ( D ) Contrast. ( E ) Color balance.

Figure 10: Detection AUC as a function of quality features. Smoothed performance curves showing the relationship between quality features and AUC for human individuals (teal, dashed), AI detectors (orange, dashed), human ensembles (teal, solid), AI ensembles (orange, solid), and hybrid ensembles (purple, solid). Shaded regions indicate 95% confidence intervals. We indicate correlations (

\rho

) in each subplot for both humans (

H

) and AI detectors (

M

Explore similar work

DF26: We Cannot Tell Fake From Real Anymore
Sep 7, 2026Severyn Shykula, Andrii Yermakov, Ivan Samarskyi +3Text-to-Video Generation OOD Generalization
Ukrainian Catholic University · Faculty of Electrical Engineering, Czech Technical University in Prague · Hover Inc., USA +1
Continuously Evolving Deepfake Detection: An Architecture and Public-Benchmark Evaluation of a Dynamic Detection System
Jul 14, 2026Ken Jon Miyachi, Dylan UysDeepfake Detection AI-Generated Image Detection
BitMind
Multi-Agent Forensic Reasoning for Generalizable Deepfake Video Detection
Aug 7, 2026Xuechao Zou, Shun Zhang, Kai Li +6Explainable Deepfake Detection Deepfake Detection
Beijing Jiaotong University · Tsinghua University · Ant Group

Sep 7, 2026J/K move · Enter open · S save

DF26: We Cannot Tell Fake From Real Anymore

Severyn Shykula, Andrii Yermakov, Ivan Samarskyi, Dmytro Mishkin+2

Ukrainian Catholic University · Faculty of Electrical Engineering, Czech Technical University in Prague · Hover Inc., USA+1

Text-to-Video Generation OOD Generalization Deepfake Detection

Open paper AskPDF arXiv

We introduce DF26, a novel benchmark for detecting AI-generated videos containing fully synthetic clips produced by recent text-to-video and image-to-video models. The videos capture single-person public-speaking scenarios, spanning direct-to-camera recordings, official statements, and studio interviews - 271 real and 2,420 synthetic videos generated by seven modern video models. The study on DF26 shows that human performance in detecting AI-generated videos, as well as state-of-the-art deepfake detectors, is close to random chance. Our results highlight the limitations of current evaluation protocols and motivate the need for benchmarks that explicitly measure robustness to modern generative model distribution shifts.

Abstract

Explore similar work

DF26: We Cannot Tell Fake From Real Anymore

Continuously Evolving Deepfake Detection: An Architecture and Public-Benchmark Evaluation of a Dynamic Detection System

Multi-Agent Forensic Reasoning for Generalizable Deepfake Video Detection