Media Intelligence Platform: Multimodal Detection of Synthetic and Manipulated Content

David Fischinger; Marcel Hasenbalg; Silvia Poletti; Davide Antonutti; Mina Schütz; Lam Pham; Justin Ilyes; Péter Rjabcsenkó

cs.CVOct 20, 2026

Media Intelligence Platform: Multimodal Detection of Synthetic and Manipulated Content

Authors: David Fischinger, Marcel Hasenbalg, Silvia Poletti, Davide Antonutti, Mina Schütz, Lam Pham, Justin Ilyes, Péter Rjabcsenkó,

Organizations: Austrian Institute of Technology, Giefinggasse 4, 1210 Vienna - Austria

Abstract

The rapid progress of generative artificial intelligence has led to a significant increase in manipulated multimedia content, including synthetic faces, forged videos, AI-generated images, fake audio, and locally edited images. Existing detection approaches are typically designed for a single manipulation type, limiting their applicability in real-world scenarios involving heterogeneous media sources. In this work, we present the Media Intelligence Platform (MIP), a unified web-based framework for multimodal media forensics integrating specialized modules for fake video detection, face morphing detection, AI-generated image detection, audio deepfake detection, localized image manipulation detection, and text analysis. Its modular architecture enables flexible integration of heterogeneous models and supports future extensions for emerging manipulation techniques. Experimental results on representative benchmark datasets demonstrate reliable detection performance across multiple media modalities while providing a practical and user-friendly interface for forensic investigations. The platform highlights the potential of unified multimodal systems for comprehensive deepfake detection in real-world environments, and is accompanied by a public demonstrator released with this workshop submission to support reproducibility and community evaluation.

Figures & tables

Figure 1: MIP overview. The platform supports multimodal input upload, analysis tool selection, and job history tracking.

Figure 2: Example result view showing model predictions and explainable outputs.

Figure 3: Schematic of the MIP microservice architecture. Modules are deployed as Kubernetes pods and communicate via standardized REST APIs. Results and metadata are stored through a shared NFS file system and PostgreSQL backend.

Figure 4: MIP-UI: reference frontend for module selection, job history, and result visualization.

AI Detectors	ImgN100 [ 15 ] R	VOC12 [ 21 ] R	OImg-v7 [ 35 ] R	FLUX.1 [ 5 ] F	Firefly [ 1 ] F	MJ 5&6 [ 46 , 47 ] F	DALL-E 3 [ 58 ] F	GLIDE [ 56 ] F
ResNet50[fc]	87.1	82.5	96.4	98.2	92.2	96.7	98.1	84.9
ResNet50[fc+l4]	97.0	92.4	99.4	98.8	96.4	87.4	92.1	86.2
ResNet50[fc+l4+l3]	99.2	96.6	99.6	95.8	79.0	89.5	96.2	74.8
ViT-B-16	94.2	89.6	97.5	94.9	86.9	93.8	97.0	89.9

Table 1: F1-score (%) comparison of AI-generated image detectors on evaluation datasets. Best results per dataset are highlighted in bold. For each dataset, the image type is reported ( R for real and F for fake).

Figure 5: Example output of the IFDL module. The manipulated input image (left) is assigned a high manipulation probability, while the predicted localization mask (right) highlights the tampered region.

	Test Datasets
Models	CASIA [ 18 ]			Columbia [ 29 ]			DSO [ 14 ]			NIST [ 52 ]			Average
	AUC	F1	IoU	AUC	F1	IoU	AUC	F1	IoU	AUC	F1	IoU	AUC	F1	IoU	Mean
DFCN [ 88 ]	0.654	0.192	0.119	0.789	0.541	0.395	0.724	0.303	0.227	0.778	0.250	0.204	0.736	0.322	0.236	0.431
FSim [ 45 ]	0.554	0.169	0.102	0.731	0.604	0.474	0.796	0.487	0.371	0.642	0.188	0.123	0.681	0.362	0.268	0.437
MNet [ 80 ]	0.776	0.130	0.086	0.747	0.357	0.258	0.795	0.344	0.253	0.634	0.088	0.054	0.738	0.230	0.163	0.377
Wu22 [ 79 ]	0.873	0.509	0.465	0.862	0.707	0.608	0.854	0.436	0.308	0.783	0.332	0.255	0.843	0.496	0.409	0.583

Table 2: Comparison of state-of-the-art approaches and our proposed IFDL-Model in default mode respectively averaging over the results of the augmented (horizontally/vertically mirrored, rotated) input image. Highest values are highlighted in bold.

Algorithm	DM	G-DiM	MGAN-II	OCV	SGAN2	UTW	Avg.
SelfMAD [ 30 ]	51.45	5.45	2.90	0.59	73.25	52.98	31.10
SelfMAD++ [ 31 ]	48.22	25.04	2.04	0.85	46.68	14.48	22.89
SPL-MAD [ 22 ]	44.32	4.38	20.59	17.75	55.62	80.47	37.19
CrEffViT [ 10 ]	18.43	42.45	18.29	22.98	12.29	17.24	21.95
FatFormer [ 41 ]	16.31	10.87	1.19	6.80	12.43	12.50	10.02
Triplet-Loss [ 73 ]	11.85	5.33	0.98	5.10	11.56	0.70	5.92

Table 3: Equal Error Rate (EER %) obtained by the evaluated methods on the different morph generation techniques. From left to right the generation methods are: DiffMorpher, Greedy-DiM, MipGAN-II, OpenCV, StyleGAN2 and UTW.

Detector Method	Category	Video AUC	Reference
EFFORT	ViT, CLIP	0.928	[ 82 ]
selimsef FbF	CNN, Ensemble	0.917	[ 65 ]
GenConvViT	CNN, ViT	0.908	[ 78 ]
CC ViT	CNN, ViT	0.903	[ 10 ]
Efficient ViT	CNN, ViT	0.869	[ 10 ]
UCF	CNN, Disentanglement, Autoencoder	0.864	[ 83 ]

Table 4: Deepfake detection methods available on the Media-Intelligence-Platform. Performance on benchmark dataset, sorted by descending video AUC.

Figure 6: For each video and each detector result the report will display a bar chart for each person with the predicted faceswap detection confidence per frame or sequence.

Test Dataset	Accuracy	F1	AUC
In-The-Wild	0.87	0.79	0.83
ASVspoof5 Eval	0.91	0.91	0.92
FakeAVCeleb	0.99	0.99	0.99

Table 5: Cross-dataset performance of the DSD module.

Figure 7: Overview of the Deepfake Speech Detection (DSD) pipeline. The model uses a three-stage training strategy based on feature extraction, discriminative embedding learning, and Mahalanobis-based inference.

Multimodal Deepfake Detection

See all 33 papers

May 28, 2026

DeepFake Forensics AI: A Multi-Modal Detection and Blockchain-Anchored Evidence Management Platform

The proliferation of AI-generated synthetic media poses a critical threat to the integrity of digital evidence in legal and forensic contexts. Existing deepfake detection systems typically address a single modality and provide no mechanism for tamper-proof evidence preservation. We present DeepFake Forensics AI, a unified platform that detects synthetic media across image, video, and audio modalities, identifies generative architecture fingerprints, and anchors forensic evidence immutably on the Ethereum blockchain. Our system trains four independent neural networks from scratch: an EfficientNet-B4 image detector (AUC = 0.9868), a Bidirectional LSTM video detector (AUC= 0.9628), an ECAPA-TDNN audio detector (EER = 18.63%), and a novel GAN fingerprinting module (accuracy = 99.88%) that identifies the generative architecture behind a fake image. Evidence files are hashed with SHA-256, stored on IPFS via Pinata, and registered on-chain via a Solidity smart contract with role-based access control. The platform provides a React frontend and FastAPI backend suitable for deployment in forensic and legal workflows. To our knowledge, this is the first system to unify multi-modal deepfake detection with blockchain-based chain-of custody management.

Naisha Minnah

Department of Computer Science Providence Women’s College, Calicut Affiliated to University of Calicut, Kerala, India · Affiliated to University of Calicut, Kerala, India

Audio Deepfake Detection Image Forgery Detection Digital Forensics Deepfake Detection AI-Generated Image Detection Multimodal Deepfake Detection

Ask PDF

Apr 27, 2026

DYMAPIA: A Multi-Domain Framework for Detecting AI-based Video Manipulation

AI-generated media are advancing rapidly, raising pressing concerns for content authenticity and digital trust. We introduce DYMAPIA, a multi-domain Deepfake detection framework that fuses spatial, spectral, and temporal cues to capture subtle traces of manipulation in visual data. The system builds dynamic anomaly masks by combining evidence from Fourier spectra, local texture descriptors, edge irregularities, and optical flow consistency, which highlight tampered regions with fine spatial accuracy. These masks guide DistXCNet, a lightweight classifier distilled from Xception and optimized with depthwise separable convolutions for fast, region-focused classification. This joint design achieves state-of-the-art results, with accuracy and F1-scores exceeding 99% on FF++, Celeb-DF, and VDFD benchmarks, while keeping the model compact enough for real-time use. Beyond outperforming existing full-frame and multidomain detectors, DYMAPIA demonstrates deployment readiness for time-critical forensic tasks, including media verification, misinformation defense, and secure content filtering.

Md Shohel Rana, Andrew H. Sung

School of Computing Georgia Southern University Statesboro, GA 30460, USA · School of Computing Sciences & Computer Engineering The University of Southern Mississippi Hattiesburg, MS 39406, USA

Digital Image Forensics Deepfake Detection Convolutional Neural Networks

Ask PDF

Nov 29, 2024

Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook

We survey deepfake generation and detection techniques, covering all deepfake media types: image, video, audio and multimodal content. We identify various kinds of deepfakes and construct taxonomies of deepfake generation and detection methods, illustrating the important groups of methods. Next, we gather datasets used for deepfake detection and provide updated rankings of the best performing detectors on the most popular datasets. In addition, we develop a novel multimodal benchmark to evaluate deepfake detectors on out-of-distribution content. The results indicate that state-of-the-art detectors fail to generalize to deepfakes generated by unseen generators. Our project page and new benchmark are available at https://github.com/CroitoruAlin/biodeep.

Florinel-Alin Croitoru, Andrei-Iulian Hiji, Vlad Hondru +7

Department of Computer Science, University of Bucharest, Romania · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · Department of Computer Science, University of Central Florida, Orlando, US

Audio Deepfake Detection Deepfake Detection Multimodal Deepfake Detection

Ask PDF

Abstract

Multimodal Deepfake Detection

Explore similar work

DeepFake Forensics AI: A Multi-Modal Detection and Blockchain-Anchored Evidence Management Platform

DYMAPIA: A Multi-Domain Framework for Detecting AI-based Video Manipulation

Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook