Media Intelligence Platform: Multimodal Detection of Synthetic and Manipulated Content

David Fischinger; Marcel Hasenbalg; Silvia Poletti; Davide Antonutti; Mina Schütz; Lam Pham; Justin Ilyes; Péter Rjabcsenkó

cs.CVOct 20, 2026

Media Intelligence Platform: Multimodal Detection of Synthetic and Manipulated Content

Authors: David Fischinger, Marcel Hasenbalg, Silvia Poletti, Davide Antonutti, Mina Schütz, Lam Pham, Justin Ilyes, Péter Rjabcsenkó,

Organizations: Austrian Institute of Technology, Giefinggasse 4, 1210 Vienna - Austria

Abstract

The rapid progress of generative artificial intelligence has led to a significant increase in manipulated multimedia content, including synthetic faces, forged videos, AI-generated images, fake audio, and locally edited images. Existing detection approaches are typically designed for a single manipulation type, limiting their applicability in real-world scenarios involving heterogeneous media sources. In this work, we present the Media Intelligence Platform (MIP), a unified web-based framework for multimodal media forensics integrating specialized modules for fake video detection, face morphing detection, AI-generated image detection, audio deepfake detection, localized image manipulation detection, and text analysis. Its modular architecture enables flexible integration of heterogeneous models and supports future extensions for emerging manipulation techniques. Experimental results on representative benchmark datasets demonstrate reliable detection performance across multiple media modalities while providing a practical and user-friendly interface for forensic investigations. The platform highlights the potential of unified multimodal systems for comprehensive deepfake detection in real-world environments, and is accompanied by a public demonstrator released with this workshop submission to support reproducibility and community evaluation.

Figures & tables

Figure 1: MIP overview. The platform supports multimodal input upload, analysis tool selection, and job history tracking.

Figure 2: Example result view showing model predictions and explainable outputs.

Figure 3: Schematic of the MIP microservice architecture. Modules are deployed as Kubernetes pods and communicate via standardized REST APIs. Results and metadata are stored through a shared NFS file system and PostgreSQL backend.

Figure 4: MIP-UI: reference frontend for module selection, job history, and result visualization.

AI Detectors	ImgN100 [ 15 ] R	VOC12 [ 21 ] R	OImg-v7 [ 35 ] R	FLUX.1 [ 5 ] F	Firefly [ 1 ] F	MJ 5&6 [ 46 , 47 ] F	DALL-E 3 [ 58 ] F	GLIDE [ 56 ] F
ResNet50[fc]	87.1	82.5	96.4	98.2	92.2	96.7	98.1	84.9
ResNet50[fc+l4]	97.0	92.4	99.4	98.8	96.4	87.4	92.1	86.2
ResNet50[fc+l4+l3]	99.2	96.6	99.6	95.8	79.0	89.5	96.2	74.8
ViT-B-16	94.2	89.6	97.5	94.9	86.9	93.8	97.0	89.9

Table 1: F1-score (%) comparison of AI-generated image detectors on evaluation datasets. Best results per dataset are highlighted in bold. For each dataset, the image type is reported ( R for real and F for fake).

Figure 5: Example output of the IFDL module. The manipulated input image (left) is assigned a high manipulation probability, while the predicted localization mask (right) highlights the tampered region.

	Test Datasets
Models	CASIA [ 18 ]			Columbia [ 29 ]			DSO [ 14 ]			NIST [ 52 ]			Average
	AUC	F1	IoU	AUC	F1	IoU	AUC	F1	IoU	AUC	F1	IoU	AUC	F1	IoU	Mean
DFCN [ 88 ]	0.654	0.192	0.119	0.789	0.541	0.395	0.724	0.303	0.227	0.778	0.250	0.204	0.736	0.322	0.236	0.431
FSim [ 45 ]	0.554	0.169	0.102	0.731	0.604	0.474	0.796	0.487	0.371	0.642	0.188	0.123	0.681	0.362	0.268	0.437
MNet [ 80 ]	0.776	0.130	0.086	0.747	0.357	0.258	0.795	0.344	0.253	0.634	0.088	0.054	0.738	0.230	0.163	0.377
Wu22 [ 79 ]	0.873	0.509	0.465	0.862	0.707	0.608	0.854	0.436	0.308	0.783	0.332	0.255	0.843	0.496	0.409	0.583

Table 2: Comparison of state-of-the-art approaches and our proposed IFDL-Model in default mode respectively averaging over the results of the augmented (horizontally/vertically mirrored, rotated) input image. Highest values are highlighted in bold.

Algorithm	DM	G-DiM	MGAN-II	OCV	SGAN2	UTW	Avg.
SelfMAD [ 30 ]	51.45	5.45	2.90	0.59	73.25	52.98	31.10
SelfMAD++ [ 31 ]	48.22	25.04	2.04	0.85	46.68	14.48	22.89
SPL-MAD [ 22 ]	44.32	4.38	20.59	17.75	55.62	80.47	37.19
CrEffViT [ 10 ]	18.43	42.45	18.29	22.98	12.29	17.24	21.95
FatFormer [ 41 ]	16.31	10.87	1.19	6.80	12.43	12.50	10.02
Triplet-Loss [ 73 ]	11.85	5.33	0.98	5.10	11.56	0.70	5.92

Table 3: Equal Error Rate (EER %) obtained by the evaluated methods on the different morph generation techniques. From left to right the generation methods are: DiffMorpher, Greedy-DiM, MipGAN-II, OpenCV, StyleGAN2 and UTW.

Detector Method	Category	Video AUC	Reference
EFFORT	ViT, CLIP	0.928	[ 82 ]
selimsef FbF	CNN, Ensemble	0.917	[ 65 ]
GenConvViT	CNN, ViT	0.908	[ 78 ]
CC ViT	CNN, ViT	0.903	[ 10 ]
Efficient ViT	CNN, ViT	0.869	[ 10 ]
UCF	CNN, Disentanglement, Autoencoder	0.864	[ 83 ]

Table 4: Deepfake detection methods available on the Media-Intelligence-Platform. Performance on benchmark dataset, sorted by descending video AUC.

Figure 6: For each video and each detector result the report will display a bar chart for each person with the predicted faceswap detection confidence per frame or sequence.

Test Dataset	Accuracy	F1	AUC
In-The-Wild	0.87	0.79	0.83
ASVspoof5 Eval	0.91	0.91	0.92
FakeAVCeleb	0.99	0.99	0.99

Table 5: Cross-dataset performance of the DSD module.

Figure 7: Overview of the Deepfake Speech Detection (DSD) pipeline. The model uses a three-stage training strategy based on feature extraction, discriminative embedding learning, and Mahalanobis-based inference.

Multimodal Deepfake Detection

See all 33 papers

DeepFake Forensics AI: A Multi-Modal Detection and Blockchain-Anchored Evidence Management Platform
May 28, 2026Naisha MinnahAudio Deepfake Detection Image Forgery Detection
Department of Computer Science Providence Women’s College, Calicut Affiliated to University of Calicut, Kerala, India · Affiliated to University of Calicut, Kerala, India
DYMAPIA: A Multi-Domain Framework for Detecting AI-based Video Manipulation
Apr 27, 2026Md Shohel Rana, Andrew H. SungDigital Image Forensics Deepfake Detection
School of Computing Georgia Southern University Statesboro, GA 30460, USA · School of Computing Sciences & Computer Engineering The University of Southern Mississippi Hattiesburg, MS 39406, USA
Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook
Nov 29, 2024Florinel-Alin Croitoru, Andrei-Iulian Hiji, Vlad Hondru +7Audio Deepfake Detection Deepfake Detection
Department of Computer Science, University of Bucharest, Romania · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · Department of Computer Science, University of Central Florida, Orlando, US

May 28, 2026J/K move · Enter open · S save

DeepFake Forensics AI: A Multi-Modal Detection and Blockchain-Anchored Evidence Management Platform

Naisha Minnah

Department of Computer Science Providence Women’s College, Calicut Affiliated to University of Calicut, Kerala, India · Affiliated to University of Calicut, Kerala, India

Audio Deepfake Detection Image Forgery Detection Digital Forensics Deepfake Detection AI-Generated Image Detection Multimodal Deepfake Detection

Open paper AskPDF arXiv

The proliferation of AI-generated synthetic media poses a critical threat to the integrity of digital evidence in legal and forensic contexts. Existing deepfake detection systems typically address a single modality and provide no mechanism for tamper-proof evidence preservation. We present DeepFake Forensics AI, a unified platform that detects synthetic media across image, video, and audio modalities, identifies generative architecture fingerprints, and anchors forensic evidence immutably on the Ethereum blockchain. Our system trains four independent neural networks from scratch: an EfficientNet-B4 image detector (AUC = 0.9868), a Bidirectional LSTM video detector (AUC= 0.9628), an ECAPA-TDNN audio detector (EER = 18.63%), and a novel GAN fingerprinting module (accuracy = 99.88%) that identifies the generative architecture behind a fake image. Evidence files are hashed with SHA-256, stored on IPFS via Pinata, and registered on-chain via a Solidity smart contract with role-based access control. The platform provides a React frontend and FastAPI backend suitable for deployment in forensic and legal workflows. To our knowledge, this is the first system to unify multi-modal deepfake detection with blockchain-based chain-of custody management.

Abstract

Multimodal Deepfake Detection

Explore similar work

DeepFake Forensics AI: A Multi-Modal Detection and Blockchain-Anchored Evidence Management Platform

DYMAPIA: A Multi-Domain Framework for Detecting AI-based Video Manipulation

Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook