Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification

Zizhao Chen; Ping Wei; Ziyang Ren; Huan Li; Xiangru Yin

cs.CVMar 27, 2026

Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification

Authors: Zizhao Chen, Ping Wei, Ziyang Ren, Huan Li, Xiangru Yin

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

Abstract

As multimodal misinformation becomes more sophisticated, its detection and grounding are crucial. However, current multimodal verification methods, relying on passive holistic fusion, struggle with sophisticated misinformation. Due to 'feature dilution,' global alignments tend to average out subtle local semantic inconsistencies, effectively masking the very conflicts they are designed to find. We introduce MaLSF (Mask-aware Local Semantic Fusion), a novel framework that shifts the paradigm to active, bidirectional verification, mimicking human cognitive cross-referencing. MaLSF utilizes mask-label pairs as semantic anchors to bridge pixels and words. Its core mechanism features two innovations: 1) a Bidirectional Cross-modal Verification (BCV) module that acts as an interrogator, using parallel query streams (Text-as-Query and Image-as-Query) to explicitly pinpoint conflicts; and 2) a Hierarchical Semantic Aggregation (HSA) module that intelligently aggregates these multi-granularity conflict signals for task-specific reasoning. In addition, to extract fine-grained mask-label pairs, we introduce a set of diverse mask-label pair extraction parsers. MaLSF achieves state-of-the-art performance on both the DGM4 and multimodal fake news detection tasks. Extensive ablation studies and visualization results further verify its effectiveness and interpretability.

Figures & tables

Figure 1: Overview of the misinformation and our framework. (a) A manipulated media content. (b) Traditional methods. (c) Our MaLSF framework. We utilize the BCV and HSA modules to perform explicit verification and aggregation of local semantics.

Figure 2: Parsers for extracting mask-label pairs. Open Vocabulary Parser generates mask-label pairs end-to-end. Caption-Anchored Parser utilizes GLIP [ 10 ] to obtain labels and corresponding bboxes, and then SAM2 [ 23 ] to obtain refined masks.

Figure 3: Overall architecture of MaLSF. In the figure, we take N=3 mask-label pairs as an example. The mask-label pairs combined with the original text and image are encoded as text features $\mathbf{l}_{\text{cap}},\mathbf{l}_{1},\mathbf{l}_{2},\mathbf{l}_{3}$ and image features $\mathbf{V}_{\text{img}},\mathbf{V}_{1},\mathbf{V}_{2},\mathbf{V}_{3}$ respectively. These features are subsequently encoded by BCV as multi-granularity validation features $\mathbf{F}_{\text{cap}}^{\text{img}},\mathbf{F}^{\text{cap}}_{\text{img}},\mathbf{F}_{\text{cap}}^{i},\mathbf{F}^{i}_{\text{img}},i=1,2,3$ .The HSA further aggregates the validation features to finally get the features $s_{\text{cap}},c_{l},c_{v},c_{b},c_{\text{bbox}}$ used for different tasks. TA, TS, FA, and FS are different text and face manipulation types, respectively.

Figure 4: Architecture of Hierarchical Semantic Aggregation. All the verification features are put through multi-label shallow fusion and multi-label deep fusion, respectively, to get the final features input to the linear header.

Categories	Binary Cls(%)			Multi-Label Cls(%)			Image Grounding(%)			Text Grounding(%)			$\Delta$ Avg
Methods	AUC	EER $\downarrow$	ACC	OF1	CF1	mAP	$\text{IoU}_{\text{mean}}$	$\text{IoU}_{\text{50}}$	$\text{IoU}_{\text{75}}$	Prec	Recall	F1	$\Delta$ Avg
CLIP [ 22 ]	83.22	24.61	76.40	66.00	59.52	62.31	49.51	50.03	38.79	58.12	22.11	32.03	–
ViLT [ 8 ]	85.16	22.88	78.38	72.37	66.14	66.00	59.32	65.18	48.10	66.48	49.88	57.00	–
HAMMER [ 26 ]	93.19	14.10	86.39	80.37	79.37	86.22	76.45	83.75	76.06	75.01	68.02	71.35	0
VIKI [ 12 ]	93.51	13.87	86.67	80.10	81.07	86.58	76.51	83.95	75.77	77.79	66.06	72.44	+0.38
HAMMER++ [ 27 ]	93.33	14.06	86.66	80.71	79.73	86.41	76.46	83.77	76.03	73.05	72.14	72.59	+0.40

Table 1: Comparison with state-of-the-art methods for multimodal media manipulation detection and grounding on DGM4. Bold and underlined text indicates the best and second-best performance metrics, respectively.

\text{MaLSF}^{\bigstar}

and

\text{MaLSF}^{\Diamond}

refer to the mask-label pairs obtained using the Caption-Anchored Parser and Open Vocabulary Parser, respectively. All metrics are reported in percentages (%).

	Method	Reference	Accuracy(%)	Fake News			Real News
	Method	Reference	Accuracy(%)	Precision(%)	Recall(%)	F1(%)	Precision(%)	Recall(%)	F1(%)
Weibo21	CAFE [ 2 ]	WWW’22	88.2	85.7	91.5	88.5	90.7	84.4	87.6
	BMR [ 39 ]	AAAI’23	92.9	90.8	94.7	92.7	94.6	90.6	92.5
	FND-CLIP [ 45 ]	ICME’23	94.3	93.5	94.5	94.0	95.0	94.2	94.6
	DAMMFND [ 19 ]	AAAI’25	94.7	–	–	94.8	–	–	94.7
	MMDFND [ 30 ]	MM’24	93.9	–	–	94.0	–	–	93.9

Table 2: Comparison with state-of-the-art methods for multimodal fake news detection on Weibo17 and Weibo21. Bold and underlined text indicates the best and second-best performance metrics, respectively.

\text{MaLSF}^{\bigstar}

and

\text{MaLSF}^{\Diamond}

refer to the mask-label pairs obtained Caption-Anchored Parser and Open Vocabulary Parser, respectively.

Components			Performance Metrics(%)
Label	Mask	ML Pair	ACC	CF1	$\text{IoU}_{\text{mean}}$	F1
			88.93	82.91	76.12	75.59
✓			89.26	83.43	80.78	76.69
	✓		89.24	83.60	82.02	75.90
		✓	89.37	83.93	82.37	77.19

Table 3: Ablation study of different modal local semantics in the proposed method. “Label” means only text labels are used, “Mask” means only image masks are used, and “ML pair” means the matching mask-label pairs are used.

Components				Performance Metrics(%)
I.B	T.B	T.V	I.V	ACC	CF1	$\text{IoU}_{\text{mean}}$	F1
	✓		✓	88.03	81.65	79.97	–
✓		✓		88.95	83.11	79.77	76.28
✓	✓	✓		88.87	83.46	81.42	76.53
✓	✓	✓	✓	89.37	83.93	82.37	77.19

Table 4: Ablation study of BCV modules. “I.B” and “T.B” denote Image Branch and Text Branch. “T.V” and “I.V” indicate text and image verification, which means using text or image as a query.

Components				Performance Metrics(%)
ML	GW	MSF	MDF	ACC	CF1	$\text{IoU}_{\text{mean}}$	F1
				88.90	83.38	81.74	75.87
✓				87.81	81.30	80.77	75.60
✓	✓			87.97	81.72	79.55	75.59
✓		✓	✓	89.18	83.65	81.35	76.97
✓	✓	✓		89.08	83.45	81.66	76.52

Table 5: Ablation study of the proposed modules. “ML” denotes mask-label pairs. “GW” denotes the gate weight module. “MSF” and “MDF” denote multi-label shallow fusion and multi-label deep fusion, respectively.

Figure 5: Qualitative analysis of results for HAMMER++ and MaLSF. The red bbox and text indicate the model prediction results. The blue text boxes and bounding boxes are ground truth. MaLSF is superior in both detection and grounding tasks.

Figure 6: Visualization of spatial attention of image features and the weights of local semantics. The mapping of local semantic weights to colors is shown as the color bar.

Method	Params	FPS	Avg. Pref.(%)
HAMMER++ [ 27 ]	441M	31.1	72.23
ASAP [ 44 ]	441M	31.1	74.24
EMSF [ 33 ]	328M	–	75.17
MaLSF(ours)	319M	22.9	76.88

Table 6: Comparison of the efficiency of the MaLSF with HAMMER++ [ 27 ] , ASAP [ 44 ] and EMSF [ 33 ] .

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 768 papers

Jun 2, 2026

KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection

Traditional fake news detection methods are falling behind as multimodal misinformation grows more advanced, seamlessly blending deceptive text, manipulated visuals, and factually incorrect claims. Most prior work focuses on text-image fusion or applies external knowledge only as a post-processing step, limiting their ability to detect deeper semantic inconsistencies. In this paper, we introduce KITE (Knowledge-Integrated Text-Image Encoder), a tri-modal fake news detection framework that jointly models textual, visual, and factual knowledge representations. KITE leverages Roberta and CLIP for linguistic and visual encoding, while a Graph Attention Network (GAT) processes structured facts retrieved from Wikidata. KITE uses cross-modal attention within a multimodal transformer to integrate text, visual, and knowledge features, helping it understand how each modality relates to one another. Modality-specific confidence scores are generated alongside the final prediction, offering interpretability by indicating which input type most influenced the decision. Evaluations on benchmark datasets demonstrate that KITE significantly outperforms unimodal and bimodal baselines, particularly in scenarios involving image-text mismatches or contradictions with external knowledge.

Kevin Patel, Shashi Bhushan Jha

Department of Computer Science, University of West Florida, Pensacola, FL, USA

Fake News Detection Multimodal Fusion

Ask PDF

Jun 23, 2026

ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection

Multimodal misinformation detection is increasingly important because viral posts now combine long multilingual narratives, several images, mixed provenance, and subtle text--image framing errors. Existing benchmarks and methods remain poorly matched to this setting: they usually isolate short captions, single images, binary labels, or one manipulation source, while agentic verification remains costly under realistic evidence search. We present ReMMD, a realistic multilingual multi-image agentic verification framework for multimodal misinformation detection. ReMMD includes ReMMDBench, a real-world multimodal misinformation detection benchmark with 500 samples, 2,756 images, five monolingual languages, two cross-lingual settings, three text-length tiers, multi-image posts, five-way veracity labels, eight distortion labels, evidence provenance, and rationales. It also includes ReMMD-Agent, a persistent-memory verifier that decomposes posts into atomic points, builds a reusable evidence set, and predicts structured L1/L2/L3 outputs. Across proprietary systems, open LVLMs, MMD-Agent, and T2-Agent, ReMMD-Agent obtains the best five-way veracity performance, with 41.80% accuracy and 39.12% macro-F1 using GPT-5.2, while reducing cost by 17.5% relative to MMD-Agent and 79.9% relative to T2-Agent. The project is available at https://dang-ai.github.io/ReMMD.

Chenhao Dang, Dantong Zhu, Jun Yang +2

Shanghai Jiaotong University · Shanghai Artificial Intelligence Laboratory · Central South University +2

Multimodal Agents Multimodal Model Evaluation

Ask PDF

Aug 2, 2026

MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection

The rapid dissemination of multimodal content has intensified the spread of fabricated news, presenting a substantial threat to social integrity. A formidable challenge for current detection systems is identifying misinformation related to novel events in zero-shot scenarios. Prevailing zero-shot methods typically assess news items in isolation via semantic matching, a strategy that fails to recognize the recycled disinformation tactics from past campaigns and lacks the sophisticated reasoning needed to identify subtle, cross-modal discrepancies. To surmount these deficiencies, we introduce MRAFnd, a novel Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection. MRAFnd emulates a collaborative team of analysts to verify news veracity. The framework initiates with Multimodal Similarity-based News Retrieval to assemble a corpus of contextually analogous articles from an unlabeled reference database. Subsequently, during the Bifurcated Evidential Reasoning stage, agents perform a dual-directional analysis to extract critical patterns from the retrieved evidence. Finally, a Multi-Agent Collaborative Debate, involving Analyst and Arbiter agents, engages in a structured discourse to arrive at a definitive and robust conclusion. Comprehensive experiments on three benchmark datasets reveal that MRAFnd markedly surpasses state-of-the-art baselines, achieving an accuracy gain of up to 2.35% on the demanding Weibo-21 dataset.

Lehan Zhang, Yinlei Cheng, Shiqi Hu Yiheng Zhou +2

Beijing Institute of Fashion Technology, Beijing, China · Shenyang University of Technology, Shenyang, China

Social Media Analysis Fake News Detection

Ask PDF

Abstract

Multimodal Fusion

Explore similar work

KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection

ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection

MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection