Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification

Zizhao Chen; Ping Wei; Ziyang Ren; Huan Li; Xiangru Yin

cs.CVMar 27, 2026

Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification

Authors: Zizhao Chen, Ping Wei, Ziyang Ren, Huan Li, Xiangru Yin

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

Abstract

As multimodal misinformation becomes more sophisticated, its detection and grounding are crucial. However, current multimodal verification methods, relying on passive holistic fusion, struggle with sophisticated misinformation. Due to 'feature dilution,' global alignments tend to average out subtle local semantic inconsistencies, effectively masking the very conflicts they are designed to find. We introduce MaLSF (Mask-aware Local Semantic Fusion), a novel framework that shifts the paradigm to active, bidirectional verification, mimicking human cognitive cross-referencing. MaLSF utilizes mask-label pairs as semantic anchors to bridge pixels and words. Its core mechanism features two innovations: 1) a Bidirectional Cross-modal Verification (BCV) module that acts as an interrogator, using parallel query streams (Text-as-Query and Image-as-Query) to explicitly pinpoint conflicts; and 2) a Hierarchical Semantic Aggregation (HSA) module that intelligently aggregates these multi-granularity conflict signals for task-specific reasoning. In addition, to extract fine-grained mask-label pairs, we introduce a set of diverse mask-label pair extraction parsers. MaLSF achieves state-of-the-art performance on both the DGM4 and multimodal fake news detection tasks. Extensive ablation studies and visualization results further verify its effectiveness and interpretability.

Figures & tables

Figure 1: Overview of the misinformation and our framework. (a) A manipulated media content. (b) Traditional methods. (c) Our MaLSF framework. We utilize the BCV and HSA modules to perform explicit verification and aggregation of local semantics.

Figure 2: Parsers for extracting mask-label pairs. Open Vocabulary Parser generates mask-label pairs end-to-end. Caption-Anchored Parser utilizes GLIP [ 10 ] to obtain labels and corresponding bboxes, and then SAM2 [ 23 ] to obtain refined masks.

Figure 3: Overall architecture of MaLSF. In the figure, we take N=3 mask-label pairs as an example. The mask-label pairs combined with the original text and image are encoded as text features $\mathbf{l}_{\text{cap}},\mathbf{l}_{1},\mathbf{l}_{2},\mathbf{l}_{3}$ and image features $\mathbf{V}_{\text{img}},\mathbf{V}_{1},\mathbf{V}_{2},\mathbf{V}_{3}$ respectively. These features are subsequently encoded by BCV as multi-granularity validation features $\mathbf{F}_{\text{cap}}^{\text{img}},\mathbf{F}^{\text{cap}}_{\text{img}},\mathbf{F}_{\text{cap}}^{i},\mathbf{F}^{i}_{\text{img}},i=1,2,3$ .The HSA further aggregates the validation features to finally get the features $s_{\text{cap}},c_{l},c_{v},c_{b},c_{\text{bbox}}$ used for different tasks. TA, TS, FA, and FS are different text and face manipulation types, respectively.

Figure 4: Architecture of Hierarchical Semantic Aggregation. All the verification features are put through multi-label shallow fusion and multi-label deep fusion, respectively, to get the final features input to the linear header.

Categories	Binary Cls(%)			Multi-Label Cls(%)			Image Grounding(%)			Text Grounding(%)			$\Delta$ Avg
Methods	AUC	EER $\downarrow$	ACC	OF1	CF1	mAP	$\text{IoU}_{\text{mean}}$	$\text{IoU}_{\text{50}}$	$\text{IoU}_{\text{75}}$	Prec	Recall	F1	$\Delta$ Avg
CLIP [ 22 ]	83.22	24.61	76.40	66.00	59.52	62.31	49.51	50.03	38.79	58.12	22.11	32.03	–
ViLT [ 8 ]	85.16	22.88	78.38	72.37	66.14	66.00	59.32	65.18	48.10	66.48	49.88	57.00	–
HAMMER [ 26 ]	93.19	14.10	86.39	80.37	79.37	86.22	76.45	83.75	76.06	75.01	68.02	71.35	0
VIKI [ 12 ]	93.51	13.87	86.67	80.10	81.07	86.58	76.51	83.95	75.77	77.79	66.06	72.44	+0.38
HAMMER++ [ 27 ]	93.33	14.06	86.66	80.71	79.73	86.41	76.46	83.77	76.03	73.05	72.14	72.59	+0.40

Table 1: Comparison with state-of-the-art methods for multimodal media manipulation detection and grounding on DGM4. Bold and underlined text indicates the best and second-best performance metrics, respectively.

\text{MaLSF}^{\bigstar}

and

\text{MaLSF}^{\Diamond}

refer to the mask-label pairs obtained using the Caption-Anchored Parser and Open Vocabulary Parser, respectively. All metrics are reported in percentages (%).

	Method	Reference	Accuracy(%)	Fake News			Real News
	Method	Reference	Accuracy(%)	Precision(%)	Recall(%)	F1(%)	Precision(%)	Recall(%)	F1(%)
Weibo21	CAFE [ 2 ]	WWW’22	88.2	85.7	91.5	88.5	90.7	84.4	87.6
	BMR [ 39 ]	AAAI’23	92.9	90.8	94.7	92.7	94.6	90.6	92.5
	FND-CLIP [ 45 ]	ICME’23	94.3	93.5	94.5	94.0	95.0	94.2	94.6
	DAMMFND [ 19 ]	AAAI’25	94.7	–	–	94.8	–	–	94.7
	MMDFND [ 30 ]	MM’24	93.9	–	–	94.0	–	–	93.9

Table 2: Comparison with state-of-the-art methods for multimodal fake news detection on Weibo17 and Weibo21. Bold and underlined text indicates the best and second-best performance metrics, respectively.

\text{MaLSF}^{\bigstar}

and

\text{MaLSF}^{\Diamond}

refer to the mask-label pairs obtained Caption-Anchored Parser and Open Vocabulary Parser, respectively.

Components			Performance Metrics(%)
Label	Mask	ML Pair	ACC	CF1	$\text{IoU}_{\text{mean}}$	F1
			88.93	82.91	76.12	75.59
✓			89.26	83.43	80.78	76.69
	✓		89.24	83.60	82.02	75.90
		✓	89.37	83.93	82.37	77.19

Table 3: Ablation study of different modal local semantics in the proposed method. “Label” means only text labels are used, “Mask” means only image masks are used, and “ML pair” means the matching mask-label pairs are used.

Components				Performance Metrics(%)
I.B	T.B	T.V	I.V	ACC	CF1	$\text{IoU}_{\text{mean}}$	F1
	✓		✓	88.03	81.65	79.97	–
✓		✓		88.95	83.11	79.77	76.28
✓	✓	✓		88.87	83.46	81.42	76.53
✓	✓	✓	✓	89.37	83.93	82.37	77.19

Table 4: Ablation study of BCV modules. “I.B” and “T.B” denote Image Branch and Text Branch. “T.V” and “I.V” indicate text and image verification, which means using text or image as a query.

Components				Performance Metrics(%)
ML	GW	MSF	MDF	ACC	CF1	$\text{IoU}_{\text{mean}}$	F1
				88.90	83.38	81.74	75.87
✓				87.81	81.30	80.77	75.60
✓	✓			87.97	81.72	79.55	75.59
✓		✓	✓	89.18	83.65	81.35	76.97
✓	✓	✓		89.08	83.45	81.66	76.52

Table 5: Ablation study of the proposed modules. “ML” denotes mask-label pairs. “GW” denotes the gate weight module. “MSF” and “MDF” denote multi-label shallow fusion and multi-label deep fusion, respectively.

Figure 5: Qualitative analysis of results for HAMMER++ and MaLSF. The red bbox and text indicate the model prediction results. The blue text boxes and bounding boxes are ground truth. MaLSF is superior in both detection and grounding tasks.

Figure 6: Visualization of spatial attention of image features and the weights of local semantics. The mapping of local semantic weights to colors is shown as the color bar.

Method	Params	FPS	Avg. Pref.(%)
HAMMER++ [ 27 ]	441M	31.1	72.23
ASAP [ 44 ]	441M	31.1	74.24
EMSF [ 33 ]	328M	–	75.17
MaLSF(ours)	319M	22.9	76.88

Table 6: Comparison of the efficiency of the MaLSF with HAMMER++ [ 27 ] , ASAP [ 44 ] and EMSF [ 33 ] .

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 771 papers

KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection
Jun 2, 2026Kevin Patel, Shashi Bhushan JhaFake News Detection Multimodal Fusion
Department of Computer Science, University of West Florida, Pensacola, FL, USA
ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection
Jun 23, 2026Chenhao Dang, Dantong Zhu, Jun Yang +2Multimodal Agents Multimodal Model Evaluation
Shanghai Jiaotong University · Shanghai Artificial Intelligence Laboratory · Central South University +2
MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection
Aug 2, 2026Lehan Zhang, Yinlei Cheng, Shiqi Hu Yiheng Zhou +2Social Media Analysis Fake News Detection
Beijing Institute of Fashion Technology, Beijing, China · Shenyang University of Technology, Shenyang, China

Jun 2, 2026J/K move · Enter open · S save

KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection

Kevin Patel, Shashi Bhushan Jha

Department of Computer Science, University of West Florida, Pensacola, FL, USA

Fake News Detection Multimodal Fusion

Open paper AskPDF arXiv

Traditional fake news detection methods are falling behind as multimodal misinformation grows more advanced, seamlessly blending deceptive text, manipulated visuals, and factually incorrect claims. Most prior work focuses on text-image fusion or applies external knowledge only as a post-processing step, limiting their ability to detect deeper semantic inconsistencies. In this paper, we introduce KITE (Knowledge-Integrated Text-Image Encoder), a tri-modal fake news detection framework that jointly models textual, visual, and factual knowledge representations. KITE leverages Roberta and CLIP for linguistic and visual encoding, while a Graph Attention Network (GAT) processes structured facts retrieved from Wikidata. KITE uses cross-modal attention within a multimodal transformer to integrate text, visual, and knowledge features, helping it understand how each modality relates to one another. Modality-specific confidence scores are generated alongside the final prediction, offering interpretability by indicating which input type most influenced the decision. Evaluations on benchmark datasets demonstrate that KITE significantly outperforms unimodal and bimodal baselines, particularly in scenarios involving image-text mismatches or contradictions with external knowledge.

Abstract

Multimodal Fusion

Explore similar work

KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection

ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection

MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection