Focus on What You Need: Global-to-Local Dynamic Attention for Multimodal Sarcasm Detection

Dehao Wu; Xingjie Zhuang

cs.LGOct 13, 2026

Focus on What You Need: Global-to-Local Dynamic Attention for Multimodal Sarcasm Detection

Authors: Dehao Wu, Xingjie Zhuang

Organizations: Xiamen University

Abstract

Multimodal sarcasm detection has attracted increasing attention due to the widespread use of images and text on social media platforms. Existing studies generally treat the semantic or emotional inconsistency between images and text as a key cue for identifying sarcastic expressions. However, images in real-world scenarios often contain multiple visual objects, many of which are irrelevant to sarcasm-related semantics. Such redundant visual information may introduce noise during cross-modal interaction, distracting models from focusing on critical sarcasm cues and potentially leading to inaccurate predictions. To address this issue, we propose GLDA (Global-to-Local Dynamic Attention), a multimodal sarcasm detection framework that models cross-modal interactions from a global-to-local perspective. Specifically, GLDA first employs a global multimodal fusion module to capture high-level semantic interactions between visual and textual modalities, enabling the model to identify coarse-grained cross-modal inconsistencies. To further reduce the influence of irrelevant objects, GLDA introduces a local gated cross-modal attention mechanism that dynamically filters redundant visual and textual features and refines multimodal representations at a finer granularity. In addition, a multi-branch learning objective is designed to jointly optimize unimodal and multimodal representations. Experimental results demonstrate that GLDA effectively improves sarcasm detection performance and achieves competitive results compared with existing approaches. The code will be available at https://anonymous.4open.science/r/GLDA-DBE6.

Figures & tables

Figure 1. Comparison of our method with the other exist multimodal methods on the multimodal sarcasm detection task.

Figure 2. The overall architecture of the proposed GLDA framework. In Stage 1, a CLIP encoder extracts image and text token representations $Z_{I}$ and $Z_{T}$ , which are pooled into global vectors $\mathbf{z}_{i}$ (image) and $\mathbf{z}_{t}$ (text). In Stage 2, Path A (Global) fuses $Z_{I}$ and $Z_{T}$ via self-attention to produce the global representation $\mathbf{F}$ , while Path B (Local) applies bidirectional cross-attention with a gating mechanism to produce the gated representation $F_{g}$ . In Stage 3, three linear classifiers are applied: a shared classifier for $\mathbf{F}$ and $F_{g}$ , and two modality-specific classifiers for $\mathbf{z}_{i}$ and $\mathbf{z}_{t}$ , yielding predictions $\hat{y}_{\text{f}}$ , $\hat{y}_{\text{g}}$ , $\hat{y}_{\text{i}}$ , and $\hat{y}_{\text{t}}$ . These predictions are aggregated in Stage 4 for final sarcasm prediction.

Figure 3. Local gated cross-modal attention schematic diagram.

Task	Datasets	Total	Train		Val		Test
Task	Datasets	Total	#Positive	#Negative	#Positive	#Negative	#Positive	#Negative
Sarcasm Detection	MMSD ( Cai et al., 2019 )	19816	8642	11174	959	1451	959	1450
Sarcasm Detection	MMSD2.0 ( Qin et al., 2023 )	19816	9576	10240	1042	1368	1037	1372
Rumor Detection	Weibo ( Jin et al., 2017 )	13272	6263	4290	864	528	827	500
Rumor Detection	Twitter ( Boididou et al., 2016 )	15991	7518	6187	608	621	660	397

Table 1. Statistics of datasets for multi-modal sarcasm detection and rumor detection.

Method	MMSD				MMSD2.0
Method	Acc.(%)	P(%)	R(%)	F1(%)	Acc.(%)	P(%)	R(%)	F1(%)
Image-Only Methods
ResNet ( He et al., 2016 )	64.76	54.41	70.80	61.53	65.50	61.17	54.39	57.58
ViT ( Dosovitskiy, 2020 )	67.83	57.93	70.07	63.40	72.02	65.26	74.83	69.72
Text-Only Methods
SMSD ( Xiong et al., 2019 )	80.90	76.46	75.18	75.82	73.56	68.45	71.55	69.97

Table 2. Comparison of different methods on MMSD ( Cai et al., 2019 ) and MMSD2.0 ( Qin et al., 2023 ) dataset, where LoRA denotes the model fine-tuned with LoRA-based supervised fine-tuning (SFT).

Model	Training time(Hour)	GPU memory(GB)
LLaVA-v1.5-7B (LoRA)	12.3	39.2
GLDA	0.6	19.8

Table 3. Training time and GPU memory usage of different models.

$\mathcal{B}_{t}$	$\mathcal{B}_{i}$	$\mathcal{B}_{l}$	$\mathcal{B}_{g}$	MMSD		MMSD2.0
$\mathcal{B}_{t}$	$\mathcal{B}_{i}$	$\mathcal{B}_{l}$	$\mathcal{B}_{g}$	Acc.(%)	F1(%)	Acc.(%)	F1(%)
Category 1: Branch-based Ablation
	✓	✓	✓	90.15	89.23	85.06	84.75
✓		✓	✓	90.58	89.67	85.44	84.96
✓	✓		✓	89.42	88.15	84.68	83.23
✓	✓	✓		90.41	89.12	85.31	84.39

Table 4. The ablation results of modules in the model.

\mathcal{B}_{t}

\mathcal{B}_{i}

\mathcal{B}_{l}

and

\mathcal{B}_{g}

denote the text-only modality learning branch, the image-only modality learning branch, the global cross-modal learning branch, and the local cross-modal learning branch, respectively.

Figure 4. Parameter sensitivity analysis of

\lambda_{1}

and

\lambda_{2}

Figure 5. Parameter sensitivity analysis of gated cross-attention layers.

Model	Acc	Rumor			Non-Rumor
Model	Acc	P	R	F1	P	R	F1
Weibo Dataset
CAFE ( Chen et al., 2022 )	0.840	0.855	0.830	0.842	0.825	0.851	0.837
MRML ( Peng et al., 2023 )	0.897	0.898	0.887	0.892	0.896	0.905	0.901
HMCAN ( Qian et al., 2021 )	0.885	0.920	0.845	0.881	0.856	0.926	0.890
MFIR ( Wu et al., 2023a )	0.925	0.901	0.840	0.869	0.906	0.848	0.876

Table 5. Performance comparison on the Weibo and Twitter datasets.

Figure 6. Cross-modal attention visualization for a sarcastic sample from MMSD2.0. Left: Grad-CAM heatmap (Image $\leftarrow$ Text). Right: gated cross-attention weights (Text $\leftarrow$ Image).

Figure 7. Cross-modal attention visualization for a rumor sample from the Twitter dataset. Left: Grad-CAM heatmap (Image $\leftarrow$ Text). Right: gated cross-attention weights (Text $\leftarrow$ Image).

Figure 8. Two representative failure cases from MMSD2.0. (a) A meme-based sarcastic sample. (b) A weather-based sarcastic sample.

Sarcasm Detection

See all 21 papers

Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection
Aug 20, 2026Hao Guo, Subin Huang, Junjie Chen +3Multimodal Fusion
School of Computer and Information Anhui Polytechnic University Wuhu, China · Bigdata and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing 100872, China
URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection
Apr 8, 2026Zhenyu Wang, Weichen Cheng, Weijia Li +3Uncertainty-Aware Multimodal Fusion Multimodal Robustness
School of Artificial Intelligence, China University of Mining and Technology-Beijing, Beijing 100083, China
GRASP: Grounded CoT Reasoning with Dual-Stage Optimization for Multimodal Sarcasm Target Identification
Apr 10, 2026Faxian Wan, Xiaocui Yang, Yifan Cao +3Sarcasm Detection Multimodal CoT Reasoning
School of Computer Science and Engineering, Northeastern University, Shenyang, China

Aug 20, 2026J/K move · Enter open · S save

Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection

Hao Guo, Subin Huang, Junjie Chen, Zhifa Geng+2

School of Computer and Information Anhui Polytechnic University Wuhu, China · Bigdata and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing 100872, China

Multimodal Fusion

Open paper AskPDF arXiv

Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention. However, accurate detection remains challenging due to instance-dependent modality contributions and misleading semantic consistency, where surface-level alignment masks underlying contradictory intent. Existing methods often rely on fixed fusion strategies and treat sarcasm as generic cross-modal mismatch, limiting their ability to capture subtle sarcasm cues and instance-specific modality interactions. To address these challenges, we propose a novel MSD framework that integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization (SaCR). Specifically, a bidirectional gated interaction module performs cross-modal feature filtering and adaptively calibrates textual and visual contributions at the instance level. A dynamic fusion gate further balances modality importance to generate more robust multimodal representations. Furthermore, SaCR is introduced as a label-aware contrastive regularization objective that encourages semantic consistency for non-sarcastic samples while suppressing misleading consistency in sarcastic cases. The proposed framework is trained end-to-end with a multi-objective learning strategy that jointly optimizes multimodal classification and auxiliary unimodal supervision. Extensive experiments on MMSD and MMSD2.0 demonstrate that the proposed method consistently outperforms strong baselines.

Abstract

Sarcasm Detection

Explore similar work

Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection

URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection

GRASP: Grounded CoT Reasoning with Dual-Stage Optimization for Multimodal Sarcasm Target Identification