URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection

Zhenyu Wang; Weichen Cheng; Weijia Li; Junjie Mou; Zongyou Zhao; Guoying Zhang

cs.CVApr 8, 2026

URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection

Authors: Zhenyu Wang, Weichen Cheng, Weijia Li, Junjie Mou, Zongyou Zhao, Guoying Zhang

Organizations: School of Artificial Intelligence, China University of Mining and Technology-Beijing, Beijing 100083, China

Abstract

Multimodal sarcasm detection (MSD) aims to identify sarcastic intent from semantic incongruity between text and image. Although recent methods have improved MSD through cross-modal interaction and incongruity reasoning, most still treat modalities as equally reliable. In real social media posts, however, text and images often differ in noise level and relevance, making deterministic fusion susceptible to noisy evidence and weakened incongruity cues. To address this issue, we propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework for robust MSD. URMF first injects visual evidence into textual representations through multi-head cross-attention, and then applies self-attention in the fused semantic space to enhance incongruity reasoning. It models textual, visual, and interaction-aware representations as learnable Gaussian posteriors to estimate modality-specific uncertainty. Based on the estimated uncertainty, URMF dynamically adjusts modality contributions during fusion to suppress unreliable evidence. We further optimize the model with a unified objective that combines information bottleneck regularization, modality prior regularization, cross-modal distribution alignment, and uncertainty-driven contrastive learning. Experiments on the public MSD and MMSD2 benchmarks show that URMF outperforms representative unimodal, multimodal, and MLLM-based baselines. The results demonstrate that explicit uncertainty modeling can improve both accuracy and robustness in multimodal sarcasm detection.

Figures & tables

Figure 1: The overall framework of URMF.

Symbol	Value	Description
$n$	$100$	Maximum length of text tokens
$m$	$49$	Number of image patches
$d_{t}$	$768$	Text embedding dimension
$d_{i}$	$768$	Image embedding dimension
$\lambda_{\mathrm{IB}}$	$10^{-3}$	Weight of the KL term in $\mathcal{L}_{\mathrm{IB}}$
$\lambda_{1}$	$10^{-3}$	Weight of $\mathcal{L}_{\mathrm{reg}}$

Table 1: Hyperparameter settings used in the experiments.

Modality	Model	Acc(%)	P(%)	R(%)	F1(%)
Image	ResNet [ 2 ]	64.76	54.51	70.80	61.53
Image	ViT [ 4 ]	67.83	57.93	70.07	63.43
Text	Bi-LSTM [ 7 ]	81.90	76.66	78.42	77.53
	SIARN [ 20 ]	80.57	75.55	75.70	75.63
	SMSD [ 30 ]	80.90	76.46	75.18	75.82
	BERT [ 3 ]	83.85	78.72	82.27	80.22

Table 2: Main results on the MSD dataset. The best results are shown in bold, and the second-best results are underlined.

Model	Acc(%)	P(%)	R(%)	F1(%)
DynRT-Net [ 21 ]	71.40	71.80	72.17	71.34
Multi-view CLIP [ 18 ]	85.64	80.33	88.24	84.10
HKE-model [ 14 ]	76.50	73.48	72.07	72.25
LLaVA1.5 [ 19 ]	85.18	85.89	85.20	85.11
Att-BERT [ 16 ]	80.03	76.28	77.82	77.04
URMF	87.84	87.52	87.84	87.65

Table 3: Main results on the MMSD2 dataset.

Variant	Acc(%)	P(%)	R(%)	F1(%)
w/o $\mathcal{L}_{\mathrm{align}}$	86.09	86.27	86.97	86.05
w/o $\mathcal{L}_{\mathrm{IB}}$	86.67	86.80	87.51	86.62
w/o $\mathcal{L}_{\mathrm{reg}}$	85.93	86.02	86.73	85.87
w/o $\mathcal{L}_{\mathrm{UCL}}$	85.31	85.58	86.24	85.26
w/o Dynamic Fusion	86.43	86.26	86.94	86.33
Standard Transformer	84.39	85.05	85.57	84.37

Table 4: Ablation results of URMF on the MMSD2 dataset.

Figure 2: t-SNE visualization of the joint latent representations of major ablation variants and the full URMF on the MSD test set.

Model	Uncertainty Modeling			Unreliable Image
Model	Image	Text	Interaction	Acc(%)	F1(%)
ViT [ 4 ]	–	–	–	58.78	58.78
SCI-GDFN [ 29 ]	–	–	–	85.26	85.24
URMF		✓		92.53	92.43
	✓			–	–
	✓	✓		93.69	93.58

Table 5: Robustness and ablation results under the unreliable image setting.

Model	Uncertainty Modeling			Unreliable Text
Model	Image	Text	Interaction	Acc(%)	F1(%)
RoBERTa [ 18 ]	–	–	–	75.09	74.99
SCI-GDFN [ 29 ]	–	–	–	76.71	76.59
URMF		✓		–	–
	✓			80.07	79.84
	✓	✓		81.11	80.91

Table 6: Robustness and ablation results under the unreliable text setting.

Uncertainty-Aware Multimodal Fusion

Uncertainty-aware multimodal fusion combines data sources according to their reliability to improve robust, calibrated predictions.

See all 78 papers

Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection
Aug 20, 2026Hao Guo, Subin Huang, Junjie Chen +3Multimodal Fusion
School of Computer and Information Anhui Polytechnic University Wuhu, China · Bigdata and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing 100872, China
Focus on What You Need: Global-to-Local Dynamic Attention for Multimodal Sarcasm Detection
Oct 13, 2026Dehao Wu, Xingjie ZhuangSarcasm Detection Cross-Modal Attention
Xiamen University
PC-MNet: Dual-Level Congruity Modeling for Multimodal Sarcasm Detection via Polarity-Modulated Attention
May 4, 2026Maoheng Li, Ling Zhou, Xiaohua Huang +3Cross-Modal Attention Multimodal Fusion
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Oulu School, Nanjing Institute of Technology, Nanjing 210096, China · School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China, and also with the Macau University of Science and Technology Zhuhai MUST Science and Technology Research Institute, Zhuhai 519099, China +2

Aug 20, 2026J/K move · Enter open · S save

Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection

Hao Guo, Subin Huang, Junjie Chen, Zhifa Geng+2

School of Computer and Information Anhui Polytechnic University Wuhu, China · Bigdata and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing 100872, China

Multimodal Fusion

Open paper AskPDF arXiv

Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention. However, accurate detection remains challenging due to instance-dependent modality contributions and misleading semantic consistency, where surface-level alignment masks underlying contradictory intent. Existing methods often rely on fixed fusion strategies and treat sarcasm as generic cross-modal mismatch, limiting their ability to capture subtle sarcasm cues and instance-specific modality interactions. To address these challenges, we propose a novel MSD framework that integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization (SaCR). Specifically, a bidirectional gated interaction module performs cross-modal feature filtering and adaptively calibrates textual and visual contributions at the instance level. A dynamic fusion gate further balances modality importance to generate more robust multimodal representations. Furthermore, SaCR is introduced as a label-aware contrastive regularization objective that encourages semantic consistency for non-sarcastic samples while suppressing misleading consistency in sarcastic cases. The proposed framework is trained end-to-end with a multi-objective learning strategy that jointly optimizes multimodal classification and auxiliary unimodal supervision. Extensive experiments on MMSD and MMSD2.0 demonstrate that the proposed method consistently outperforms strong baselines.

Abstract

Uncertainty-Aware Multimodal Fusion

Explore similar work

Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection

Focus on What You Need: Global-to-Local Dynamic Attention for Multimodal Sarcasm Detection

PC-MNet: Dual-Level Congruity Modeling for Multimodal Sarcasm Detection via Polarity-Modulated Attention