Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection
Organizations: School of Computer and Information Anhui Polytechnic University Wuhu, China · Bigdata and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing 100872, China
Abstract
Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention. However, accurate detection remains challenging due to instance-dependent modality contributions and misleading semantic consistency, where surface-level alignment masks underlying contradictory intent. Existing methods often rely on fixed fusion strategies and treat sarcasm as generic cross-modal mismatch, limiting their ability to capture subtle sarcasm cues and instance-specific modality interactions. To address these challenges, we propose a novel MSD framework that integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization (SaCR). Specifically, a bidirectional gated interaction module performs cross-modal feature filtering and adaptively calibrates textual and visual contributions at the instance level. A dynamic fusion gate further balances modality importance to generate more robust multimodal representations. Furthermore, SaCR is introduced as a label-aware contrastive regularization objective that encourages semantic consistency for non-sarcastic samples while suppressing misleading consistency in sarcastic cases. The proposed framework is trained end-to-end with a multi-objective learning strategy that jointly optimizes multimodal classification and auxiliary unimodal supervision. Extensive experiments on MMSD and MMSD2.0 demonstrate that the proposed method consistently outperforms strong baselines.
Figures & tables
| Dataset | Split | Total | Sarcastic | Non-sarcastic |
|---|---|---|---|---|
| MMSD | Train | 19,816 | 8,642 | 11,174 |
| Val | 2,410 | 959 | 1,451 | |
| Test | 2,409 | 959 | 1,450 | |
| MMSD2.0 | Train | 19,816 | 9,576 | 10,240 |
| Val | 2,410 | 1,042 | 1,368 | |
| Test | 2,409 | 1,037 | 1,372 |
| Modality | Method | MMSD | MMSD2.0 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Acc.% | P% | R% | F1% | Acc.% | P% | R% | F1% | ||
| Text | TextCNN | 80.03 | 74.29 | 76.39 | 75.32 | 71.61 | 64.62 | 75.22 | 69.52 |
| SMSD | 80.90 | 76.46 | 75.18 | 75.82 | 73.56 | 68.45 | 71.55 | 69.97 | |
| BERT | 83.60 | 78.50 | 82.51 | 80.45 | 76.52 | 74.48 | 73.09 | 73.91 | |
| Image | ResNet | 64.76 | 54.41 | 70.80 | 61.53 | 65.50 | 61.17 | 54.39 | 57.58 |
| ViT | 67.83 | 57.93 | 70.07 | 63.40 | 72.02 | 65.26 | 74.83 | 69.72 | |
| Variant | MMSD | MMSD2.0 | ||
|---|---|---|---|---|
| Acc.% | F1% | Acc.% | F1% | |
| Full | 92.62 | 92.33 | 89.66 | 89.36 |
| w/o CMI | 87.91 | 87.42 | 85.10 | 84.97 |
| w/o BiXAtt (only v t) | 87.48 | 87.03 | 81.86 | 81.70 |
| w/o BiXAtt (only t v) | 81.84 | 81.07 | 80.53 | 80.37 |
| w/o VGate | 92.08 | 91.77 | 88.71 | 88.59 |
Multimodal Fusion
Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.
See all 757 papers