Moving beyond the traditional binary classification paradigm of Multimodal Sarcasm Detection, Multimodal Sarcasm Target Identification (MSTI) presents a more formidable challenge, requiring precise localization of fine-grained targets such as textual phrases and visual regions. Existing approaches predominantly rely on implicit cross-modal alignment, offering limited interpretability and suboptimal fine-grained localization. To address these limitations, we propose GRASP, Grounded Chain-of-Thought ReAsoning with Dual-Stage Optimization for Multimodal Sarcasm Prediction and Target Identification, a framework that integrates visual grounding with explicit Chain-of-Thought (CoT) reasoning to move beyond black-box MSTI. Specifically, we curate MSTI-MAX, a refined dataset that mitigates class imbalance and enriches multimodal sarcasm cues. We introduce Grounded CoT reasoning, which explicitly anchors sarcasm-related visual regions within the reasoning trajectory and prompts the model to articulate rationales before predicting the final classification labels and sarcasm targets. Furthermore, we employ a dual-stage outcome-supervised joint optimization strategy: Supervised Fine-Tuning with a coordinate-aware weighted loss, followed by Fine-Grained Target Policy Optimization. Extensive experiments demonstrate that GRASP outperforms existing baselines in fine-grained sarcasm target identification across modalities, and an LLM-as-a-Judge evaluation quantitatively measures the quality of internal reasoning chains. Our dataset and source code will be released on GitHub.
Figures & tables
Figure 1. An illustrative example of MSTI. The sarcasm arises from the contradiction between the specific textual span “soooo expected. not” and the localized visual region (depicting a perilously low-flying airplane), which collectively conveys the sarcastic intent. Our GRASP framework not only outputs fine-grained multimodal targets directly but also yields explicit grounding chain-of-thought reasoning. An example of multimodal sarcasm. The text expresses surprise with heavy irony, while the image shows a dangerously low-flying aircraft over highway traffic, creating a strong text-image semantic contradiction.
Dataset
Split
Sarcasm
Non-sarcasm
Total
MSTI 2.0
Train
3,500
46
3,546
Val
711
16
727
Test
729
13
742
Total
4,940
75
5,015
MSTI-MAX
Train
3,500
3,046
6,546
Val
711
516
1,227
Table 1. Statistics of MSTI 2.0 and MSTI-MAX datasets.
Figure 2. Comparison of the fine-grained distribution of sarcasm targets across different modalities in the MSTI-MAX and MSTI 2.0 datasets. To reflect the diverse nature of real-world sarcastic expressions, the targets within each instance are categorized into three types: exclusively in the text (Text-only), exclusively in the visual modality (Visual-only), and present in both modalities (Text & Visual). Proportions of different datasets, Fine-grained Sarcasm Modality Distribution of MSTI-MAX.
Figure 3. Overview of the GRASP training pipeline. The framework consists of two stages: (1) Supervised Fine-Tuning (SFT) with coordinate-aware weighted loss employing structured grounded CoT reasoning; (2) Fine-Grained Target Policy Optimization (FTPO) with multidimensional rewards, which further strengthens reasoning quality and visual grounding precision. Overview of the GRASP training pipeline.
Methods
Sarcasm Detection
Text Target
Visual Target
Acc
Precision
Recall
F1
EM
F1
AP
AP50
AP75
Text-only Baselines
BERT-base
80.99
87.86
78.46
82.89
17.23
39.20
–
–
–
BERT-large
81.59
85.98
79.97
82.87
21.74
42.53
–
–
–
Task-specific Multimodal Model
CofiPara
80.20
73.52
85.32
78.98
49.52
48.74
9.79
23.18
8.00
Table 2. Main results on the MSTI-MAX dataset. The best results in each column are highlighted in bold , and the second-best are underlined. “–” indicates the metric is not applicable. “*” means the reproduced result.
Figure 4. LLM-as-a-Judge evaluation results. Higher values indicate better performance on a 1–5 scale. LLM-as-a-Judge evaluation results.
Variants
Sarcasm Detection
Text Target
Visual Target
Acc
Precision
Recall
F1
EM
F1
AP
AP50
AP75
Ablation Study of GRASP Based on Qwen3-VL-4B
GRASP (Qwen3-VL-4B)
78.50
77.11
90.12
83.11
43.16
35.04
41.31
45.84
40.74
w/o FTPO (SFT only)
73.26
83.02
68.44
75.03
46.45
28.88
34.21
38.25
33.70
w/o SFT (FTPO only)
72.71
77.69
75.03
76.34
45.11
7.62
12.94
17.24
12.58
w/o Bbox-Weight
72.79
75.16
80.11
77.56
41.14
29.39
30.03
33.70
29.38
Table 3. Ablation study of the proposed components based on Qwen3-VL-4B and generalization performance across various foundational MLLMs on the MSTI-MAX dataset. The best results within each group are highlighted in bold .
Method
Acc
Precision
Recall
F1
LLaVA (Zero-Shot)
51.06
40.09
46.40
43.02
Qwen-VL (Zero-Shot)
40.63
32.44
35.53
33.63
GPT-4o (Zero-Shot)
71.07
79.52
71.07
70.24
LLaVA (CoT)
48.69
40.93
65.17
50.28
Qwen-VL (CoT)
58.86
56.82
58.67
57.26
GPT-4o (CoT)
74.26
65.81
72.68
68.92
Table 4. Generalization results on the MMSD 2.0 benchmark. Results of baselines are taken from GDCNet ( Zhang et al., 2026 ) .
Figure 5. A case study of GRASP on the MSTI task. A case study showing the complete GRASP reasoning and target identification process on a ride-hailing app sarcasm example.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
SFT
FTPO
LoRA Rank / Alpha
4 / 16
8 / 32
Learning Rate
3.0e-5
1.0e-5
Warmup
80 steps
5% ratio
Num Epochs
2
1
Effective Batch Size
8
16
Max Sequence Length
2048
2048
Appendix
Table 5. Training hyperparameters for SFT and FTPO stages.
λcoord
Sarcasm Detection
Text Target
Visual Target
Acc
Prec.
Rec.
F1
EM
F1
AP
AP50
AP75
1
72.79
75.16
80.11
77.56
41.14
29.39
30.03
33.70
29.38
5
77.54
82.61
78.19
80.34
46.14
30.06
37.62
41.82
37.06
10
78.50
77.11
90.12
83.11
43.16
35.04
41.31
45.84
40.74
15
69.00
67.77
89.99
77.31
46.38
11.90
22.92
25.39
22.70
Appendix
Table 6. Effect of the coordinate weight λcoord in the SFT stage on GRASP (Qwen3-VL-4B). Results for λcoord∈{1,10} are reported in the main paper; the additional settings λcoord∈{5,15} are supplemented here.
Setting
Sarcasm Detection
Text Target
Visual Target
Acc
Prec.
Rec.
F1
EM
F1
AP
AP50
AP75
GRASP
78.50
77.11
90.12
83.11
43.16
35.04
41.31
45.84
40.74
Equal-weight
78.66
79.67
85.46
82.46
43.32
32.66
39.81
43.83
39.36
Acc-heavy
75.12
77.78
80.66
79.19
42.67
31.20
36.46
40.30
35.85
Format-enhanced
76.73
76.51
87.11
81.46
40.18
32.29
39.22
43.33
38.88
Appendix
Table 7. Effect of FTPO reward weight allocation on GRASP (Qwen3-VL-4B). Larger weights on the grounding-oriented rewards Rbox and Rtxt yield the best overall trade-off across MSD and MSTI metrics.
Multimodal sarcasm detection requires resolving pragmatic incongruity across textual, acoustic, and visual cues through cross-modal reasoning. To enable robust sarcasm reasoning with foundation models, we propose SarcasmMiner, a reinforcement learning based post-training framework that resists hallucination in multimodal reasoning. We reformulate sarcasm detection as structured reasoning and adopt a dual-track distillation strategy: high-quality teacher trajectories initialize the student model, while the full set of trajectories trains a generative reward model (GenRM) to evaluate reasoning quality. The student is optimized with group relative policy optimization (GRPO) using decoupled rewards for accuracy and reasoning quality. On MUStARD++, SarcasmMiner increases F1 from 59.83% (zero-shot), 68.23% (supervised finetuning) to 70.22%. These findings suggest that reasoning-aware reward modeling enhances both performance and multimodal grounding.
Zhu Li, Yongjian Chen, Huiyuan Lai +3
Speech Technology Lab, University of Groningen, The Netherlands · Center for Language and Cognition, University of Groningen, The Netherlands
Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention. However, accurate detection remains challenging due to instance-dependent modality contributions and misleading semantic consistency, where surface-level alignment masks underlying contradictory intent. Existing methods often rely on fixed fusion strategies and treat sarcasm as generic cross-modal mismatch, limiting their ability to capture subtle sarcasm cues and instance-specific modality interactions. To address these challenges, we propose a novel MSD framework that integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization (SaCR). Specifically, a bidirectional gated interaction module performs cross-modal feature filtering and adaptively calibrates textual and visual contributions at the instance level. A dynamic fusion gate further balances modality importance to generate more robust multimodal representations. Furthermore, SaCR is introduced as a label-aware contrastive regularization objective that encourages semantic consistency for non-sarcastic samples while suppressing misleading consistency in sarcastic cases. The proposed framework is trained end-to-end with a multi-objective learning strategy that jointly optimizes multimodal classification and auxiliary unimodal supervision. Extensive experiments on MMSD and MMSD2.0 demonstrate that the proposed method consistently outperforms strong baselines.
Hao Guo, Subin Huang, Junjie Chen +3
School of Computer and Information Anhui Polytechnic University Wuhu, China · Bigdata and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing 100872, China
Multimodal sarcasm detection has attracted increasing attention due to the widespread use of images and text on social media platforms. Existing studies generally treat the semantic or emotional inconsistency between images and text as a key cue for identifying sarcastic expressions. However, images in real-world scenarios often contain multiple visual objects, many of which are irrelevant to sarcasm-related semantics. Such redundant visual information may introduce noise during cross-modal interaction, distracting models from focusing on critical sarcasm cues and potentially leading to inaccurate predictions. To address this issue, we propose GLDA (Global-to-Local Dynamic Attention), a multimodal sarcasm detection framework that models cross-modal interactions from a global-to-local perspective. Specifically, GLDA first employs a global multimodal fusion module to capture high-level semantic interactions between visual and textual modalities, enabling the model to identify coarse-grained cross-modal inconsistencies. To further reduce the influence of irrelevant objects, GLDA introduces a local gated cross-modal attention mechanism that dynamically filters redundant visual and textual features and refines multimodal representations at a finer granularity. In addition, a multi-branch learning objective is designed to jointly optimize unimodal and multimodal representations. Experimental results demonstrate that GLDA effectively improves sarcasm detection performance and achieves competitive results compared with existing approaches. The code will be available at https://anonymous.4open.science/r/GLDA-DBE6.