Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection

Zhu Li; Xiyuan Gao; Yuqing Zhang; Shekhar Nayak; Matt Coler

cs.CLSep 18, 2025

Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection

Authors: Zhu Li, Xiyuan Gao, Yuqing Zhang, Shekhar Nayak, Matt Coler

Organizations: University of Groningen, The Netherlands

Abstract

Sarcasm detection remains challenging because sarcastic intent often depends on information across text, speech, and vision. We systematically evaluate large language models (LLMs) and multimodal LLMs (MLLMs) for sarcasm detection on English (MUStARD++) and Chinese (MCSD), considering both feature-extraction settings and direct classification. In the feature-extraction settings, audio provides the strongest unimodal performance on both datasets, while multimodal fusion shows dataset-dependent benefits. In direct classification, Qwen2.5-Omni achieves the strongest zero-shot performance among the evaluated models, with further substantial gains after fine-tuning. Error analysis further shows that additional modalities can both correct and introduce errors, highlighting the need for selective multimodal integration.

Figures & tables

Model	Setup	MCSD			MUStARD++
		P	R	F1	P	R	F1
LLaMA 3	ZS	55.3	53.7	48.5	58.8	55.7	51.2
8B	FS	64.3	56.0	46.7	64.1	58.0	52.6
	FT	71.0	70.9	70.9	59.5	58.9	58.3
Qwen2-Audio	ZS	54.5	50.7	35.6	56.2	55.1	52.7
7B	FS	53.9	53.9	53.7	54.2	54.0	53.6

Table 1: Macro-precision (P), macro-recall (R), and macro-F1 scores (F1) on MCSD and MUStARD++ under zero-shot (ZS), few-shot (FS), and fine-tuning (FT) settings. For the FS results, we report performance at

k=2

Setup	MCSD			MUStARD++
	P	R	F1	P	R	F1
T	58.6	58.6	58.4	56.5	55.5	53.9
A	66.4	61.2	56.3	58.6	58.6	58.6
V	52.8	52.3	51.0	51.3	51.2	51.0
T+A	59.5	57.9	54.8	61.9	61.8	61.8
T+V	56.1	56.0	55.4	59.0	58.3	57.7

Table 2: Zero-shot performance of Qwen2.5-Omni under different modality configurations on MCSD and MUStARD++. T, A, and V denote text, audio, and visual inputs, respectively. The text-only condition uses reference transcripts. The unimodal, bimodal, and trimodal settings provide a controlled analysis of modality contribution and reduced-modality availability.

Panel of Figure 2 — Figure 2: Confusion matrices for different modality configurations on (top) MCSD and (bottom) MUStARD++.

Figure 3: Errors corrected and newly introduced by adding modalities as input to Qwen2.5-Omni on MCSD (top) and MUStARD++ (bottom). T, A, and V denote text, audio, and visual inputs, respectively; “

\rightarrow

” denotes a transition from the modality configuration on the left to that on the right.

Figure 4: Few-shot performance comparison of MLLMs under different

k

values.

Figure 5: Fine-tuning performance comparison of large-scale models under different train sizes.

Figure 6: Macro-F1 score comparison for different models on MCSD and MUStARD++ across unimodal settings.

Model	MCSD			MUStARD++
	P	R	F1	P	R	F1
Base (T+A)	74.8	74.9	74.7	67.1	66.8	66.5
Large (T+A)	73.6	73.8	73.5	70.4	70.3	70.3
Base (T+V)	66.7	66.6	66.6	64.7	64.6	64.6
Large (T+V)	67.4	67.5	67.4	64.7	64.7	64.6
Base (A+V)	69.1	69.0	69.0	66.3	66.3	66.3

Table 3: Precision (P), Recall (R), and F1-scores (F1) in bimodal and trimodal settings, using models as feature extractors. Base (T): BERT; Large (T): LLaMA 3; Base (A): Wav2Vec 2.0; Large (A): Qwen2-Audio; Base (V): ResNet-50; Large (V): Qwen2.5-VL.

Sarcasm Detection

See all 21 papers

CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection
Jul 13, 2026Qiyang Sun, Yi Chang, Yupei Li +3Language Model Calibration Multimodal Sentiment Analysis
GLAM – the Group on Language, Audio, & Music, Imperial College London, UK · College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, China · College of Computer Science and Electronic Engineering, Hunan University, China +2
When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection
Aug 31, 2026Yongjian Chen, Pengfei Wei, Yiqun Sun +2Audio-Language Model Evaluation Pragmatic Reasoning in Language Models
Magellan Technology Research Institute (MTRI) · Center for Language and Cognition, University of Groningen, the Netherlands
SarcasmMiner: A Dual-Track Post-Training Framework for Robust Audio-Visual Sarcasm Reasoning
Mar 5, 2026Zhu Li, Yongjian Chen, Huiyuan Lai +3Sarcasm Detection Multimodal Reasoning
Speech Technology Lab, University of Groningen, The Netherlands · Center for Language and Cognition, University of Groningen, The Netherlands

Jul 13, 2026J/K move · Enter open · S save

CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection

Qiyang Sun, Yi Chang, Yupei Li, Xi Shao+2

GLAM – the Group on Language, Audio, & Music, Imperial College London, UK · College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, China · College of Computer Science and Electronic Engineering, Hunan University, China+2

Language Model Calibration Multimodal Sentiment Analysis Multimodal Fusion Sarcasm Detection

Open paper AskPDF arXiv

Sarcasm detection, the identification of discrepancies between literal and intended meaning, is a fundamental task in affective computing. However, zero-shot instruction-tuned Large Language Models (LLMs) systematically over-predict the positive (sarcastic) class across the entire capability spectrum, while the prosodic cues humans rely on remain underexploited and transfer unevenly across languages. We introduce CHARM (Charge Calibration and Acoustic Rescue for Multimodal Sarcasm Detection), a training-free framework that couples two modules. Bidirectional Charge Calibration (BiCAL) steers the LLM toward opposing sarcastic and literal verdicts along a symmetric axis of charged prompts; the induced directional biases cancel by construction, and a simple aggregation recovers an unbiased pragmatic signal. Acoustic Late-Fusion Rescue (ALFR) then fuses the calibrated votes with prosodic descriptors and LLM-generated auditory-perception probes through a shallow classifier, actively down-weighting saturated text votes in favour of acoustic evidence. Without fine-tuning any backbone, BiCAL attains the highest reported zero-shot text-only Macro-F1 of 0.787 on MUStARD, while ALFR lifts weak backbones by up to +0.382 Macro-F1 on CMMA. A Stouffer meta-analysis confirms statistical significance on MUStARD and CMMA (Z = 13.89 and Z = 34.64, respectively; p < 10^-43). Our analysis further uncovers a cross-cultural prosodic decoupling: low-level acoustics fail to transfer across languages, whereas high-level perceptual abstractions remain robust. Together, these components yield an explainable, cross-lingual multimodal detector.

Abstract

Sarcasm Detection

Explore similar work

CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection

When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection

SarcasmMiner: A Dual-Track Post-Training Framework for Robust Audio-Visual Sarcasm Reasoning