SarcasTraps: Effect of Sarcasm on Turkish Hate Speech Detection

Niyazi Ahmet Metin; Dilara Keküllüoğlu

cs.CLOct 12, 2026

SarcasTraps: Effect of Sarcasm on Turkish Hate Speech Detection

Authors: Niyazi Ahmet Metin, Dilara Keküllüoğlu

Organizations: Sabanci University Istanbul, Türkiye

Abstract

Standard hate speech detection datasets often emphasize explicit harmful language, but directed harm can also be expressed indirectly, without clearly hateful wording. Sarcasm is one such form of indirect expression, as it can convey harmful intent through mock praise, exaggeration, humor, or apparent politeness. In this paper, we introduce SarcasTraps, a targeted external test set for analyzing the effect of sarcasm on Turkish hate speech detection. SarcasTraps contains 479 entries annotated along two dimensions: hate speech and sarcasm. We train classifiers using BERTurk and XLM-RoBERTa on two existing Turkish hate speech datasets and evaluate them on SarcasTraps, using it strictly as an external test set. We also evaluate GPT-4.1 and Llama-3.3-70B in a zero-shot setting on SarcasTraps. Results show that sarcasm substantially affects hate speech detection. Fine-tuned encoder models lose recall on sarcastic hate, while zero-shot LLMs maintain higher recall but suffer substantial precision drops on sarcastic examples. These findings show that both fine-tuned hate speech detectors and zero-shot LLMs struggle when sarcasm is involved, and SarcasTraps provides quantitative evidence of this challenge through a targeted Turkish test set. Due to the presence of sensitive and hateful language, SarcasTraps will be shared upon request for academic research purposes. Warning: This paper has examples from the dataset which contains offensive and hateful content towards groups of people.

Figures & tables

Title	Entry	H	S
Turkish and Greek soldiers swearing at each other	Made me laugh. Trash talk like this is better than actually fighting.	0	0
Do the Freemasons or the Jews rule the world?	Minibus drivers and taxi drivers rule the world.	0	1
A message for Afghans and Pakistanis	First of all, you are not our brothers. We don’t have predatory, dishonorable brothers like you. Leave. Get the f*** out of our country.	1	0
The invasion of Caddebostan Beach by Syrians	Oh, but they are Ansar; they are our brothers. What’s the big deal if they went into the sea in a chador, or if their men disturbed the peace of society? There is a war! They got tired while fleeing the war; shouldn’t they enjoy the beach a little bit?	1	1

Table 1: Examples from the four conditions in SarcasTraps. H indicates hate speech and S indicates sarcasm.

Model	In-domain Test				SarcasTraps (0), $n=278$
Model	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1
BERTurk ${}_{\text{SU-HS}}$	0.831 $\pm$ 0.010	0.704 $\pm$ 0.017	0.684 $\pm$ 0.058	0.693 $\pm$ 0.030	0.817 $\pm$ 0.009	0.871 $\pm$ 0.016	0.787 $\pm$ 0.011	0.827 $\pm$ 0.007
BERTurk ${}_{\text{Mayda}}$	0.940 $\pm$ 0.004	0.886 $\pm$ 0.010	0.852 $\pm$ 0.017	0.869 $\pm$ 0.009	0.812 $\pm$ 0.020	0.838 $\pm$ 0.018	0.818 $\pm$ 0.042	0.828 $\pm$ 0.022
XLM-RoBERTa ${}_{\text{SU-HS}}$	0.823 $\pm$ 0.005	0.689 $\pm$ 0.016	0.673 $\pm$ 0.013	0.680 $\pm$ 0.007	0.778 $\pm$ 0.028	0.898 $\pm$ 0.020	0.675 $\pm$ 0.046	0.770 $\pm$ 0.034
XLM-RoBERTa ${}_{\text{Mayda}}$	0.916 $\pm$ 0.006	0.854 $\pm$ 0.009	0.771 $\pm$ 0.040	0.810 $\pm$ 0.019	0.746 $\pm$ 0.027	0.854 $\pm$ 0.034	0.655 $\pm$ 0.041	0.740 $\pm$ 0.031

Table 2: Encoder performance on the corresponding in-domain test sets and the non-sarcastic subset of SarcasTraps. Precision, recall, and F1 are computed for the hate speech class. Results are reported as mean

\pm

standard deviation over five random seeds.

Model	SarcasTraps (0), $n=278$				SarcasTraps (1), $n=201$
Model	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1
BERTurk ${}_{\text{SU-HS}}$	0.817 $\pm$ 0.009	0.871 $\pm$ 0.016	0.787 $\pm$ 0.011	0.827 $\pm$ 0.007	0.630 $\pm$ 0.026	0.692 $\pm$ 0.034	0.372 $\pm$ 0.063	0.483 $\pm$ 0.060
BERTurk ${}_{\text{Mayda}}$	0.812 $\pm$ 0.020	0.838 $\pm$ 0.018	0.818 $\pm$ 0.042	0.828 $\pm$ 0.022	0.597 $\pm$ 0.007	0.604 $\pm$ 0.014	0.404 $\pm$ 0.045	0.483 $\pm$ 0.031
XLM-RoBERTa ${}_{\text{SU-HS}}$	0.778 $\pm$ 0.028	0.898 $\pm$ 0.020	0.675 $\pm$ 0.046	0.770 $\pm$ 0.034	0.623 $\pm$ 0.013	0.746 $\pm$ 0.024	0.294 $\pm$ 0.030	0.421 $\pm$ 0.032
XLM-RoBERTa ${}_{\text{Mayda}}$	0.746 $\pm$ 0.027	0.854 $\pm$ 0.034	0.655 $\pm$ 0.041	0.740 $\pm$ 0.031	0.536 $\pm$ 0.020	0.509 $\pm$ 0.035	0.317 $\pm$ 0.009	0.390 $\pm$ 0.014
GPT-4.1 Zero Shot	0.878	0.826	0.987	0.899	0.622	0.566	0.819	0.670

Table 3: Hate speech detection performance on the non-sarcastic and sarcastic subsets of SarcasTraps. Precision, recall, and F1 are computed for the hate speech class. Encoder results are reported as mean

\pm

standard deviation over five random seeds. Zero-shot LLM results are from deterministic decoding with temperature 0.

Hate Speech Detection

Hate Speech Detection identifies hateful or abusive language, its targets and intent, to support safer and more accountable content moderation.

See all 98 papers

CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection
Jul 13, 2026Qiyang Sun, Yi Chang, Yupei Li +3Language Model Calibration Multimodal Sentiment Analysis
GLAM – the Group on Language, Audio, & Music, Imperial College London, UK · College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, China · College of Computer Science and Electronic Engineering, Hunan University, China +2
Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text
Feb 25, 2026Bitan Majumder, Anirban SenSarcasm Detection Small Language Models
Pondicherry University · Ashoka University
Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection
Sep 18, 2025Zhu Li, Xiyuan Gao, Yuqing Zhang +2Sarcasm Detection Multimodal Model Evaluation
University of Groningen, The Netherlands

Jul 13, 2026J/K move · Enter open · S save

CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection

Qiyang Sun, Yi Chang, Yupei Li, Xi Shao+2

GLAM – the Group on Language, Audio, & Music, Imperial College London, UK · College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, China · College of Computer Science and Electronic Engineering, Hunan University, China+2

Language Model Calibration Multimodal Sentiment Analysis Multimodal Fusion Sarcasm Detection

Open paper AskPDF arXiv

Sarcasm detection, the identification of discrepancies between literal and intended meaning, is a fundamental task in affective computing. However, zero-shot instruction-tuned Large Language Models (LLMs) systematically over-predict the positive (sarcastic) class across the entire capability spectrum, while the prosodic cues humans rely on remain underexploited and transfer unevenly across languages. We introduce CHARM (Charge Calibration and Acoustic Rescue for Multimodal Sarcasm Detection), a training-free framework that couples two modules. Bidirectional Charge Calibration (BiCAL) steers the LLM toward opposing sarcastic and literal verdicts along a symmetric axis of charged prompts; the induced directional biases cancel by construction, and a simple aggregation recovers an unbiased pragmatic signal. Acoustic Late-Fusion Rescue (ALFR) then fuses the calibrated votes with prosodic descriptors and LLM-generated auditory-perception probes through a shallow classifier, actively down-weighting saturated text votes in favour of acoustic evidence. Without fine-tuning any backbone, BiCAL attains the highest reported zero-shot text-only Macro-F1 of 0.787 on MUStARD, while ALFR lifts weak backbones by up to +0.382 Macro-F1 on CMMA. A Stouffer meta-analysis confirms statistical significance on MUStARD and CMMA (Z = 13.89 and Z = 34.64, respectively; p < 10^-43). Our analysis further uncovers a cross-cultural prosodic decoupling: low-level acoustics fail to transfer across languages, whereas high-level perceptual abstractions remain robust. Together, these components yield an explainable, cross-lingual multimodal detector.

Abstract

Hate Speech Detection

Explore similar work

CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection

Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text

Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection