Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text

Bitan Majumder; Anirban Sen

cs.CLFeb 25, 2026

Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text

Authors: Bitan Majumder, Anirban Sen

Organizations: Pondicherry University · Ashoka University

Abstract

Sarcasm detection in multilingual and code-mixed environments remains a challenging task for natural language processing models due to structural variations, informal expressions, and low-resource linguistic availability. This study compares four large language models, Llama 3.1, Mistral, Gemma 3, and Phi-4, with a fine-tuned DistilBERT model for sarcasm detection in code-mixed Hinglish text. The results indicate that the smaller, sequentially fine-tuned DistilBERT model achieved the highest overall accuracy of 84%, outperforming all of the LLMs in zero and few-shot set ups, using minimal LLM generated code-mixed data used for fine-tuning. These findings indicate that domain-adaptive fine-tuning of smaller transformer based models may significantly improve sarcasm detection over general LLM inference, in low-resource and data scarce settings.

Figures & tables

Figure 1: DistilBERT training and fine-tuning pipeline

Original Sentence	Translated Sentence
Bernie Sanders wins Maine democratic caucus	Bernie Sanders n Λ Maine democratic caucus Ejt Λ
Meet the white house’s newest star: a whiteboard	White house k Λ newest star s Λ Emlo: ek whiteboard
Four consequences of a $15 minimum wage	$15 minimum wage k Λ cAr consequences
15 things to bring on your summer adventures	apn Λ summer adventures pr l Λ jAn Λ k Λ Ela Λ 15 things

Table 1: Example English-to-Hinglish translations

Panel of Figure 2 — Figure 2: Prompt used in Gemini 2.5 Pro for synthetic code-mixed data generation

Dataset Split	Sarcastic	Non-Sarcastic
Training	9380	9380
Fine-Tune (English)	1171	1171
Fine-Tune (Hinglish)	1171	1171
Testing	1172	1172

Table 2: Sarcasm dataset distribution

Split	Positive	Negative	Neutral
Fine-tuning	644	646	645

Table 3: Sentiment dataset distribution

Panel of Figure 3 — Figure 3: Prompt used for LLM-based sarcasm classification

Models	Zero-Shot				Few-Shot
Models	English		Code-mixed		English		Code-mixed
	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score
Llama 3.1	0.6681	0.66	0.4953	0.34	0.6907	0.68	0.6229	0.60
Mistral	0.6079	0.55	0.5341	0.42	0.5328	0.41	0.5030	0.34
Gemma 3	0.6553	0.60	0.5806	0.53	0.5034	0.34	0.5614	0.47
Phi 4	0.6945	0.69	0.6036	0.55	0.6540	0.62	0.6028	0.56

Table 4: LLM performance for sarcasm detection under zero-shot and few-shot prompting.

Fine-tuning data	Testing data	Accuracy	F1 Score
No fine-tune	Code-mixed [Sarcasm]	0.56868	0.47
English [Sarcasm]	Code-mixed [Sarcasm]	0.64462	0.61
Code-mixed [Sarcasm]	Code-mixed [Sarcasm]	0.83873	0.84
English [Sentiment]	Code-mixed [Sarcasm]	0.53882	0.47
Code-mixed [Sentiment]	Code-mixed [Sarcasm]	0.58873	0.59

Table 5: DistilBERT result summary: Code-mixed fine-tuning leads to best performance

Figure 4: AUPRC Curve after Fine-tuning with Code-mixed Sarcasm (red) and English Sarcasm (blue) data

Figure 5: Model performance with respect to different Training Size before Fine-tuning on Code-Mixed Sarcasm data

Figure 6: Model performance with respect to different Training Size before Fine-tuning on English Sarcasm data

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Table 6: Sarcastic code-mixed sentences misclassified by the best performing model

Table 7: Non-Sarcastic code-mixed sentences

Table 8: Examples of translated (code-mixed) sentences, with manually annotated quality of translation

	Pred. Sarc	Pred. Non-Sarc
Actual Sarc	1103	69
Actual Non-Sarc	860	312

Appendix

Table 9: Confusion matrix for Phi 4 (zero-shot)

	Pred. Sarc	Pred. Non-Sarc
Actual Sarc	474	698
Actual Non-Sarc	186	986

Appendix

Table 10: Confusion matrix for Llama 3.1 (few-shot)

Table 11: Confusion matrices for sentiment based transfer learning using DistilBERT

	Pred. Sarc	Pred. Non-Sarc
Actual Sarc	909	263
Actual Non-Sarc	115	1057

Appendix

Table 12: Confusion matrix for the best performing DistilBERT model

Table 13: Examples of Misclassified Sentences by the best performing model (DistilBERT fine-tuned on code-mixed)

Sarcasm Detection

See all 21 papers

Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection
Sep 18, 2025Zhu Li, Xiyuan Gao, Yuqing Zhang +2Sarcasm Detection Multimodal Model Evaluation
University of Groningen, The Netherlands
CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection
Jul 13, 2026Qiyang Sun, Yi Chang, Yupei Li +3Language Model Calibration Multimodal Sentiment Analysis
GLAM – the Group on Language, Audio, & Music, Imperial College London, UK · College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, China · College of Computer Science and Electronic Engineering, Hunan University, China +2
SarcasTraps: Effect of Sarcasm on Turkish Hate Speech Detection
Oct 12, 2026Niyazi Ahmet Metin, Dilara KeküllüoğluHate Speech Detection
Sabanci University Istanbul, Türkiye

Sep 18, 2025J/K move · Enter open · S save

Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection

Zhu Li, Xiyuan Gao, Yuqing Zhang, Shekhar Nayak+1

University of Groningen, The Netherlands

Sarcasm Detection Multimodal Model Evaluation Modality Reliance in Multimodal Models Audio-Language Model Evaluation

Open paper AskPDF arXiv

Sarcasm detection remains challenging because sarcastic intent often depends on information across text, speech, and vision. We systematically evaluate large language models (LLMs) and multimodal LLMs (MLLMs) for sarcasm detection on English (MUStARD++) and Chinese (MCSD), considering both feature-extraction settings and direct classification. In the feature-extraction settings, audio provides the strongest unimodal performance on both datasets, while multimodal fusion shows dataset-dependent benefits. In direct classification, Qwen2.5-Omni achieves the strongest zero-shot performance among the evaluated models, with further substantial gains after fine-tuning. Error analysis further shows that additional modalities can both correct and introduce errors, highlighting the need for selective multimodal integration.

Abstract

Sarcasm Detection

Explore similar work

Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection

CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection

SarcasTraps: Effect of Sarcasm on Turkish Hate Speech Detection