Benchmarking Large Language Models for Detecting Satirical Code-Mixed Sentences

Răzvan-Alexandru Smădu; Ioana Buhnila; Daniela Tanase; Dumitru-Clementin Cercel; Florin Pop; Mihaela-Claudia Cercel

cs.CLOct 21, 2026

Benchmarking Large Language Models for Detecting Satirical Code-Mixed Sentences

Authors: Răzvan-Alexandru Smădu, Ioana Buhnila, Daniela Tanase, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel

Organizations: National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania · Academy of Romanian Scientists, Bucharest, Romania · Center for Data Science in Humanities, Chosun University, South Korea · National Institute for Research & Development in Informatics - ICI Bucharest, Bucharest, Romania ·

Abstract

Code-switching is a natural linguistic phenomenon, yet its impact on high-level reasoning tasks remains largely unexplored in under-resourced languages like Romanian. Natural language processing tasks, such as satire detection, rely heavily on nuanced cultural and syntactic cues that large language models struggle to capture. In this paper, we investigate four methods (i.e., random replacement, phrase translation, dependency parsing, and large language models) for generating synthetic code-mixed Romanian sentences with English, German, French, and Italian. We start with an existing dataset for satire detection, compiled from Romanian news articles and annotated at the sentence level by humans, and augment it with over 600k samples, namely CS-SeLeRoSa. We provide an in-depth analysis of the dataset generation process. Then, we evaluate several multilingual large language models across zero-shot, chain-of-thought, and fine-tuning regimes on both the synthetic corpus and a human-adjudicated gold benchmark, namely CS-SeLeRoSa-Gold. While older open models suffer substantial degradation (dropping to 37%-55% macro-F1), newer models like Gemma 4 31B achieve 68.1% zero-shot, rivaling frontier models like GPT-5 mini (69.5%). Furthermore, fine-tuning significantly reduces formatting errors and improves the results, increasing Gemma 4 31B to 77.5% on CS-SeLeRoSa and 72.6% on the gold standard, outperforming frontier zero-shot baselines.

Figures & tables

Figure 1: End-to-end dataset generation for CS-SeLeRoSa and CS-SeLeRoSa-Gold . The data generation pipeline starts from the original Romanian monolingual SeLeRoSa corpus. Four code-switching (CS) generation strategies are applied to create the CS-SeLeRoSa dataset across

K=3

iterations, as presented in § 3.3 : random replacement, phrase translation, dependency tree-based, and LLM generation. From the resulting 600k+ code-switched sentences, we select 715 unique sentences which are then human-validated to produce CS-SeLeRoSa-Gold . The four methods are illustrated on a sentence showing generated variants (e.g.,

k=1

k=2

) for English.

Language
RO	EN	DE	FR	IT
13,873	165,725	165,725	165,725	165,803

Table 1: Summary of the entire CS-SeLeRoSa code-mixed dataset statistics.

Model	SeLeRoSa			CS-SeLeRoSa			CS-SeLeRoSa-Gold
Model	ZS	CoT	FT	ZS	CoT	FT	ZS	CoT	FT
Llama 3.1 8B	46.3 ±11.1	53.3 ±1.9	69.0 ±0.1	44.8 ±15.2	47.9 ±2.2	74.9 ±0.8	43.0 ±12.8	43.2 ±2.2	65.7 ±0.0
Gemma 3 27B	49.9 ±0.6	53.2 ±0.4	74.3 ±1.3	37.0 ±0.3	41.3 ±1.8	75.4 ±0.8	36.1 ±0.8	39.1 ±3.2	67.2 ±0.8
Gemma 4 26B A4B	67.0 ±0.0	68.1 ±0.5	70.2 ±0.5	63.7 ±0.1	64.3 ±0.1	70.2 ±0.6	58.2 ±0.3	61.5 ±0.3	63.7 ±0.8
Gemma 4 31B	70.2 ±0.1	69.4 ±0.1	78.3 ±0.8	68.1 ±0.0	66.5 ±0.0	77.5 ±0.1	61.8 ±0.2	63.0 ±0.5	72.6 ±0.4
RoMistral 7B	41.2 ±11.5	49.6 ±0.6	72.5 ±1.4	36.8 ±14.4	48.3 ±1.0	74.6 ±0.6	35.7 ±13.7	46.6 ±1.5	70.4 ±0.5

Table 2: Macro-averaged F1-score (%) results on the three evaluation sets from monolingual SeLeRoSa, CS-SeLeRoSa, and CS-SeLeRoSa-Gold. Models are evaluated in zero-shot (ZS), chain-of-thought (CoT), and fine-tuning (FT) settings. The first set of models are open-source models evaluated in all settings, the second set are closed-source models evaluated only in ZS or CoT settings, and the third set represents frontier models evaluated only on the CS-SeLeRoSa-Gold test set. Bold indicates the best performance per setting and dataset, across all models.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Model	BLEU $\uparrow$	chrF $\uparrow$	TER $\downarrow$	COMET $\uparrow$
RO $\rightarrow$ EN
Gemma 3 12B	41.04	67.30	46.23	0.89
Gemma 3 27B	42.80	69.37	43.31	0.90
GPT-5 mini	40.07	67.60	46.13	0.89
RO $\rightarrow$ DE
Gemma 3 12B	23.61	52.45	70.19	0.82

Appendix

Table 3: Translation performance across models and language pairs RO

\rightarrow

X (EN, DE, FR, IT) on the FLORES+ dataset.

Method	CMI	M-Index	I-Index	LE	SP-Avg
RO $\rightarrow$ EN
Random	39.29	0.44	0.47	0.94	0.46
Phrase	15.93	0.18	0.22	0.55	0.21
Dependency	18.75	0.21	0.16	0.57	0.16
LLM	18.64	0.21	0.19	0.62	0.21
RO $\rightarrow$ DE

Appendix

Table 4: Aggregated code-switching metrics by language and code-switching method.

Method	BERTScore			Jaccard
Method	Mean	Std	Median	Mean	Std	Median
RO $\rightarrow$ EN
Random	8.03	1.91	7.90	73.28	7.85	72.77
Phrase	3.33	1.51	3.37	29.80	13.23	30.68
Dependency	3.31	0.95	3.22	37.37	6.53	37.89
LLM	3.61	2.05	3.08	39.06	19.66	34.41

Appendix

Table 5: Semantic distance and Jaccard dissimilarity for code-switched sentences by language and method.

Label	Method	Original (RO)	Code-Switched	Translated (EN)
RO $\rightarrow$ EN
Satire	Phrase	Ăla poate să distrugă şi şalele transplantate, şi pe ale tale.	Ăla poate să destroy şi şalele transplantate, şi pe ale tale.	That one can destroy both the transplanted loins and your own.
Satire	LLM	De s , i sunt pe cartelă, după ce consumam net, primeam altul.	De s , i sunt pe cartelă, after I consumam net, primeam another one.	Although I am on a prepaid card, after I consumed the internet, I would receive another one.
Regular	Random	Totul din lemn, calitate norvegiană.	Everything din wood, quality Norwegian.	Everything made of wood, Norwegian quality.
Regular	Dependency	Cât să mai răbdăm, că nici noi nu suntem de fier!	Cât să mai răbdăm, that we are not made of iron either!	How much more must we endure, since we are not made of iron either!
RO $\rightarrow$ DE

Appendix

Table 6: Examples of original Romanian sentences, their synthetically generated code-switched counterparts, and their English translations from the CS-SeLeRoSa dataset. The selection is grouped by target languages (EN, DE, FR, IT), covering all four generation methods and annotation labels (Satire vs. Regular) with non-trivial examples.

Model Name	Checkpoint	Size
Gemma 3 12B	google/gemma-3-12b-it	12B
Gemma 3 27B	google/gemma-3-27b-it	27B
Gemma 4 26B A4B	google/gemma-4-26B-A4B-it	26B
Gemma 4 31B	google/gemma-4-31B-it	31B
Llama 3.1 8B	meta-llama/Llama-3.1-8B-Instruct	8B
RoGemma 2 9B	OpenLLM-Ro/RoGemma2-9b-Instruct-DPO	9B

Appendix

Table 7: Model checkpoints used in our experiments.

Figure 2: Screenshot of an example from the Google Form used during the annotation process. It indicates the batch identifier, progress through the batch, sample identifier, matrix, and target languages, the original and synthetic sentences, and the original label. After this, there are the five mandatory questions and an optional comments field. The original screenshot was anonymized and split into three columns for presentation purposes.

Language	Adjudicated	Keep	AR (%)
English	199	117	58.8%
German	159	70	44.0%
French	198	118	59.6%
Italian	159	86	54.1%
Total	715	391	54.7%

Appendix

Table 8: Language distribution of the original annotated set and human verification acceptance rates (AR) after the annotation process.

Figure 3: The acceptance rate by method and target language.

Figure 4: The main reasons the samples were rejected, as a percentage of the rejected samples. Note that multiple reasons could lead to a rejected outcome.

Figure 5: Per-criteria distribution of the human labels after the adjudication process, on class preservation, linguistic acceptability, meaning preservation, and code-switching quality.

Figure 6: Final verdict distribution after the adjudication process.

Lang.	CP	LA	MP	CQ	Acc.
English	-0.001	1.373	2.070	1.844	94.0
German	0.770	1.883	1.004	1.333	90.6
French	0.209	1.232	2.127	1.833	93.4
Italian	0.156	1.759	1.468	1.884	91.8
All	0.556	1.858	2.089	1.858	92.9

Appendix

Table 9: Logistic regression coefficients indicating criterion influence on final verdict, along with the model accuracy (Acc.) as a percentage. Higher values indicate a stronger influence on accepting the sample.

Figure 7: Main disagreements between annotators considering category, generation method, and original class.

Criterion	$\alpha$	Agreement (%)
Class Preservation	0.178	85.0%
Linguistic Acceptability	0.603	72.3%
Meaning Preservation	0.609	76.5%
CS Quality	0.655	66.6%
Final Verdict	0.518	75.9%

Appendix

Table 10: Inter-annotator agreement in terms of Krippendorff’s alpha (

\alpha

) and agreement rate across all criteria for all N=715 samples.

Language	CP	LA	MP	CSQ	Verdict
English	0.205	0.476	0.629	0.679	0.438
German	0.315	0.723	0.503	0.731	0.634
French	-0.024	0.364	0.644	0.514	0.416
Italian	0.220	0.672	0.675	0.670	0.617

Appendix

Table 11: Inter-annotator agreement (Krippendorff’s

\alpha

) across target languages by verdict, linguistic acceptability (LA), code-switching quality (CSQ), class preservation (CP), and meaning preservation (MP).

Method	EN	DE	FR	IT	Total
Random	4	1	9	3	17
Phrase	25	12	26	17	80
Dependency	43	26	39	31	139
LLM	45	31	44	35	155
Total	117	70	118	86	391

Appendix

Table 12: Distribution of the gold standard dataset across code-switching generation methods and target languages.

Statistic	Value
Total samples	391
Unique source sentences	50
CS methods	4
Target languages	4
K values	3
Satire samples	218 (55.8%)

Appendix

Table 13: Summary statistics of the gold standard dataset used for evaluating code-switching generation methods.

Method	CMI	M-Index	I-Index	LE	SP-Avg
RO $\rightarrow$ EN
Random	24.60	0.28	0.34	0.65	0.34
Phrase	15.51	0.18	0.21	0.56	0.21
Dependency	23.47	0.26	0.19	0.68	0.19
LLM	17.14	0.20	0.20	0.59	0.20
RO $\rightarrow$ DE

Appendix

Table 14: Aggregated code-switching metrics by language and code-switching method on the gold set (i.e., keep set, N=391).

Model	Zero-Shot		Chain-of-Thought		Fine-Tuning
Model	Acc (%)	F1 (%)	Acc (%)	F1 (%)	Acc (%)	F1 (%)
Llama 3.1 8B	43.9 ±14.9	46.3 ±11.1	52.2 ±2.3	53.3 ±1.9	69.0 ±0.1	69.0 ±0.1
Gemma 3 27B	53.0 ±0.4	49.9 ±0.6	55.5 ±0.2	53.2 ±0.4	74.9 ±1.4	74.3 ±1.3
Gemma 4 26B A4B	66.9 ±0.0	67.0 ±0.0	68.0 ±0.5	68.1 ±0.5	70.1 ±0.5	70.2 ±0.5
Gemma 4 31B	70.2 ±0.1	70.2 ±0.1	69.4 ±0.1	69.4 ±0.1	78.5 ±0.8	78.3 ±0.8
RoMistral 7B	39.7 ±14.2	41.2 ±11.5	48.6 ±0.2	49.6 ±0.6	73.1 ±1.7	72.5 ±1.4

Appendix

Table 15: Monolingual baseline results on the Romanian-only (non-code-switched) test split across zero-shot (ZS), chain-of-thought (CoT), and fine-tuning (FT) settings. Bold indicates the best performance.

Model	Zero-Shot		Chain-of-Thought		Fine-Tuning
Model	Acc (%)	F1 (%)	Acc (%)	F1 (%)	Acc (%)	F1 (%)
Llama 3.1 8B	46.0 ±24.2	47.5 ±21.3	48.7 ±2.3	49.8 ±2.2	60.7 ±3.1	60.6 ±3.1
Gemma 3 27B	42.7 ±2.3	39.5 ±2.8	46.7 ±7.0	44.3 ±8.1	68.0 ±5.3	66.7 ±5.3
Gemma 4 26B A4B	52.0 ±0.0	51.9 ±0.0	60.0 ±0.0	60.0 ±0.8	57.3 ±2.3	56.9 ±2.6
Gemma 4 31B	64.7 ±1.2	63.1 ±1.0	66.0 ±2.0	64.7 ±2.1	72.0 ±2.0	70.8 ±1.8
RoMistral 7B	35.3 ±16.8	35.8 ±15.7	40.7 ±1.2	41.8 ±1.9	65.3 ±3.1	64.5 ±3.5

Appendix

Table 16: Monolingual baseline results on the original 50 Romanian sentences used to derive CS-SeLeRoSa-Gold test set across zero-shot (ZS), chain-of-thought (CoT), and fine-tuning (FT) settings. Bold indicates the best performance.

Model	Zero-Shot		Chain-of-Thought		Fine-Tuning
Model	Acc (%)	F1 (%)	Acc (%)	F1 (%)	Acc (%)	F1 (%)
Llama 3.1 8B	42.0 ±19.6	44.8 ±15.2	47.8 ±2.6 †	47.9 ±2.2 †	75.8 ±0.2 †	74.9 ±0.8 †
Gemma 3 27B	44.9 ±0.2 †	37.0 ±0.3 †	47.3 ±1.0 †	41.3 ±1.8 †	75.7 ±0.9 †	75.4 ±0.8 †
Gemma 4 26B A4B	63.8 ±0.0 †	63.7 ±0.1 †	64.3 ±0.1 †	64.3 ±0.1 †	70.2 ±0.6 †	70.2 ±0.6 †
Gemma 4 31B	68.1 ±0.0	68.1 ±0.0	66.5 ±0.0	66.5 ±0.0	78.0 ±0.1	77.5 ±0.1
RoMistral 7B	35.7 ±15.2	36.8 ±14.4	46.3 ±0.7 †	48.3 ±1.0 †	75.8 ±0.2 †	74.6 ±0.6 †

Appendix

Table 17: Results on the code-switched dataset across zero-shot (ZS), chain-of-thought (CoT), and fine-tuning (FT) settings. Bold indicates the best performance. † indicates statistically significant difference compared to Gemma 4 31B, or its fine-tuned variant in the FT setting (

p<0.05

; for

N>1

, Welch’s

t

-test; for

N=1

, paired McNemar’s test for accuracy and a paired bootstrap for F1).

Model	Zero-Shot		Chain-of-Thought		Fine-Tuning
Model	Acc (%)	F1 (%)	Acc (%)	F1 (%)	Acc (%)	F1 (%)
Llama 3.1 8B	40.2 ±17.2	43.0 ±12.8	43.0 ±2.3 †	43.2 ±2.2 †	67.7 ±1.0 †	65.7 ±0.0 †
Gemma 3 27B	41.9 ±0.7 †	36.1 ±0.8 †	43.3 ±2.4 †	39.1 ±3.2 †	68.7 ±1.2 †	67.2 ±0.8 †
Gemma 4 26B A4B	58.2 ±0.3 †	58.2 ±0.3 †	61.5 ±0.4 †	61.5 ±0.3 †	63.9 ±0.8 †	63.7 ±0.8 †
Gemma 4 31B	62.4 ±0.3	61.8 ±0.2	63.6 ±0.4	63.0 ±0.5	73.9 ±0.3	72.6 ±0.4
RoMistral 7B	35.0 ±14.0	35.7 ±13.7	44.2 ±1.8 †	46.6 ±1.5 †	73.0 ±0.6	70.4 ±0.5 †

Appendix

Table 18: Results on the gold dataset across zero-shot (ZS), chain-of-thought (CoT), and fine-tuning (FT) settings. Bold indicates the best performance. † indicates statistically significant difference compared to Gemma 4 31B or its fine-tuned variant in the FT setting (

p<0.05

; for

N>1

, Welch’s

t

-test; for

N=1

, paired McNemar’s test for accuracy and a paired bootstrap for F1).

Model	Random			Phrase			Dependency			LLM
Model	ZS	CoT	FT	ZS	CoT	FT	ZS	CoT	FT	ZS	CoT	FT
RO $\rightarrow$ EN
Llama 3.1 8B	44.3 ±16.5 †	44.0 ±4.0 †	75.6 ±1.1	45.4 ±15.9	49.8 ±2.1 †	75.0 ±0.8	47.5 ±13.2	52.3 ±2.0	77.0 ±0.2	45.1 ±15.0	49.6 ±2.9 †	74.5 ±0.8
Gemma 3 27B	30.3 ±0.1 †	31.0 ±0.9 †	74.0 ±0.4	38.6 ±0.2 †	41.3 ±1.5 †	74.4 ±0.8	40.8 ±0.6 †	44.8 ±2.5 †	75.9 ±1.0	33.4 ±0.4 †	40.0 ±3.9 †	75.3 ±1.2
Gemma 4 26B A4B	58.4 ±0.2 †	54.1 ±0.3 †	67.7 ±1.4 †	63.8 ±0.1 †	63.5 ±0.1 †	69.5 ±0.7	66.9 ±0.1	67.0 ±0.2 †	71.1 ±0.5	66.7 ±0.3	67.4 ±0.2	69.1 ±0.2
Gemma 4 31B	64.3 ±0.2 †	58.3 ±0.2 †	76.4 ±0.4 †	67.8 ±0.0 †	66.2 ±0.3 †	77.0 ±0.2 †	69.5 ±0.1	68.4 ±0.2 †	78.5 ±0.1	69.9 ±0.2	68.9 ±0.1	77.7 ±0.3

Appendix

Table 19: F1-scores breakdown for each model and code-switching generation method, in the zero-shot (ZS), chain-of-thought (CoT), and fine-tuning (FT) settings, for CS-SeLeRoSa test set. Bold indicates the best performance for each method and language pair. † indicates statistically significant degradation compared to the monolingual counterpart (original Romanian input with the same model for ZS/CoT; the Romanian-only fine-tuned counterpart for FT; sentence-level paired bootstrap,

p<0.05

Model	Random			Phrase			Dependency			LLM
Model	ZS	CoT	FT	ZS	CoT	FT	ZS	CoT	FT	ZS	CoT	FT
RO $\rightarrow$ EN
Llama 3.1 8B	50.0 ±16.7	31.7 ±16.1	73.3 ±0.0	50.3 ±13.1	48.7 ±7.5	68.6 ±1.8	44.6 ±18.4	46.9 ±2.6	63.6 ±2.2	50.7 ±11.1	46.7 ±5.3	61.7 ±3.6
Gemma 3 27B	28.9 ±7.7	33.3 ±0.0	73.3 ±0.0	33.2 ±0.0	38.3 ±1.7	62.8 ±3.5	33.6 ±1.7	35.0 ±4.1	69.2 ±3.9	33.6 ±0.0	38.4 ±6.8	61.6 ±2.9
Gemma 4 26B A4B	57.8 ±13.5	50.0 ±0.0	65.6 ±13.5	64.0 ±0.0	61.3 ±6.1	70.3 ±2.1	54.7 ±2.9	54.2 ±2.7	58.8 ±3.5	65.2 ±1.3	65.1 ±2.5	63.7 ±1.3
Gemma 4 31B	50.0 ±0.0	50.0 ±0.0	73.3 ±0.0	70.3 ±0.0	67.9 ±2.1	72.5 ±2.2	59.7 ±1.2	60.4 ±1.2	72.3 ±2.1	64.7 ±1.2	66.1 ±2.1	72.5 ±3.5

Appendix

Table 20: F1-scores breakdown for each model and code-switching generation method, in the zero-shot (ZS), chain-of-thought (CoT), and fine-tuning (FT) settings, for the CS gold standard test set. Bold indicates the best performance for each method and language pair.

Model	Method	F1 (%)	$\Delta$ (%)
Llama 3.1 8B	None	46.3	-
	Random	43.3	-3.0
	Phrase	44.7	-1.6
	Dependency	46.4	+0.0
	LLM	43.9	-2.4
Gemma 3 27B	None	49.9	-

Appendix

Table 21: F1-scores and performance degradation (

\Delta

) relative to the monolingual baseline (None) across code-switching generation methods. Bold indicates the lowest degradation (i.e., best performance) among code-switching methods for each model, while underline indicates the highest degradation (i.e., worst performance).

Model	Method	FPR (%)	FNR (%)
Llama 3.1 8B	None	47.2	47.3
	Random	57.4	34.1
	Phrase	52.4	39.3
	Dependency	49.7	40.3
	LLM	52.5	40.2
Gemma 3 27B	None	76.3	4.2

Appendix

Table 22: Breakdown of the False Positive Rate (FPR) and False Negative Rate (FNR) for each model across code-switching generation methods. The None method indicates the original monolingual baseline. Unparseable model outputs are counted as classification errors.

Model	Language	FPR (%)	FNR (%)
Llama 3.1 8B	EN	51.6	37.9
	DE	56.0	38.9
	FR	51.3	42.0
	IT	53.2	35.3
	RO	47.2	47.3
Gemma 3 27B	EN	93.2	1.2

Appendix

Table 23: Breakdown of False Positive Rate (FPR) and False Negative Rate (FNR) for each model across different target languages. RO indicates the original monolingual Romanian baseline. Unparseable model outputs are counted as classification errors.

Model	Setting	FPR (%)	FNR (%)
Llama 3.1 8B	Zero-Shot	63.8	49.6
	CoT	74.9	19.1
	Fine-Tuning	20.1	30.2
Gemma 3 27B	Zero-Shot	92.0	1.2
	CoT	87.0	2.6
	Fine-Tuning	26.1	21.5

Appendix

Table 24: Comparison of False Positive Rate (FPR) and False Negative Rate (FNR) across zero-shot (ZS), chain-of-thought (CoT), and fine-tuning (FT) settings. Unparseable model outputs are counted as classification errors.

Zero-Shot
Metric	Seed 1	Seed 2	Seed 3
Predicted Positive	1,343	1,325	1,170
Predicted Negative	721	737	780
Unparseable	8	10	122
Chain-of-Thought
Predicted Positive	1,665	1,549	1,613

Appendix

Table 25: Prediction counts of RoLlama 3.1 8B on the monolingual baseline test split, across each seed.

ID	Count	Topic Label	Share (%)	Top c-TF-IDF Keywords
Reasoning Traces
0	319	Interpreting Satire and Irony	96.1	cuvânt, afirma t , ie, exprima, sentiment, satiric, spune
1	13	Romanian Satire Explanations	3.9	românesc, român, satiric, cuvânt, sarcasm, comentariu
Input Sentences
$-1$	131	People Leaving and Changing Positions (Outliers)	39.5	pleca, rămâne, veni, începe, urmare, întâmpla
0	71	Reported Problems and Conditions	21.4	totuşi, pericol, fapt, explica, spune, convinşi

Appendix

Table 26: Topics found by BERTopic for reasoning traces and input sentences across the 332 false positive errors on the RoLlama 3.1 8B, seed 1, when evaluated on the monolingual SeLeRoSa test set.

Model	ZS (%)	CoT (%)	FT (%)
Llama 3.1 8B	24.00	7.91	0.13
Gemma 3 27B	0.00	0.00	0.02
Gemma 4 26B A4B	0.12	0.29	0.15
Gemma 4 31B	0.07	0.04	0.00
RoMistral 7B	24.52	10.97	0.00
RoLlama 3.1 8B	2.02	0.88	0.02

Appendix

Table 27: Rate of unparseable model outputs on the CS-SeLeRoSa test set.

Model	ZS (%)	CoT (%)	FT (%)
Llama 3.1 8B	23.10	6.56	0.00
Gemma 3 27B	0.00	0.00	0.00
Gemma 4 26B A4B	0.00	0.43	0.34
Gemma 4 31B	0.00	0.00	0.00
RoMistral 7B	24.72	11.25	0.00
RoLlama 3.1 8B	2.81	1.36	0.09

Appendix

Table 28: Rate of unparseable model outputs on the CS-SeLeRoSa-Gold test set.

Code-Switching

Code-switching studies how people combine languages in speech and text and how AI systems can recognize, understand, and generate these mixtures.

See all 22 papers

Feb 25, 2026

Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text

Sarcasm detection in multilingual and code-mixed environments remains a challenging task for natural language processing models due to structural variations, informal expressions, and low-resource linguistic availability. This study compares four large language models, Llama 3.1, Mistral, Gemma 3, and Phi-4, with a fine-tuned DistilBERT model for sarcasm detection in code-mixed Hinglish text. The results indicate that the smaller, sequentially fine-tuned DistilBERT model achieved the highest overall accuracy of 84%, outperforming all of the LLMs in zero and few-shot set ups, using minimal LLM generated code-mixed data used for fine-tuning. These findings indicate that domain-adaptive fine-tuning of smaller transformer based models may significantly improve sarcasm detection over general LLM inference, in low-resource and data scarce settings.

Bitan Majumder, Anirban Sen

Pondicherry University · Ashoka University

Sarcasm Detection Small Language Models LLM Evaluation Domain Adaptation

Ask PDF

May 15, 2026

Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood. We introduce PyLang, a minimal imperative language absent from all pretraining corpora, and evaluate frontier models zero-shot and fine-tuned Qwen3 (4B, 8B, 32B) on 352 problems. We find that fine-tuning quickly teaches syntax but fails to transfer semantic competence: Python outperforms PyLang by up to 19% across all configurations, and no intervention (multi-task learning, preference tuning, code infilling, or latent-space objectives) closes the gap. An LLM judge reveals that frontier models select an identical algorithm to Python 80% of the time, yet cannot translate it into a working PyLang implementation., and CKA analysis confirms that fine-tuned models converge to nearly identical internal representations across languages (CKA > 0.97) while diverging at the output stage. We term this the implementation fidelity gap: models possess language-agnostic algorithmic understanding but cannot express it in an unfamiliar language. Our findings highlight the need for training methods that decouple reasoning from language-specific realization.

Vinayshekhar Bannihatti Kumar, Disha Makhija, Manoj Ghuhan Arivazhagan +1

AWS AI Labs

LLM Fine-Tuning Code Language Models LLM Code Generation

Ask PDF

Jun 30, 2026

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

Relation extraction (RE) for low-resource languages is typically constrained by the lack of annotated corpora. We investigate the feasibility of cross-lingual RE for Romanian by combining automatic dataset translation with large language model (LLM) inference. We translate the SemEval-2010 Task 8 benchmark from English to Romanian using an LLM-based translation pipeline and evaluate Gemma 4 31B under zero-shot, few-shot, and QLoRA fine-tuned configurations, against four encoder baselines spanning 125M to 560M parameters: XLM- RoBERTa (base and large), Romanian BERT, and RoBERT- large. We assess two task formulations: relation classification with marked entities and end-to-end extraction. Our results show that Romanian incurs a 3 to 5 percentage point (pp) drop relative to English in prompt-only settings, that few-shot prompting provides marginal gains over zero-shot, and that QLoRA fine-tuning improves macro F1-Score by more than 22 percentage points in both languages while reducing the cross-lingual gap from 3.3 to 1.4pp. The encoder baselines come within 1-4pp of QLoRA Gemma on Romanian despite being 50-250 times smaller, with monolingual Romanian BERT at 125M parameters matching multilingual XLM-R at 278M. The case for using a 31B model for single-task RE on Romanian is therefore weak in deployment scenarios where compute matters. We release the translated dataset, evaluation code, and trained models.

Dragos-Mitrut Vasile, Elena-Simona Apostol, Stefan-Adrian Toma +2

National University of Science and Technology POLITEHNICA Bucharest, Splaiul Independent,ei 313, Bucures,ti 060042, Romania · Military Technical Academy ‘Ferdinand I’, Bulevardul George Cos,buc 39-49, 050141, Bucharest, Romania · Freie Universität Berlin, Arnimallee 14, Berlin, 14195, Germany +2

Relation Extraction LLM Fine-Tuning LLM Information Extraction Low-Resource Language Processing Cross-Lingual Transfer

Ask PDF

Abstract

Code-Switching

Explore similar work

Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text

Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian