TurEngMix: A Text Corpus and Benchmark for Turkish-English Code-Mixed Language Identification and Named Entity Recognition

Ilayda Dogan; Phuong-Anh Nguyen-Le; Julia Mendelsohn

cs.CLSep 7, 2026

TurEngMix: A Text Corpus and Benchmark for Turkish-English Code-Mixed Language Identification and Named Entity Recognition

Authors: Ilayda Dogan, Phuong-Anh Nguyen-Le, Julia Mendelsohn

Organizations: Computer Science University of Maryland

Abstract

Natural language processing systems underperform on code-mixed text, particularly for low-resource language pairs. Turkish-English poses a further challenge: it lets English stems combine with Turkish suffixes to form single mixed-language tokens. We introduce TurEngMix, a corpus of 5.5K noisy, naturally occurring social media posts (486,974 tokens) rich in Turkish-English code-mixing. From this corpus, we construct a new Turkish-English benchmark for code-mixed language identification (LID) and named entity recognition (NER), comprising 15K expert-annotated tokens. Evaluating both decoder LLM and fine-tuned encoder baselines, we find that monolingual Turkish and English tokens are labeled reliably, but all models have high error rates on mixed-language tokens for both LID and NER. For morphologically integrated tokens, NER error rates were 5.2x and 6.3x higher for GPT-4o and Qwen, respectively. This highlights how morphological integration remains a challenge. We release the corpus, annotations, and code to support future computational and sociolinguistic research on Turkish-English code-mixing.

Figures & tables

Token	Gloss
influencerlarımız	influencer - lar- ımız influencer- pl - 1 pl.poss ‘our influencers’
frameworklerinin	framework - ler- i- nin framework- pl - 3 pl.poss - gen ‘of their frameworks’
tweetlememesi	tweet - le- me- me- si tweet- vblz - neg - nmlz - 3 sg.poss ‘the fact that they do not tweet’

Table 1: Mixed-language tokens with English stems and Turkish suffixes.

Statistic	Value
Unique posts	5,549
Total tokens	486,974
Post length range	3–500 words
Mean post length	87.75 words
Median post length	57 words

Table 2: Summary statistics of the TurEngMix corpus.

Figure 2: Word count distribution in the final dataset. Post word count was majority within 100 words, with longer posts scaling to 500.

Figure 3: Ten most-frequent topic tags in the final dataset are primarily tech and entertainment.

Code-mixing type	Original post	English translation
Isolated English insertions	her açtığımda belki gülhan şen kaydolmuştur diye search edip havamı aldığım trendy oluşum.	The trendy platform that, every time I open it, I search thinking ‘maybe Gülhan Şen has signed up’ and end up being disappointed.
Mixed-language phrase	love is an excuse to get hurt diye çok mantıklı bir açıklamasını yapmış birileri	Someone gave a very reasonable explanation that “love is an excuse to get hurt.”
Mixed-language token	Bu link’teki video mutlaka izlenmeli bence.	The video at this link should definitely be watched.

Table 3: Different types of code-mixing found in the dataset, including isolated English insertions, longer mixed language phrases, and mixed-language tokens.

Figure 4: An example annotation sentence from the golden annotated dataset. Translation: Netflix’s biggest selling point is the ease of use which the Westerners refer to as ’convenience.’

LID Label	Count
TR	11,723
EN	1,992
NE	1042
Mixed	189
Ambiguous	62
Other Language	4

Table 4: Distribution of LID labels. Note that NE tokens included additional morphologically mixed examples.

NER Label	Count
Non-Entity (O)	13,971
Product	359
Title	246
Organization	129
Person	119
Location	88

Table 5: Distribution of named entity labels.

Split	Posts
Embedded Mixed Token	165
Isolated English Token	154
Embedded English Phrase	148

Table 6: Distribution of code-mixing patterns in the benchmark.

Split	Posts	Sentences	Tokens
Train	200	254	12,466
Validation	25	26	1,173
Test	25	41	1,373
Total	250	321	15,012

Table 7: Dataset statistics across train, validation, and test splits.

Model	Setting	TR	EN	MIXED	NE	OTHER	AMBIGUOUS
GPT-4o	zero-shot	0.93	0.89	0.42	0.62	0.19	0.00
GPT-4o	three-shot	0.98	0.90	0.45	0.67	0.13	0.00
Qwen3-8B	zero-shot	0.98	0.86	0.33	0.60	0.07	0.12
Qwen3-8B	three-shot	0.98	0.86	0.39	0.54	0.10	0.00
XLM-RoBERTa	fine-tuned	0.98	0.87	0.00	0.68	0.00	0.00
BERTurk	fine-tuned	0.97	0.84	0.13	0.64	0.00	0.00

Table 8: Per-class F1 scores for LID. Decoder LLMs were tested on the full dataset. Encoder models were fine-tuned on the train/dev and evaluated on the test set.

Model	Setting	Macro F1
GPT-4o	Zero-shot	0.5170
GPT-4o	Three-shot	0.5238
Qwen3-8B	Zero-shot	0.4920
Qwen3-8B	Three-shot	0.4788
Baseline	–	0.1414
XLM-RoBERTa	Fine-tuned	0.4358 ± 0.0137

Table 9: Macro F1 scores for LID task. Decoder LLMs were tested on the full dataset. Encoders were fine-tuned on the train/dev and evaluated on the test set.

Model	Setting	Macro F1
GPT-4o	Zero-shot	0.5669
GPT-4o	Three-shot	0.6140
Qwen3-8B	Zero-shot	0.3993
Qwen3-8B	Three-shot	0.4162
Baseline	–	0.0502
XLM-RoBERTa	Fine-tuned	0.0772 $\pm$ 0.0246

Table 10: BIO-level NER macro F1.

Model	Setting	Non-NE	NE
GPT-4o	Zero-shot	0.98	0.78
GPT-4o	Three-shot	0.98	0.80
Qwen3-8B	Zero-shot	0.97	0.70
Qwen3-8B	Three-shot	0.98	0.73
XLM-RoBERTa	Fine-tuned	0.97	0.67
BERTurk	Fine-tuned	0.96	0.47

Table 11: F1 scores for binary NER. BIO labels are collapsed into two categories: NE and Non-NE .

Model	Other Tokens	Mixed Tokens
GPT-4o	3.8%	19.7%
Qwen3-8B	5.3%	33.5%

Table 12: Comparison of NER error rates between morphologically integrated English borrowings and other evaluated tokens.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

LID Label	Cohen’s $\kappa$
TR	0.98
EN	0.98
NE	0.94
MIXED	0.75

Appendix

Table A1: Per-class inter-annotator agreement for language identification (LID).

NER Label	Cohen’s $\kappa$
B-LOC	1.00
I-LOC	1.00
B-PER	1.00
I-PER	1.00
B-PROD	1.00
B-TITLE	1.00

Appendix

Table A2: Per-class inter-annotator agreement for named entity recognition (NER). The low

\kappa

score for B-OTHER is likely due to the small number of instances in this category.

Model	O	ORG	PER	LOC	PROD	GROUP	TITLE	EVENT	TIME	OTHER
GPT-4o Zero-shot	0.98	0.48	0.79	0.83	0.53	0.35	0.69	0.46	0.20	0.29
GPT-4o Few-shot	0.98	0.57	0.83	0.84	0.67	0.42	0.72	0.43	0.09	0.34
Qwen3-8B Zero-shot	0.97	0.43	0.55	0.65	0.46	0.35	0.66	0.00	0.09	0.43
Qwen3-8B Few-shot	0.98	0.47	0.63	0.64	0.52	0.28	0.53	0.00	0.07	0.34

Appendix

Table A3: Per-class F1 scores for decoder LLMs on NER task. O denotes non-entity tokens.

Model	O	LOC	ORG	PER	PROD	TITLE	OTHER
Baseline	1.00	0.00	0.00	0.00	0.00	0.00	0.00
XLM-RoBERTa	0.97	0.00	0.00	0.00	0.28	0.00	0.00
BERTurk	0.97	0.00	0.091	0.44	0.27	0.14	0.00
TurkishBERTweet	0.96	0.00	0.00	0.00	0.27	0.06	0.00

Appendix

Table A4: Per-class F1 scores for named entity recognition using encoder models. O denotes non-entity tokens.

Token	Gold Label	Model Prediction	Error Explanation
influencerlar	MIXED	TR	English-origin stem influencer with Turkish plural/possessive morphology. The token requires identifying both the English lexical root and Turkish morphological integration.
google’a	B-PROD	B-ORG	In context, Google refers to the search product/service rather than the organization.
sony’nin	NE	MIXED	The token contains a Turkish possessive suffix attached to a named entity. The model incorrectly prioritized morphological mixing over named entity identification.

Appendix

Table A5: Representative error cases involving morphological integration and named entity ambiguity.

Code-Switching

Code-switching studies how people combine languages in speech and text and how AI systems can recognize, understand, and generate these mixtures.

See all 22 papers

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing
Sep 10, 2026Pruthwik Mishra, Rudra Trivedi, Avi Patel +2Language Identification
Sardar Vallabhbhai National Institute of Technology, Surat, India · TCS Research, Hyderabad, India
MixLID: Token-Level Language Identification in Mixed-Language Text
Oct 12, 2026Seongtae Hong, Youngjoon Jang, Seungyoon Lee +2Language Identification
Department of Computer Science and Engineering, Korea University
PERCEPT: A Corpus for POS Tagging and Analysis of Persian-English Code-Mixing
Aug 10, 2026Ghazal Kalhor, Zahra Jafari, Amirarsalan Shahbazi +1Social Media Analysis Code-Switching
School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · School of Engineering Science, College of Engineering, University of Tehran, Tehran, Iran · Tehran Institute for Advanced Studies, Khatam University, Tehran, Iran

Sep 10, 2026J/K move · Enter open · S save

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

Pruthwik Mishra, Rudra Trivedi, Avi Patel, Ashok Urlana+1

Sardar Vallabhbhai National Institute of Technology, Surat, India · TCS Research, Hyderabad, India

Language Identification

Open paper AskPDF arXiv

Language identification in code-mixed text, largely observed in social media, is highly essential when users frequently switch between multiple languages within a single utterance. Accurately identifying the languages of code-mixed tokens becomes an urgent necessity. Traditional language identification models, designed for monolingual text, are not well suited for token-level language identification in code-mixed settings. We formulate the task as a sequence labeling problem and fine-tune contextual transformer-based models MuRIL and XLM-RoBERTa best suited for Indian languages. We evaluate these systems on three different data configurations (Hindi, Gujarati, and Bengali) to predict language labels for individual tokens. We release a benchmark for language identification in code-mixed tokens with manually annotated test sets. We propose two approaches of code-mixed generation using parallel sentences of three languages. The trained models demonstrate the effectiveness of contextual embeddings for token-level language identification in multilingual social media text. For reproducibility and to facilitate future research, we publicly release our fine-tuned models.

Abstract

Code-Switching

Explore similar work

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

MixLID: Token-Level Language Identification in Mixed-Language Text

PERCEPT: A Corpus for POS Tagging and Analysis of Persian-English Code-Mixing