MixLID: Token-Level Language Identification in Mixed-Language Text

Seongtae Hong; Youngjoon Jang; Seungyoon Lee; Jungseob Lee; Heuiseok Lim

cs.CLOct 12, 2026

MixLID: Token-Level Language Identification in Mixed-Language Text

Authors: Seongtae Hong, Youngjoon Jang, Seungyoon Lee, Jungseob Lee, Heuiseok Lim

Organizations: Department of Computer Science and Engineering, Korea University

Abstract

Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 language-script combinations. MixLID follows a two-stage training process: it first learns to distinguish individual languages using the language labels of monolingual examples, then continues training on mixed-language data. The second stage uses concatenated examples from existing code-switching data alongside augmented examples created through word substitution based on correspondences between English source words and loanwords. For monolingual identification, MixLID outperforms all baselines in both accuracy and macro F1 on four benchmarks within the language coverage shared with each baseline. For mixed-language identification, it outperforms the baselines in identifying the language of individual tokens on every evaluated language pair. MixLID achieves high performance across a broad range of languages in both monolingual and mixed-language settings. The model is publicly available at https://huggingface.co/nlpai-lab/MixLID.

Figures & tables

Benchmark	Metric	OpenLID-v3		GlotLID-v3		NLLB-LID		fastText lid.176		CLD3		CommonLingua		CONLID		Polyglot Tagger
Benchmark	Metric	Baseline	MixLID	Baseline	MixLID	Baseline	MixLID	Baseline	MixLID	Baseline	MixLID	Baseline	MixLID	Baseline	MixLID	Baseline	MixLID
FLORES+	#L	110		110		106		70		74		96		110		70
	Acc	99.19	99.77	99.38	99.77	98.97	99.75	95.59	99.94	97.37	99.91	95.40	99.89	98.89	99.77	99.46	99.91
	Macro F1	99.42	99.76	99.57	99.76	99.15	99.75	95.47	99.94	97.31	99.91	96.18	99.89	99.32	99.76	99.58	99.92
WiLI-2018	#L	82		82		80		70		70		81		82		67
	Acc	97.58	98.43	97.72	98.41	97.52	98.37	95.88	98.91	95.76	98.59	97.05	98.41	97.64	98.41	98.58	98.84

Table 1: Sentence-level language identification performance (%) on the language–script intersection between MixLID and each baseline. #L denotes the intersection size. Bold indicates the higher score in each paired comparison.

Training	Dataset	Source	Augmentation	#L	#Samples
Mono	OpenLID-v3	Mono	—	111	123.4M
Mixed	ACS Corpus	CS	concatenation	10	509.8K
	SmolSent	Mono	Substitution	87	46.8K
	SmolDoc	Mono	Substitution	92	20.7K
	Belebele	Mono	Substitution	111	56.6K

Table 2: Training data composition and statistics.

Figure 1: Language-wise F1 scores for the 36 languages shared by all models and all four sentence-level benchmarks. Each cell shows the average score across these datasets.

	Pair	CLD3	MaskLID	Polyglot	MixLID
ACS	da–en	56.1 / 32.6	38.7 / 20.7	62.5 / 37.9	83.3 / 73.5
	de–en	56.4 / 33.9	35.9 / 22.9	78.2 / 62.0	98.0 / 96.9
	es–en	48.2 / 28.2	22.9 / 15.3	74.6 / 56.9	96.1 / 94.1
	fr–en	54.8 / 32.1	24.4 / 15.5	71.0 / 47.7	93.1 / 90.4
	id–en	31.7 / 18.4	19.3 / 12.1	38.0 / 24.2	92.1 / 89.1
	it–en	56.3 / 33.3	29.6 / 19.3	74.3 / 55.7	95.6 / 93.6

Table 3: Token-level LID results (Acc / Macro F1).

	Stage 1		Stage 2		Stage 1+2
Benchmark	Acc	Macro F1	Acc	Macro F1	Acc	Macro F1
FLORES+	99.67	99.64	97.39	97.21	99.77	99.76
WiLI-2018	98.52	98.62	94.05	94.74	98.43	98.58
UDHR	98.69	98.34	96.32	95.86	99.38	99.32
CommonLID	93.64	81.42	90.91	74.67	96.95	85.79
ACS	53.75	34.31	87.32	80.78	93.23	88.78

Table 4: Effect of training stages on LID performance at the sentence and token levels across six benchmarks.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Label	Language	Label	Language	Label	Language	Label
Afrikaans	afr_Latn	Hausa	hau_Latn	Marathi	mar_Deva	Spanish	spa_Latn
Amharic	amh_Ethi	Hebrew	heb_Hebr	Modern Greek	ell_Grek	Standard Estonian	ekk_Latn
Arabic	ara_Arab	Hindi	hin_Deva	Nepali	npi_Deva	Standard Latvian	lvs_Latn
Armenian	hye_Armn	Hungarian	hun_Latn	Nigerian Fulfulde	fuv_Latn	Standard Malay	zsm_Latn
Assamese	asm_Beng	Icelandic	isl_Latn	North Azerbaijani	azj_Latn	Sundanese	sun_Latn
Bambara	bam_Latn	Igbo	ibo_Latn	Northern Uzbek	uzn_Latn	Swahili	swh_Latn

Appendix

Table 5: Supported languages and their training labels.

Setting	Monolingual	Mixed
Optimizer	AdamW	AdamW
LR schedule	Linear	Linear
Maximum length	8,192	8,192
Learning rate	$5\times 10^{-5}$	$1\times 10^{-5}$
Epochs	1	1
Batch size per GPU	512	128

Appendix

Table 6: Training hyperparameters for both stages.

Type	Name	URL
Model	OpenLID-v3	https://huggingface.co/HPLT/OpenLID-v3
	GlotLID-v3	https://huggingface.co/cis-lmu/glotlid
	NLLB-LID	https://huggingface.co/facebook/fasttext-language-identification
	fastText lid.176	https://fasttext.cc/docs/en/language-identification.html
	CommonLingua	https://huggingface.co/PleIAs/CommonLingua
	CONLID	https://huggingface.co/epfl-nlp/ConLID

Appendix

Table 7: Sources of baseline models and evaluation benchmarks.

Language Identification

Language identification determines which language or languages appear in text or speech, including dialects, code-switching, and low-resource varieties.

See all 26 papers

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing
Sep 10, 2026Pruthwik Mishra, Rudra Trivedi, Avi Patel +2Language Identification
Sardar Vallabhbhai National Institute of Technology, Surat, India · TCS Research, Hyderabad, India
What Language is This? Ask Your Tokenizer
Feb 19, 2026Clara Meister, Ahmetcan Yavuz, Pietro Lesci +1Language Identification Text Classification
EPFL · ETH Zürich · University of Cambridge
Improving Language Identification for Code-Switched Utterances with Integer Linear Programming
Sep 4, 2026Joanna Radoła, Josep Maria Crego, François YvonLanguage Identification
Sorbonne Université, CNRS, ISIR, Paris, France · SYSTRAN by ChapsVision, Paris, France

Sep 10, 2026J/K move · Enter open · S save

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

Pruthwik Mishra, Rudra Trivedi, Avi Patel, Ashok Urlana+1

Sardar Vallabhbhai National Institute of Technology, Surat, India · TCS Research, Hyderabad, India

Language Identification

Open paper AskPDF arXiv

Language identification in code-mixed text, largely observed in social media, is highly essential when users frequently switch between multiple languages within a single utterance. Accurately identifying the languages of code-mixed tokens becomes an urgent necessity. Traditional language identification models, designed for monolingual text, are not well suited for token-level language identification in code-mixed settings. We formulate the task as a sequence labeling problem and fine-tune contextual transformer-based models MuRIL and XLM-RoBERTa best suited for Indian languages. We evaluate these systems on three different data configurations (Hindi, Gujarati, and Bengali) to predict language labels for individual tokens. We release a benchmark for language identification in code-mixed tokens with manually annotated test sets. We propose two approaches of code-mixed generation using parallel sentences of three languages. The trained models demonstrate the effectiveness of contextual embeddings for token-level language identification in multilingual social media text. For reproducibility and to facilitate future research, we publicly release our fine-tuned models.

Abstract

Language Identification

Explore similar work

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

What Language is This? Ask Your Tokenizer

Improving Language Identification for Code-Switched Utterances with Integer Linear Programming