Automatic identification of code-switched (CS) utterances remains a challenge for language identification (LID) systems, causing such texts to be underrepresented in the training data of Large Language Models. In this paper, we revisit MaskLID, a state-of-the art approach for CS identification, which requires no training and detects arbitrary language combinations. We make three main contributions: (a) we reveal, and address, a major issue of MaskLID: its overreliance on word-level language association scores; (b) we reformulate the underlying optimization algorithm as an Integer Linear Program, enabling us to experiment with a large set of clear and interpretable constraints; (c) each of these improvements vastly improves the baseline system, as we illustrate in experiments involving 10~diverse languages, where we observe a strong boost in performance on CS benchmarks. We release our code and data for reproducibility.
Figures & tables
tur-eng
eus-spa
hin-eng
npi-eng
tur-deu
spa-eng
ind-eng
fra-wol
# sent in dev / test
100 / 256
100 / 965
100 / 1928
100 / 6959
100 / 1143
0 / 8269
0 / 627
0 / 4386
mean # char/sent
84
79
72
56
75
57
103
73
mean len L1
57.7[ ± 29.4]
-
47.0[ ± 32.0]
36.4[ ± 18.4]
50.6[ ± 27.9]
41.1[ ± 20.7]
64.7[ ± 36.4]
-
mean len L2
19.0[ ± 11.9]
-
13.1[ ± 11.4]
12.4[ ± 8.6]
21.4[ ± 14.3]
11.2[ ± 8.3]
24.0[ ± 17.0]
-
mean # switches
2.8
-
2.9
2.8
1.9
2.9
2.8
-
type
sm
conv+sm
sm
sm
conv
sm
sm
conv
Table 1: Statistics of the CS datasets used the experiments. The uncertainties reported are standard deviations. The eus-spa and fra-wol datasets do not contain the word-level labels necessary to compute relevant CS-specific statistics. sm=social media corpus, conv=conversational corpus.
Figure 1: Top: GlotLID distribution of probabilities assigned to the gold label on a subset of FLORES sentences (left) and words (right). Mean score for a sentence: 0.98. Mean score for a word: 0.31. Bottom: Same figure for LiteLID -v2. Mean score for a sentence: 0.96. Mean score for a word: 0.50.
Figure 2: F1 as a function of the input length (in chars).
CS
mono
all
Model
config
EM
F1
EM
F1
EM
F1
GlotLID
k=1
0.0
0.55
0.98
0.98
0.49
0.76
GlotLID
k=2
0.22
0.56
0.0
0.66
0.11
0.61
GlotLID
p>0.1
0.09
0.61
0.94
0.97
0.51
0.79
LiteLID -v2
k=1
0.0
0.63
0.99
0.99
0.5
0.81
LiteLID -v2
k=2
0.39
0.68
0.0
0.67
0.19
0.67
Table 2: Exact match and F1 on the development data. (*) marks the best configuration.
CS
mono
all
Model
config
EM
F1
EM
F1
EM
F1
L-v2 + MaskLID
τ =5
0.39
0.77
0.96
0.98
0.68
0.88
L-v2 + MaskLID
τ =10
0.35
0.76
0.98
0.99
0.67
0.87
L-v2 + ILP4LID
AVG
0.60
0.83
0.91
0.97
0.76
0.90
L-v2 + ILP4LID
CS
0.68
0.83
0.06
0.69
0.37
0.76
L-v2 + ILP4LID
MONO
0.47
0.81
0.98
0.99
0.73
0.90
Table 3: Exact Match and F1 on the development data for various ILP configurations, compared to the MaskLID baseline of Section 3 .
CS
mono
all
Model
config
EM
F1
Pr
Re
EM
F1
Pr
Re
EM
F1
Pr
Re
GlotLID
k =1
0.0
0.58
0.87
0.44
0.97
0.97
0.97
0.97
0.49
0.78
0.92
0.71
GlotLID
k =2
0.20
0.56
0.56
0.56
0.0
0.66
0.49
0.98
0.10
0.61
0.53
0.77
GlotLID
p>0.1
0.06
0.62
0.85
0.49
0.95
0.97
0.97
0.98
0.5
0.8
0.91
0.74
LiteLID-v2
k =1
0.0
0.63
0.95
0.47
0.99
0.99
0.99
0.99
0.50
0.81
0.97
0.73
LiteLID-v2
k =2
0.36
0.67
0.67
0.67
0.0
0.67
0.50
1.0
0.18
0.67
0.58
0.83
Table 4: Results on the monolingual and CS test sets. k =1,2 when the LID always predicts k languages; p>0.1 when it predicts any language having a probability >0.1. The three highest Exact Match values per column are in bold.
ILP4LID (CS)
MaskLID ( τ=5 )
pair
Pr
Re
F1
Un
Pr
Re
F1
Un
HIN-ENG
0.87
0.77
0.80
1.3
0.90
0.68
0.77
21.5
IND-ENG
0.94
0.90
0.92
0.4
0.92
0.82
0.87
8.3
NPI-ENG
0.88
0.70
0.77
3.3
0.94
0.57
0.71
32.5
SPA-ENG
0.93
0.83
0.88
0.9
0.95
0.75
0.84
15.7
TUR-DEU
0.98
0.95
0.96
0.5
0.99
0.84
0.91
13.5
Table 5: Word-level evaluation results on code-switched test data. "Un" is the percentage of Unassigned words.
Figure 3: Runtime analysis showing a linear dependency between the number of language labels and the execution time on a desktop machine for 80 examples (in seconds). CS configuration (left) - AVG configuration (right). Curves correspond to buckets of different average lengths, ranging from ~25 to ~100 characters.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm 1 MaskLID: Iterative Masking
Figure 4: Percentage of cases when a given language label appears in top-10 languages for GlotLID (left) and LiteLID (right) in the monolingual development set. First row is English, second is Hindi.
v
buckets
n-gram
data
v0
1e6
2-5
sents: GlotLID on 125 labels
v0.1
2e6
2-5
sents
v0.2
1e6
3-6
sents
v0.3
2e6
3-6
sents
v1
2e6
3-6
words
v1.1
1e6
2-5
sents+words
Appendix
Table 6: FastText training configurations explored in training LiteLID .
Figure 5: Code-switching LID as an ILP problem. The inputs are C(w) , the cost matrix, and A , which contains information about the presence of each language l in the top-M list of each word. Y is a feasible solution, with U the associated language vector. Y satisfies the constraints: at most one language for each word, no more than two languages ( ∑lul=2 ), a minimal length of 10 chars for each span. O(Y∗) is the value of the objective function computed for this assignment.
Inputs
cl,t
T×L (costs)
al,t
T×L (top-M languages)
Variables
yl,t
T×L (word assignments)
ul
L (sentence assignments)
All
(T+1)×L
Appendix
Table 7: Counts of variables and constraints in the core model.
Inputs
cl,t
T×L (costs)
al,t
T×L (top-M languages)
Variables
yk,l,t
K×T×L (word-level language and rank assignments)
uk,l
K×L (sentence-level language and rank assignments)
zl,t
T×L (word-level language assignment)
Appendix
Table 8: Counts of variables and constraints in the extended model.
CS
mono
all
config
EM
F1
EM
F1
EM
F1
0 : CS
0.68
0.83
0.06
0.69
0.37
0.76
1 : 0+ αk=[1,0.95]
0.68
0.83
0.09
0.70
0.39
0.77
1+ P=10
0.67
0.84
0.59
0.88
0.63
0.86
1+ P=25
0.57
0.83
0.88
0.96
0.73
0.90
1+ P=50
0.33
0.76
0.99
0.99
0.66
0.88
Appendix
Table 9: Exact match and F1 on the development data with various ILP configurations. "C3,C5,P" stands for C3 τ=5 , C5 M=10 , P=10 .
G + MaskLID
L-v2 + MaskLID
L-v2+ILP4LID AVG
(baseline)
(first improvement)
(second improvement)
EM
F1
Pr
Re
EM
F1
Pr
Re
EM
F1
Pr
Re
eus-spa
0.59
0.86
0.94
0.79
0.55
0.87
0.99
0.77
0.73
0.9
0.95
0.86
hin-eng
0.1
0.56
0.7
0.47
0.17
0.68
0.9
0.55
0.34
0.73
0.82
0.65
ind-eng
0.38
0.74
0.83
0.67
0.47
0.83
0.95
0.73
0.52
0.81
0.87
0.76
npi-eng
0.03
0.32
0.39
0.27
0.16
0.6
0.78
0.49
0.38
0.68
0.71
0.65
Appendix
Table 10: Detailed performances for monolingual and mixed language corpora. G stands for GlotLID, L for LiteLID -v2. Best performances for each code-switched language pair are in bold. Those for which the performance has been improved by a factor of two are underlined.
Language identification determines which language or languages appear in text or speech, including dialects, code-switching, and low-resource varieties.
Code-switched (CS) speech leaks through the monolingual language identification (LID) filters used to curate massive speech corpora, calling for CS-aware LID (CS-LID). We formulate utterance-level CS-LID as multi-label language-set prediction and propose a set generator that directly outputs the languages in an utterance, comparing it against atomic-pair and score-based classification baselines. Oracle Top-k is the strongest baseline, but thresholding fails because no single threshold separates CS from monolingual speech. Our set generator predicts the correct language count on unseen pairs without assuming the number of languages, but underperforms oracle Top-k in exact set accuracy. Our analysis identifies the key obstacles to robust CS-LID: oracle cardinality, threshold instability, language bias in CS training data, and the synthetic-to-real gap.
Shunsuke Mitsumori, Matthew Wiesner, Shigeo Morishima +1
Waseda University, Tokyo, Japan · Johns Hopkins University, Baltimore, USA · Carnegie Mellon University, Pittsburgh, USA
Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 language-script combinations. MixLID follows a two-stage training process: it first learns to distinguish individual languages using the language labels of monolingual examples, then continues training on mixed-language data. The second stage uses concatenated examples from existing code-switching data alongside augmented examples created through word substitution based on correspondences between English source words and loanwords. For monolingual identification, MixLID outperforms all baselines in both accuracy and macro F1 on four benchmarks within the language coverage shared with each baseline. For mixed-language identification, it outperforms the baselines in identifying the language of individual tokens on every evaluated language pair. MixLID achieves high performance across a broad range of languages in both monolingual and mixed-language settings. The model is publicly available at https://huggingface.co/nlpai-lab/MixLID.
Seongtae Hong, Youngjoon Jang, Seungyoon Lee +2
Department of Computer Science and Engineering, Korea University
We present CS-YODAS, a Creative Commons-licensed dataset of in-the-wild code-switched speech mined from multilingual YouTube data. Code-switching (CS), or the alternation between languages within an utterance or conversation, is common in multilingual settings but remains underrepresented in existing CS speech resources, which are typically small, domain-specific, or artificially constructed. Building on the YODAS corpus, we develop a scalable, human-in-the-loop pipeline for identifying and validating naturally occurring code-switching. The resulting dataset, which totals 313 hours and spans 7 matrix languages, provides diverse, real-world examples of spontaneous code-switched speech. We further analyze the distribution and characteristics of code-switching in the wild, examining language-pair frequencies and switching patterns, and report baseline results for spoken language identification. We hope that CS-YODAS will encourage broader and more comprehensive research on code-switched speech. Dataset link: https://huggingface.co/datasets/byan/cs-yodas.
Brian Yan, Qingzheng Wang, Matthew Wiesner +9
Carnegie Mellon University · Johns Hopkins University · University of Texas at Austin +4