The Romansh language has several regional varieties, called idioms, which sometimes have limited mutual intelligibility. Despite this linguistic diversity, there has been a lack of documented efforts to build a language identification (LID) system that can distinguish between these idioms. Since Romansh LID should also be able to recognize Rumantsch Grischun, a supra-regional variety that combines elements of several idioms, this makes for a novel and interesting classification problem. In this paper, we present a LID system for Romansh idioms based on an SVM approach. We evaluate our model on a newly curated benchmark across two domains and find that it reaches an average in-domain accuracy of 97%, enabling applications such as idiom-aware spell checking or machine translation. Our classifier is publicly available.
Figures & tables
Classifier
Accuracy
Macro F1
Weighted F1
Macro Recall
Logistic Regression (LR)
79.5
76.9
79.1
74.6
Linear SVM
78.1
75.4
77.9
73.9
SGD (Linear SVM)
76.3
72.4
75.2
68.0
Naive Bayes (counts)
75.1
72.1
74.6
70.6
Naive Bayes (TF)
75.1
72.1
74.6
70.6
Naive Bayes (TF-IDF)
73.7
72.0
73.9
71.7
Table 1: Results from the preliminary experiments with different baseline classifiers, where classifiers are ordered by macro F1 score.
Figure 1: Row-normalized confusion matrices for all test sets. The model achieves near-perfect classification on the balanced in-domain set ( test-b ), while confusion increases on out-of-domain data ( test-d ).
Test Set
Domain
Balance
Acc.
F1
test-a
in-domain
unbal.
96.8
94.7
test-b
in-domain
bal.
98.1
98.1
test-c
in-domain
bal.
96.2
80.5
test-d
out-domain
unbal.
90.7
69.1
Table 2: Accuracy and macro F1 scores per test set. The highest scores are achieved on the balanced (bal.) vs. unbalanced (unbal.), in-domain set test-b .
Test Set
Accuracy
Macro F1
Masked
Unmasked
Masked
Unmasked
test-a
96.8
96.9
94.7
94.5
test-b
98.1
98.8
98.1
98.8
test-c
96.2
96.2
80.5
80.4
test-d
90.7
90.5
69.1
69.0
Table 3: Comparison of masked vs. unmasked training. No major effect can be observed.
Idiom
1st
Type
2nd
Type
3rd
Type
Sursilvan
ei
char
scha
char
iu
char
Sutsilvan
àn
char
ù
char
eing
char
Surmiran
eir
char
dall
char
ous
char
Puter
ron
char
aunt
char
ọ
char
Vallader
ạ
char
ọn
char
ị
char
RG
␣
char
vegnis
word
ì
char
Table 4: Top 3 most discriminative features per idiom. “char” denotes character n -gram and “word” denotes word unigram. Character n -grams emerge as the most discriminative features across all varieties, with empty space ␣ being the most informative one for RG.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Source and URL to Dataset (if available)
Idiom
Total # of Samples
Total # of Tokens
Avg. # of Tokens/Sample
Sursilvan
34,724
92,182
2.65
Pledari Grond https://www.pledarigrond.ch
Sutsilvan
32,175
307,799
9.57
Surmiran
43,954
301,390
6.86
Puter
59,852
963,940
16.11
Vallader
72,470
1,141,688
15.75
RG
172,616
785,831
4.55
Appendix
Table 5: Summary of the number of samples extracted from each data source for each idiom, along with the number of whitespace tokens across all collected samples per data source and the average number of tokens per sample.
Set
Idiom
Total # of Samples
Total # of Tokens
Avg. # of Tokens/Sample
Sursilvan
55,574
3,663,641
65.92
Sutsilvan
44,188
1,342,065
30.37
Surmiran
55,543
1,457,845
26.25
train-set
Puter
74,355
2,353,323
31.65
Vallader
86,313
2,372,016
27.48
RG
171,199
1,190,808
6.96
Appendix
Table 6: Detailed statistics for all data splits by idiom, including number of samples, tokens, and average tokens per sample. The training set exhibits class imbalance, with RG containing the most samples but the shortest average length due to dictionary entries from Pledari Grond. test-c and test-d lack RG samples by construction.
Language identification determines which language or languages appear in text or speech, including dialects, code-switching, and low-resource varieties.
Lemmatization -- the task of mapping an inflected word form to its dictionary form -- is a crucial component of many NLP applications. In this paper, we present RUMLEM, a lemmatizer that covers the five main varieties of Romansh as well as the supra-regional standard variety Rumantsch Grischun. It is based on comprehensive, community-driven morphological databases for Romansh, enabling RUMLEM to cover 77-84% of the words in a typical Romansh text. Since there is a dedicated database for each Romansh variety, an additional application of RUMLEM is variety-aware language classification. Evaluation on 30'000 Romansh texts of varying lengths shows that RUMLEM correctly identifies the variety in 95% of cases. In addition, a proof of concept demonstrates the feasibility of Romansh vs. non-Romansh language classification based on the lemmatizer.
Dominic P. Fischer, Zachary Hopton, Jannis Vamvas
Department of Computational Linguistics, University of Zurich
We present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic noise. Our dataset has two parts: First, we collected sentences shared across mutually intelligible language pairs (Czech/Slovak, Spanish/Catalan, Portuguese/Galician, Danish/Norwegian). The second part tests for orthography noise: we transliterate text across multiple scripts, remove diacritics, simulate homoglyph attacks, and use Internet slang. We evaluate four widely used language identification systems on CHALIS and demonstrate that all struggle substantially in these scenarios, especially on lower-resource languages within cousin pairs and on transliterated input. The resource is publicly available at https://huggingface.co/datasets/michal-tichy/CHALIS.
Michal Tichý, Jindřich Libovický
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, V Holešovičkách 747/2, 180 00 Praha, Czechia
Automatic identification of code-switched (CS) utterances remains a challenge for language identification (LID) systems, causing such texts to be underrepresented in the training data of Large Language Models. In this paper, we revisit MaskLID, a state-of-the art approach for CS identification, which requires no training and detects arbitrary language combinations. We make three main contributions: (a) we reveal, and address, a major issue of MaskLID: its overreliance on word-level language association scores; (b) we reformulate the underlying optimization algorithm as an Integer Linear Program, enabling us to experiment with a large set of clear and interpretable constraints; (c) each of these improvements vastly improves the baseline system, as we illustrate in experiments involving 10~diverse languages, where we observe a strong boost in performance on CS benchmarks. We release our code and data for reproducibility.
Joanna Radoła, Josep Maria Crego, François Yvon
Sorbonne Université, CNRS, ISIR, Paris, France · SYSTRAN by ChapsVision, Paris, France