The Romansh language has several regional varieties, called idioms, which sometimes have limited mutual intelligibility. Despite this linguistic diversity, there has been a lack of documented efforts to build a language identification (LID) system that can distinguish between these idioms. Since Romansh LID should also be able to recognize Rumantsch Grischun, a supra-regional variety that combines elements of several idioms, this makes for a novel and interesting classification problem. In this paper, we present a LID system for Romansh idioms based on an SVM approach. We evaluate our model on a newly curated benchmark across two domains and find that it reaches an average in-domain accuracy of 97%, enabling applications such as idiom-aware spell checking or machine translation. Our classifier is publicly available.
Figures & tables
Classifier
Accuracy
Macro F1
Weighted F1
Macro Recall
Logistic Regression (LR)
79.5
76.9
79.1
74.6
Linear SVM
78.1
75.4
77.9
73.9
SGD (Linear SVM)
76.3
72.4
75.2
68.0
Naive Bayes (counts)
75.1
72.1
74.6
70.6
Naive Bayes (TF)
75.1
72.1
74.6
70.6
Naive Bayes (TF-IDF)
73.7
72.0
73.9
71.7
Table 1: Results from the preliminary experiments with different baseline classifiers, where classifiers are ordered by macro F1 score.
Figure 1: Row-normalized confusion matrices for all test sets. The model achieves near-perfect classification on the balanced in-domain set ( test-b ), while confusion increases on out-of-domain data ( test-d ).
Test Set
Domain
Balance
Acc.
F1
test-a
in-domain
unbal.
96.8
94.7
test-b
in-domain
bal.
98.1
98.1
test-c
in-domain
bal.
96.2
80.5
test-d
out-domain
unbal.
90.7
69.1
Table 2: Accuracy and macro F1 scores per test set. The highest scores are achieved on the balanced (bal.) vs. unbalanced (unbal.), in-domain set test-b .
Test Set
Accuracy
Macro F1
Masked
Unmasked
Masked
Unmasked
test-a
96.8
96.9
94.7
94.5
test-b
98.1
98.8
98.1
98.8
test-c
96.2
96.2
80.5
80.4
test-d
90.7
90.5
69.1
69.0
Table 3: Comparison of masked vs. unmasked training. No major effect can be observed.
Idiom
1st
Type
2nd
Type
3rd
Type
Sursilvan
ei
char
scha
char
iu
char
Sutsilvan
àn
char
ù
char
eing
char
Surmiran
eir
char
dall
char
ous
char
Puter
ron
char
aunt
char
ọ
char
Vallader
ạ
char
ọn
char
ị
char
RG
␣
char
vegnis
word
ì
char
Table 4: Top 3 most discriminative features per idiom. “char” denotes character n -gram and “word” denotes word unigram. Character n -grams emerge as the most discriminative features across all varieties, with empty space ␣ being the most informative one for RG.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Source and URL to Dataset (if available)
Idiom
Total # of Samples
Total # of Tokens
Avg. # of Tokens/Sample
Sursilvan
34,724
92,182
2.65
Pledari Grond https://www.pledarigrond.ch
Sutsilvan
32,175
307,799
9.57
Surmiran
43,954
301,390
6.86
Puter
59,852
963,940
16.11
Vallader
72,470
1,141,688
15.75
RG
172,616
785,831
4.55
Appendix
Table 5: Summary of the number of samples extracted from each data source for each idiom, along with the number of whitespace tokens across all collected samples per data source and the average number of tokens per sample.
Set
Idiom
Total # of Samples
Total # of Tokens
Avg. # of Tokens/Sample
Sursilvan
55,574
3,663,641
65.92
Sutsilvan
44,188
1,342,065
30.37
Surmiran
55,543
1,457,845
26.25
train-set
Puter
74,355
2,353,323
31.65
Vallader
86,313
2,372,016
27.48
RG
171,199
1,190,808
6.96
Appendix
Table 6: Detailed statistics for all data splits by idiom, including number of samples, tokens, and average tokens per sample. The training set exhibits class imbalance, with RG containing the most samples but the shortest average length due to dictionary entries from Pledari Grond. test-c and test-d lack RG samples by construction.
Language identification determines which language or languages appear in text or speech, including dialects, code-switching, and low-resource varieties.