RUMLEM: A Dictionary-Based Lemmatizer for Romansh
Organizations: Department of Computational Linguistics, University of Zurich
Abstract
Lemmatization -- the task of mapping an inflected word form to its dictionary form -- is a crucial component of many NLP applications. In this paper, we present RUMLEM, a lemmatizer that covers the five main varieties of Romansh as well as the supra-regional standard variety Rumantsch Grischun. It is based on comprehensive, community-driven morphological databases for Romansh, enabling RUMLEM to cover 77-84% of the words in a typical Romansh text. Since there is a dedicated database for each Romansh variety, an additional application of RUMLEM is variety-aware language classification. Evaluation on 30'000 Romansh texts of varying lengths shows that RUMLEM correctly identifies the variety in 95% of cases. In addition, a proof of concept demonstrates the feasibility of Romansh vs. non-Romansh language classification based on the lemmatizer.
Figures & tables
| Unique Entries | Single Words | DE Translations | POS Tags | Gender | Infl. Verbs | |
|---|---|---|---|---|---|---|
| Sursilvan | 147,977 | 93,211 | 147,971 | 67,201 | 72,460 | 5,031 |
| Sutsilvan | 58,584 | 39,191 | 58,581 | 42,817 | 30,876 | 3,021 |
| Surmiran | 74,986 | 44,365 | 74,986 | 39,224 | 36,956 | 2,947 |
| Puter | 89,908 | 32,084 | 89,807 | 13,712 | 36,918 | 3,383 |
| Vallader | 106,690 | 35,322 | 106,438 | 10,854 | 48,435 | 3,779 |
| RG | 249,169 | 94,291 | 249,165 | 98,046 | 161,942 | 3,867 |
| Vocab | Mapped Forms | Lemmas | Noun | Adj | Verb | Other | |
|---|---|---|---|---|---|---|---|
| Sursilvan | 223,826 | 222,860 | 36,505 | 23,206 | 4,977 | 5,858 | 2,464 |
| Sutsilvan | 129,519 | 87,902 | 19,326 | 12,467 | 2,671 | 3,033 | 1,155 |
| Surmiran | 149,078 | 84,481 | 22,838 | 15,145 | 3,107 | 3,204 | 1,382 |
| Puter | 180,361 | 107,758 | 26,201 | 15,534 | 3,122 | 3,102 | 4,443 |
| Vallader | 165,354 | 109,090 | 30,479 | 19,841 | 4,821 | 3,625 | 2,192 |
| RG | 180,690 | 112,914 | 43,118 | 31,200 | 6,099 | 4,049 | 1,770 |
| Variety | Form + features | Gloss |
| rm-surmiran | fomanto [PoS=ADJ; Gender=FEM; Number=SG] | hungrig |
| rm-surmiran | fomantar [PoS=V; VerbForm=PTCP; Tense=PST; Gender=FEM; Number=SG] | aushungern |
| rm-vallader | fomantà [PoS=ADJ; Gender=FEM; Number=SG] | ausgehungert |
| rm-vallader | fomantà [PoS=ADJ; Gender=FEM; Number=SG] | hungrig |
| rm-vallader | fomantada [PoS=N; Gender=FEM; Number=SG] | Ausgehungerte |
| rm-vallader | fomantada [PoS=N; Gender=FEM; Number=SG] | Hungrige |
| Length | 2–10 | 10–50 | 50–300 | 300–800 | 800+ | All |
|---|---|---|---|---|---|---|
| Sursilvan | 0.85 | 0.85 | 0.87 | 1.00 | 1.00 | 0.86 |
| Sutsilvan | 1.0 | 0.99 | 1.0 | 1.00 | 1.00 | 1.00 |
| Surmiran | 0.92 | 0.94 | 0.99 | 1.00 | 1.00 | 0.95 |
| Puter | 0.97 | 0.98 | 0.99 | 1.00 | 1.00 | 0.98 |
| Vallader | 0.94 | 0.91 | 0.93 | 1.00 | 1.00 | 0.92 |
| RG | 0.89 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Length | 2–10 | 10–50 | 50–300 | 300–800 | 800+ | All |
|---|---|---|---|---|---|---|
| Surs. | 0.76 | 0.81 | 0.84 | 0.83 | 0.78 | 0.84 |
| Suts. | 0.71 | 0.77 | 0.77 | 0.73 | 0.73 | 0.77 |
| Surm. | 0.73 | 0.80 | 0.84 | 0.82 | 0.79 | 0.82 |
| Puter | 0.79 | 0.83 | 0.84 | 0.84 | 0.81 | 0.84 |
| Vall. | 0.66 | 0.77 | 0.80 | 0.83 | 0.81 | 0.80 |
| RG | 0.93 | 0.79 | 0.79 | 0.80 | 0.78 | 0.79 |
| Length | 2–10 | 10–50 | 50–300 | 300–800 | 800+ | Tot |
|---|---|---|---|---|---|---|
| Surs. | 68 | 2647 | 4173 | 7 | 5 | 6900 |
| Suts. | 6 | 1190 | 1795 | 5 | 7 | 3003 |
| Surm. | 1113 | 3751 | 2204 | 6 | 6 | 7080 |
| Puter | 660 | 2783 | 2468 | 5 | 7 | 5923 |
| Vall. | 277 | 2209 | 3202 | 5 | 7 | 5700 |
| RG | 9 | 1218 | 3052 | 6 | 6 | 4291 |
| Length | 50–300 | 300–800 | 800–2000 | Tot |
|---|---|---|---|---|
| French | 2517 | 1551 | 693 | 4761 |
| Italian | 2493 | 1620 | 671 | 4784 |
| Romanian | 2128 | 1513 | 922 | 4563 |
| Catalan | 2595 | 1575 | 593 | 4763 |
| Romansh | 4242 | 661 | 75 | 4978 |
| Tot | 13975 | 6920 | 2954 | 23849 |
Low-Resource Language Processing
Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.
See all 565 papers