Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages

Felix Schneider; Maria Gogolev; Sven Sickert; Joachim Denzler

cs.CLFeb 24, 2026

Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages

Authors: Felix Schneider, Maria Gogolev, Sven Sickert, Joachim Denzler

Organizations: Computer Vision Group Friedrich Schiller University Jena

Abstract

Tokenization and sub-tokenization based models like word2vec, BERT and the GPTs are the state-of-the-art in natural language processing. Typically, these approaches have limitations with respect to their input representation. They fail to fully capture orthographic similarities and morphological variations, especially in highly inflected and under-resource languages. To mitigate this problem, we propose to computes word vectors directly from character strings, integrating both semantic and syntactic information. We denote this transformer-based approach Rich Character Embeddings (RCE). Furthermore, we propose a hybrid model that combines transformer and convolutional mechanisms. Both vector representations can be used as a drop-in replacement for dictionary- and subtoken-based word embeddings in existing model architectures. It has the potential to improve performance for both large context-based language models like BERT and small models like word2vec for under-resourced and morphologically rich languages. We evaluate our approach on various tasks like the SWAG, declension prediction for inflected languages, metaphor and chiasmus detection for various languages. Our experiments show that it outperforms traditional token-based approaches on limited data using OddOneOut and TopK metrics.

Figures & tables

Figure 1: An example for potential of subtokenization based embedding and our solution. We see the words first in a subtokenized manner and then with in our novel approach. The tokens that qualification is split into are distinct to the token for quality . Also, one word is split into three vectors. In contrast, our novel approach computes a single vector representation for each word and takes the spelling similarity into account.

Figure 3: This figure shows the model architecture for the Rich Character Embedding. The input token, in this case the word Token gets represented in the input as its character string, with the capital T split up into an [UP] modifier and the base character t . The character string is then transformed into a vector representation by the transformer encoder. The output of the encoder is then used as the word embedding.

Category	Tokens
Base characters	a–z
Digits	0–9
Special symbols	ø , þ , ð , ł , ŋ , ° , ^ , ! , " , § , $ , % , & , / , ( , ) , = , ? , ` , ´ , + , * , ~ , # , ' , _ , . , : , , , ; , < , > , @ , € , { , [ , ] , } , … , „ , “ , ” , ‚ , ’ , — , » , «
Special word-level tokens	[BEG] (beginning of a word), [END] (end of a word), [UNK] (unknown character), [PAD] (padding token at the end of a word)
Special character-level tokens	Padding and unknown characters
Modifiers for diacritics	Acute Accent ( ´ ), Grave Accent ( ` ), Macron ( ¯ ), Tilde ( ~ ), Horn ( ˛ ), Diaeresis ( ¨ ), Ligature modifier, Uppercase modifier, Sharp S modifier ( ß ) with base character ’s’

Table 1: Token categories and their corresponding characters.

language	categories	total words	average words per category
Faroese	8	161	20
Latin	9	158	18
Uzbek	9	141	16

Table 2: This table shows the statistics of the datasets we created for the TopK and OddOneOut evaluation methods.

target language	training language	RCE	c2v	RCE+c2v	FastText
Faroese	Faroese	0.26	0.20	0.25	0.21
Faroese	Norwegian	0.22	0.20	0.24	0.24
Faroese	Icelandic	0.24	0.22	0.26	0.22
Latin	Latin	0.23	0.25	0.27	0.25
Latin	Italian	0.16	0.26	0.24	0.15
Uzbek	Uzbek	0.20	0.25	0.25	0.20

Table 3: This table shows the results for the TopK evaluation. The scores are bounded between 0 and 0, with 1 being a perfect score and 0 being the worst score. The experiments were conducted on a Faroese dataset, using models trained on either Faroese, Icelandic, or Norwegian. We compare the results of RCE, c2v, and FastText. Since the whole testing dataset was used without any training splits, we can not provide standard deviations.

target language	training language	RCE	c2v	RCE+c2v	FastText
Faroese	Faroese	0.13	0.12	0.29	0.08
Faroese	Norwegian	0.22	0.29	0.24	0.09
Faroese	Icelandic	0.11	0.14	0.16	0.09
Latin	Latin	0.25	0.60	0.30	0.10
Latin	Italian	0.18	0.35	0.31	0.08
Uzbek	Uzbek	0.17	0.36	0.39	0.10

Table 4: This table shows the results for the OddOneOut evaluation. The scores are bounded between 0 and 0, with 1 being a perfect score and 0 being the worst score.

training language	RCE	c2v	RCE+c2v	FastText
Latin	$0.85\pm 0.02$	$0.90\pm 0.01$	$0.90\pm 0.01$	$0.85\pm 0.02$
Italian	$0.85\pm 0.01$	$0.88\pm 0.02$	$0.84\pm 0.01$	$0.50\pm 0.01$

Table 5: This table shows the results for the inflection experiments. The experiments were conducted on a Latin dataset. We compare the results of RCE, c2v and FastText. The cores are bound between 0 and 1, with 1 being a perfect score and 0 being the worst score.

	RCE	c2v	RCE+c2v	FastText
all features	$0.34\pm 0.17$	$0.32\pm 0.13$	$0.30\pm 0.11$	$0.31\pm 0.13$
only embedding	$0.19\pm 0.06$	$0.02\pm 0.00$	$0.32\pm 0.14$	$0.02\pm 0.00$

Table 6: Results for the chiasmus detection experiments. We compare the results of RCE, c2v, and FastText. The values are bounded between 0 and 1, with 1 being a perfect score and 0 being the worst score.

RCE	c2v	RCE+c2v	FastText
$0.73\pm 0.05$	$0.71\pm 0.05$	$0.74\pm 0.04$	$0.67\pm 0.04$

Table 7: This table shows the results for the metaphor detection experiments. We compare the results of RCE, c2v, RCE+c2v, and FastText. The results are for the German dataset. The values are bounded between 0 and 1, with 1 being a perfect score and 0 being the worst score.

pretraining	task training	testing	WordPiece	RCE
en	en	en	0.561	0.578
de	de	de	0.542	0.628
fo	fo	fo	0.279	0.265
no	no	fo	0.277	0.285
no	fo	fo	0.266	0.256
is	is	fo	0.244	0.280

Table 8: This table shows the results for the SWAG-like next sentence prediction task. The values are bounded between 0 and 1, with 1 being a perfect score and 0 being the worst score. In this case Word Piece and RCE were used as an embedding layer for a BERT-like model.

Word Embeddings

Word embeddings map words or tokens to vectors that capture patterns of meaning, usage, and linguistic structure.

See all 40 papers

LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation
Mar 23, 2026Hailay Teklehaymanot, Dren Fazlija, Wolfgang NejdlLow-Resource Language Processing Word Embeddings
L3S Research Center, Hannover, Germany
ROMEVA: Geometry-Preserving Vocabulary Expansion for Roman Urdu Language Models
Jun 21, 2026Mahnoor Khan, Afsheen Asif, Milhan Afzal Khan +2Multilingual Language Models Word Embeddings
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan · Department of Computer Science, University of Agriculture Faisalabad, Faisalabad, Pakistan
When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
Aug 27, 2026Sanjeev Kumar, Atsuki Yamaguchi, Nikolaos AletrasByte-Level Language Model Low-Resource Language Processing
Department of Computer Science and Engineering, IIT Bombay, India · School of Computer Science, University of Sheffield, United Kingdom

Mar 23, 2026J/K move · Enter open · S save

LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation

Hailay Teklehaymanot, Dren Fazlija, Wolfgang Nejdl

L3S Research Center, Hannover, Germany

Low-Resource Language Processing Word Embeddings Multilingual Language Models

Open paper AskPDF arXiv

Adapting pretrained language models to low-resource, morphologically rich languages remains a significant challenge. Existing vocabulary expansion methods typically rely on arbitrarily segmented subword units, resulting in fragmented lexical representations and loss of critical morphological information. To address this limitation, we propose the Lexically Grounded Subword Embedding Initialization (LGSE) framework, which introduces morphologically informed segmentation for initializing embeddings of novel tokens. Instead of using random vectors or arbitrary subwords, LGSE decomposes words into their constituent morphemes and constructs semantically coherent embeddings by averaging pretrained subword or FastText-based morpheme representations. When a token cannot be segmented into meaningful morphemes, its embedding is constructed using character n-gram representations to capture structural information. During Language-Adaptive Pretraining, we apply a regularization term that penalizes large deviations of newly introduced embeddings from their initialized values, preserving alignment with the original pretrained embedding space while enabling adaptation to the target language. To isolate the effect of initialization, we retain the original pre-trained model vocabulary and tokenizer and update only the new embeddings during adaptation. We evaluate LGSE on three NLP tasks: Question Answering, Named Entity Recognition, and Text Classification, in two morphologically rich, low-resource languages: Amharic and Tigrinya, where morphological segmentation resources are available. Experimental results show that LGSE consistently outperforms baseline methods across all tasks, demonstrating the effectiveness of morphologically grounded embedding initialization for improving representation quality in underrepresented languages. Project resources are available in the GitHub link.

Abstract

Word Embeddings

Explore similar work

LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation

ROMEVA: Geometry-Preserving Vocabulary Expansion for Roman Urdu Language Models

When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages