SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models

SungHo Kim; Juhyeong Park; Eda Atalay; SangKeun Lee

cs.CLApr 14, 2026

SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models

Authors: SungHo Kim, Juhyeong Park, Eda Atalay, SangKeun Lee

Organizations: Department of Artificial Intelligence, Korea University, Seoul, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea

Abstract

Korean is a morphologically rich language with a featural writing system in which each character is systematically composed of subcharacter units known as Jamo. These subcharacters not only determine the visual structure of Korean but also encode frequent and linguistically meaningful morphophonological processes. However, most current Korean language models (LMs) are based on subword tokenization schemes, which are not explicitly designed to capture the internal compositional structure of characters. To address this limitation, we propose SCRIPT, a model-agnostic module that injects subcharacter compositional knowledge into Korean PLMs. SCRIPT allows to enhance subword embeddings with structural granularity, without requiring architectural changes or additional pre-training. As a result, SCRIPT enhances all baselines across various Korean natural language understanding (NLU) and generation (NLG) tasks. Moreover, beyond performance gains, detailed linguistic analyses show that SCRIPT reshapes the embedding space in a way that better captures grammatical regularities and semantically cohesive variations. Our code is available at https://github.com/SungHo3268/SCRIPT.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Examples of the components of Hangul . This figure illustrates two characters, such as ‘춥 ${}_{\text{cold}}$ ’ and ‘다 ${}_{\text{ending suffix}}$ ’, with each subcharacter highlighted in blue . (b) Examples of linguistic phenomena arising from the inflection of predicate ‘춥다 ${}_{\text{be cold}}$ ’ at the subcharacter-level, with the transformed subcharacters highlighted in red .

Figure 2: Morphological modifications in a large-scale Korean POS-tagged corpus: the left panel distinguishes subcharacter- and character-level MOD cases, and the right lists the ten most frequent MOD types with raw target characters and corresponding factors.

Panel of Figure 3 — Figure 3: (a) Overall illustration of the PLM enhanced with SCRIPT . (b) Detailed architectural example of the SCRIPT , starting from the word ‘대한민국 ${}_{\text{South Korea}}$ ’, which is tokenized into 12 Jamo units: I : [ㄷ, ㅎ, ㅁ, ㄱ], V: [ㅐ, ㅏ, ㅣ, ㅜ ], F: [ , ㄴ, ㄴ, ㄱ]. Each sub-process (A-C) represents a successive fusion step: (A) fusion of Choseong and Jungseong (§ 3.2.1 ), (B) addition of Jongseong (§ 3.2.1 ), and (C) character-to-subword (§ 3.2.2 ).

Model	KorNLI	KorSTS	NSMC	PAWS-X	KoBEST
Model	KorNLI	KorSTS	NSMC	PAWS-X	BoolQ	COPA	WiC	HellaSwag	SentiNeg
KOMBO ${}_{\text{base}}$	75.97	77.28	88.34	73.40	61.40	61.00	68.91	63.80	79.07
BERT ${}_{\text{base}}$	75.85	76.72	88.96	72.38	60.75	60.90	73.14	63.20	83.12
BERT ${}_{\text{base}}$ + SCRIPT	76.49	77.68	88.96	73.68	62.32	61.30	74.30	64.40	83.38
KoGPT2 ${}_{\text{base}}$	72.24	73.82	88.90	76.33	67.22	68.90	67.07	69.10	88.50
KoGPT2 ${}_{\text{base}}$ + SCRIPT	72.47	74.27	88.80	76.61	68.28	70.90	68.18	72.40	89.47

Table 1: Performance on nine Korean NLU tasks. The evaluation metrics for each task are as follows: KorSTS is evaluated using Spearman correlation

\times

100, while other tasks are evaluated based on accuracy (%). The best results in each family of models are highlighted in boldface .

Model	KoCommonGen
Model	BLEU 3	BLEU 4	ROUGE-2	ROUGE-L	METEOR	mBERTScore	KoBERTScore
KoGPT2 ${}_{\text{base}}$	18.29	10.33	44.24	54.50	40.05	83.37	91.21
KoGPT2 ${}_{\text{base}}$ + SCRIPT	25.01	15.57	47.42	60.00	42.53	84.67	91.46
KoGPT3-1.2B	26.19	17.20	58.85	62.53	52.11	85.41	91.17
KoGPT3-1.2B + SCRIPT	28.89	19.58	59.28	64.80	52.37	86.26	91.78
EXAONE-2.4B	40.11	28.41	62.25	64.84	54.84	87.65	93.12

Table 2: Performance on KoCommonGen generative task. We use eight automatic evaluation metrics, including n-gram based measures like BLEU, ROUGE, and METEOR, and two BERT-based scores for semantic similarity. The best results in each family of models are highlighted in boldface .

SCRIPT		Fusion w/ PLM	KoBEST					Avg.
Initial Token Unit	Compression	Fusion w/ PLM	BoolQ	COPA	WiC	HellaSwag	SentiNeg	Avg.
Jamo	Principles	CrossAttention	68.28	70.90	68.18	72.40	89.47	73.85
Stroke	Principles	CrossAttention	68.35	65.20	67.81	71.00	89.46	72.36
Cji	Principles	CrossAttention	68.82	65.20	67.98	71.40	88.01	72.28
BTS	Principles	CrossAttention	67.99	64.70	66.67	71.50	88.97	71.97
Character	Principles	CrossAttention	66.00	59.30	63.37	69.40	88.40	69.29

Table 3: Ablation results for various architecture of SCRIPT applied to KoGPT2

{}_{\text{base}}

. The first row presents the best-performing variant. Cells corresponding to ablated components are highlighted in light blue . The global-best results are highlighted in boldface .

Model	PLM Tokenization	KoBEST					Avg.
Model	PLM Tokenization	BoolQ	COPA	WiC	HellaSwag	SentiNeg	Avg.
BERT ${}_{\text{base}}$	Word	60.04	5 7.60	62.70	55.00	5 2.39	57.55
BERT ${}_{\text{base}}$ + SCRIPT ${}_{\text{Jamo}}$	Word	6 0.47	5 7.60	6 4.76	5 7.20	5 2.39	5 8.48( ▲ 0.94)
BERT ${}_{\text{base}}$	Morpheme	63.75	58.50	71.75	61.80	78.59	66.88
BERT ${}_{\text{base}}$ + SCRIPT ${}_{\text{Jamo}}$	Morpheme	6 5.03	6 0.00	72.06	6 2.20	8 0.35	6 7.93( ▲ 1.05)
BERT ${}_{\text{base}}$	Subword	67.22	67.10	68.90	69.10	88.50	72.16

Table 4: Comparison of the effectiveness of compositional knowledge integration into PLMs across different tokenization methods. The global-best results are highlighted in boldface and local-best results for each section are highlighted in underline , respectively.

Figure 4: PCA visualization of subword embeddings for word pairs exhibiting subcharacter-level alternations. Each pair (e.g., 자다 ${}_{\text{sleep}}$ –잤다 ${}_{\text{slept}}$ , 눕다 ${}_{\text{lie}}$ –누웠다 ${}_{\text{lay}}$ ) shares the same root meaning but differs in tense.

Figure 5: PCA visualization of word embeddings averaged over tokens for five semantically related Korean predicate inflections derived from the predicate ‘춥다 ${}_{\text{be cold}}$ ’. The dashed boundaries indicate the dispersion ranges of the smallest baseline, KoGPT2 ${}_{\text{base}}$ , and its SCRIPT -augmented counterpart.

Figure 6: Three structural types of Korean syllable blocks, classified by the spatial arrangement of Choseong, Jungseong, and Jongseong.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Input Character	Oracle Actions	Output Lemma Units
런 (reon)	B-MOD-럽, I-MOD-ㄴ	럽 (reob), ㄴ (n)
했 (haess)	B-MOD-하, I-MOD-았	하 (ha), 았 (ass)
다 (da)	B-KEEP	다 (da)

Appendix

Table 5: Examples of oracle actions aligned with lemma units. Each input character may correspond to multiple oracle actions (e.g., B-MOD, I-MOD) depending on the type and structure of the morphological transformation. The output shows the resulting lemma units produced by these actions.

Algorithm 1: Subword Representation with SCRIPT

Input:

\mathbf{s}

(raw input sentence), embedding dimension

D

Output:

\mathbf{e}_{\text{F}}

(fused subword-level embeddings).

\mathbf{t}_{\text{subchar}}\leftarrow\textsc{Tokenize}_{\text{subchar}}(\mathbf{s})

// subcharacter sequence

\mathbf{e}\leftarrow\textsc{Embedding}_{\text{subchar}}(\mathbf{t}_{\text{subchar}})

// subcharacter representation

3 if first token in

\mathbf{e}

is [CLS] then // (for encoder-only model)

\mathbf{e}_{\text{CLS}}\leftarrow\mathbf{e}[1]

Appendix

Table 6: The input text

\mathbf{s}

is tokenized into subcharacter tokens (line 1) and subword tokens (line 10), respectively (§ 3.2 ). While structure-aware subword embeddings are built bottom-up from subcharacter to character to subword (lines 2-9), the original subword embeddings are looked up (lines 10-11). These two subword representation streams are fused by cross-attention to yield the final subword representation

\mathbf{e}_{\text{F}}

(lines 12-13) (§ 3.3 ).

Figure 7: Implementation of SCRIPT for the BTS unit. This illustrates the hierarchical integration of subword representations derived from BTS unit in SCRIPT , using the example word ‘대한민국 ${}_{\text{South Korea}}$ ’. The word ‘대한민국 ${}_{\text{South Korea}}$ ’ consists of two subwords ([S]: 대한, 민국), four characters (대, 한, 민, 국), and eighteen subcharacters: initial consonants ([ I ]: ㄴ, -, ㅇ, -, ㅁ, ㄱ), vowels ([V]: ㅣ, $\cdot$ , ㅣ, ㅣ, $\cdot$ , ㅣ, ㅡ, $\cdot$ ), and final consonants ([F]: , ㄴ, ㄴ, ㄱ).

Task	Epoch	Batch Size	Learning Rate	Dropout Ratio	Warmup Ratio	LoRA r	LoRA $\alpha$
KorNLI	5	64	BERT: {5e-05, 1e-04} GPT : {1e-05, 5e-05, 1e-04, 1e-03, 3e-03, 1e-02}	0.03	0.1	32	128
KorSTS	15	64
NSMC	5	64
PAWS-X	10	64
BoolQ	10	8	BERT: {1e-05, 5e-05} GPT : {1e-05, 5e-05, 1e-04, 1e-03, 3e-03, 1e-02}	0.03	0.1	32	128
COPA	15	16

Appendix

Table 7: Hyperparameters used in all experiments in this paper for each task. For the learning rate, we select the value that yields the best performance for each baseline. Here, “BERT” refers to encoder-only models, including BERT and KOMBO, while “GPT” encompasses all decoder-only models, such as KoGPT2, KoGPT3, mGPT, and EXAONE.

Model	XL-Sum
Model	BLEU 3	BLEU 4	ROUGE-2	ROUGE-L	METEOR	mBERTScore	KoBERTScore
KoGPT2 ${}_{\text{base}}$	7.27	4.98	12.91	26.83	13.43	76.22	88.97
KoGPT2 ${}_{\text{base}}$ + SCRIPT	7.64	5.20	13.30	27.24	13.67	76.31	89.20
KoGPT3-1.2B	9.14	6.21	15.88	30.18	16.91	76.95	89.69
KoGPT3-1.2B + SCRIPT	9.39	6.40	15.99	30.31	16.97	77.46	89.77

Appendix

Table 8: Performance on XL-Sum multilingual summarization task. We evaluate only on the Korean summarization dataset. We use seven automatic evaluation metrics, including n-gram-based measures like BLEU, ROUGE, and METEOR; and two BERT-based scores Zhang et al. (2019) , mBERTScore and KoBERTScore, for semantic similarity. The global-best results are highlighted in boldface and local-best results for each model are highlighted in underline , respectively.

Model	Kor-Learner				Kor-Native
Model	$M^{2}_{pre}$	$M^{2}_{rec}$	$M^{2}_{F_{0.5}}$	GLEU	$M^{2}_{pre}$	$M^{2}_{rec}$	$M^{2}_{F_{0.5}}$	GLEU
KoGPT2 ${}_{\text{base}}$	29.35	16.11	25.19	21.60	72.12	55.19	67.76	61.45
KoGPT2 ${}_{\text{base}}$ + SCRIPT	30.02	16.83	25.34	23.54	72.95	56.15	69.00	62.25
KoGPT3-1.2B	47.05	23.01	38.89	35.22	84.76	69.54	81.20	75.21
KoGPT3-1.2B + SCRIPT	49.45	26.36	41.68	39.45	85.49	70.02	81.87	75.40

Appendix

Table 9: Performance on two Korean GEC tasks. As the evaluation metrics, we use

{M^{2}}

scorer Dahlmeier and Ng (2012) , which measures precision, recall, and F 0.5 scores based on edits and GLEU Napoles et al. (2015) for the simple n-gram matching. The global-best results are highlighted in boldface and local-best results for each model are highlighted in underline , respectively.

Model	KoBEST					Avg.
Model	BoolQ	COPA	WiC	HellaSwag	SentiNeg	Avg.
KoGPT3-1.2B	77.32	82.80	72.78	78.90	96.31	81.62
KoGPT3-1.2B + SCRIPT	77.63	82.80	74.65	79.30	96.48	82.17
mGPT-1.3B	71.19	69.70	68.38	76.10	89.40	74.95
mGPT-1.3B + SCRIPT	70.72	70.60	69.17	76.70	90.42	75.52

Appendix

Table 10: Performance of KoGPT3-1.2B and mGPT-1.3B on KoBEST benchmark. The evaluation metrics for each task are accuracy (%). The global-best results are highlighted in boldface and local-best results for each model are highlighted in underline , respectively.

Model	KoCommonGen							Avg.
Model	BLEU 3	BLEU 4	ROUGE-2	ROUGE-L	METEOR	mBERTScore	KoBERTScore	Avg.
KoGPT3-1.2B	26.19	17.20	58.85	62.53	52.11	85.41	91.17	56.21
KoGPT3-1.2B + SCRIPT	28.89	19.58	59.28	64.80	52.37	86.26	91.78	57.57
mGPT-1.3B	15.16	8.07	37.77	50.99	33.31	80.17	89.34	44.97
mGPT-1.3B + SCRIPT	16.59	9.11	39.31	52.68	34.77	80.82	89.95	46.18

Appendix

Table 11: Performance of KoGPT3-1.2B and mGPT-1.3B on KoCommonGen dataset. We use eight automatic evaluation metrics: BLEU, ROUGE, and METEOR for n-gram-based measures; and mBERTScore and KoBERTScore for semantic similarity. The global-best results are highlighted in boldface and local-best results for each model are highlighted in underline , respectively.

Task	Lang	Example
KoCommonGen	Ko	Input: {있, 선로, 길, 옆, 열차 } Gold Label: 길 옆의 선로 에 열차가 있다. KoGPT2: 선로가 길 옆의 길 옆에 열차가 세워져 있다. KoGPT2 + SCRIPT : 열차들이 길 옆의 선로 에 있다.
KoCommonGen	En	Input: {be, tracks, road, beside, train } Gold Label: There is a train on the tracks beside the road. KoGPT2: The train is parked beside the track beside the road. KoGPT2 + SCRIPT : The trains are on the tracks beside the road.
Kor-Learner	Ko	Input: 그리고 가장 중요한 영향은 그 앞으로 그 여행으로 이전보다 훨씬 더 ‘처음’ 을 접할 거다 . Gold Label: 그리고 가장 중요한 영향은 앞으로 여행으로 이전보다 훨씬 더 ‘처음’ 을 접할 것이라는 사실이다 . KoGPT2: 그리고 가장 중요한 영향을 그 앞으로 그 여행으로 이전보다 훨씬 더 ‘처음’ 을 접할 거다 . KoGPT2 + SCRIPT : 그리고 가장 중요한 영향은 그 앞으로 그 여행으로 이전보다 훨씬 더 ‘처음’ 을 접할 거라는 것이다 .
Kor-Learner	En	Input: And the most important impact would that, through that journey, they will encounter the ‘first’ much more than before. Gold Label: And the most important impact is the fact that, through future travels, they will encounter the ‘first’ much more than before. KoGPT2: And the most important impact would that, through that journey, they will encounter the ‘first’ much more than before. KoGPT2 + SCRIPT : And the most important impact is the thing that, through that journey, they will encounter the ‘first’ much more than before.
Kor-Native	Ko	Input: 주말에 함께 테니그 를 쳐여. Gold Label: 주말에 함께 테니스 를 쳐요. KoGPT2: 주말에 함께 테니그 를 쳐요. KoGPT2 + SCRIPT : 주말에 함께 테니스 를 쳐요.
	En	Input: Let’s play tennig together on the weekend. Gold Label: Let’s play tennis together on the weekend. KoGPT2: Let’s play tennig together on the weekend. KoGPT2 + SCRIPT : Let’s play tennis together on the weekend.

Appendix

Table 12: Examples for the qualitative analysis of four generation tasks. For each task, examples are composed of the input provided to the model, the gold label, and the predictions generated by two baselines: KoGPT2

{}_{\text{base}}

and KoGPT2

{}_{\text{base}}

applied with SCRIPT . The model outputs were generated in Korean, with English translations provided alongside for clarity. The asterisk (*) indicates a ungrammatical word. Red-colored characters represent incorrect parts, while blue-colored characters indicate correct representations.

Task	Language	Example
XL-Sum	Korean	Input: 최종 목적은 2㎏ 정도의 ‘토양’ 표본을 상승선, 귀환선에 전달해 지구까지 가져오는 것이다 중국국가우주국(CNSA)은 달의 암석과 토양 표본을 수집해 지구로 가져오기 위해 출발한 무인 달 탐사선 ‘창어 5호’가 1일 밤 착륙에 성공했다고 2일 밝혔다. 창어 5호는 ‘폭풍의 바다’(Oceanus Procellarum)라는 지역 내 ‘몽스 륌케르’(Mons Rümker) 화산지대 북쪽에 안착했다. 이곳에서 며칠간 달 표면의 흙과 암석 표본 등을 수집한다. 창어 5호 탐사선에는 작업을 돕기 위한 카메라, 레이더, 드릴, 삽 등이 탑재돼있다. 최종 목적은 2㎏ 정도의 표토 표본을 상승선과 궤도선을 거쳐 귀환선에 전달해 지구까지 가져오는 것이다. 달의 토양 표본을 지구로 가져온 탐사선은 44년 전 1976년 옛 소련의 루나 24호가 마지막으로, 당시 200g의 토양을 지구로 옮기는 데 성공했다. 창어 5호 프로젝트 팀이 환호하는 모습 이날 달 착륙 모습은 일주일 전 발사 때와 달리 생중계 되지 않았다. 중국 TV 채널에서는 성공적인 착륙이 확인되고 나서야 정규 방송을 중단하고 이를 녹화 중계했다. 공개된 착륙 과정에는 탐사선의 다리가 달의 먼지 쌓인 표면에 그림자를 드리우는 장면 등이 포함됐다. … KoGPT2: 중국 국가우주국이 달 착륙에 성공한 창어 6호 달 착륙선에 탑재된 카메라와 레이더를 통해 달 표면 표본을 지구까지 운반했다. KoGPT2 + SCRIPT : 중국의 달 탐사 프로젝트가 성공적으로 마무리됐다.
	English	Input: The primary goal is to bring approximately 2 kg of lunar soil samples back to Earth by transferring them from the ascent and return modules. The China National Space Administration (CNSA) announced on the 2nd that its unmanned lunar probe, Chang’e-5, successfully landed on the night of the 1st to collect lunar rock and soil samples to return to Earth. Chang’e-5 has landed in the volcanic area north of Mons Rümker within the region known as Oceanus Procellarum. Over the next few days, it will collect samples of lunar soil and rock. Equipped with cameras, radar, drills, and shovels to aid in its operations, the ultimate goal of Chang’e-5 is to gather about 2 kg of surface samples, which will be transferred from the ascent and orbital modules to the return module for their journey back to Earth. The last mission to bring lunar soil samples to Earth was the Soviet Union’s Luna 24 in 1976, which successfully transported 200 g of lunar soil back to Earth. On the day of the landing, the Chang’e-5 team celebrated. Unlike the launch, the landing was not broadcast live, and Chinese TV channels interrupted regular programming to air recorded footage after confirming a successful landing. The released landing process included images of the lander casting a shadow on the dusty lunar surface. … Gold Label: China has landed another probe on the surface of the moon. KoGPT2: The China National Space Administration successfully transported lunar surface samples to Earth using the camera and radar aboard the Chang’e 6 lunar lander. KoGPT2 + SCRIPT : China’s lunar exploration project has been successfully completed.

Appendix

Table 13: Examples for the qualitative analysis of XL-Sum task. For each task, examples are composed of the input provided to the model, the gold label, and the predictions generated by two baselines: KoGPT2

{}_{\text{base}}

and KoGPT2

{}_{\text{base}}

applied with SCRIPT . The model outputs were generated in Korean, with English translations provided alongside for clarity.

Figure 8: Visualization of the similarities between word representations, measured from conjugated word pairs. Each word set contains five words that share the same root meaning. The first word set, {‘춥다’, ‘추움’, ‘추위’, ‘추웠어’, ‘춥디춥다’}, conveys the meaning ‘cold’; the second set, {‘걷다’, ‘걷기’, ‘걸어’, ‘걸었어’, ‘걸음’}, represents ‘walk’; the third set, {‘돕다’, ‘도움’, ‘도와’, ‘도왔어’, ‘돕기’}, signifies ‘help’; and the final set, {‘묻다’, ‘물어보다’, ‘물었다’, ‘물어보기’, ‘물어’}, conveys the meaning ‘ask’. Using these word sets, we compared three different types of embeddings: those derived from subcharacter embeddings in SCRIPT , subword embeddings from the PLM, and fusion embeddings augmented by SCRIPT .

Model	Embedding Layer	Transformer Stacks	Restoration Layer
BERT	$O(1)$	$O({N_{\text{s}}}^{2}D)$	-
BERT + SCRIPT	$O(N_{\text{j}}D^{2}+{N_{\text{s}}}^{2}D$ )	$O({N_{\text{s}}}^{2}D)$	-
KOMBO	$O(N_{\text{j}}D^{2}+{N_{\text{j}}}^{2}D)$	$O({N_{\text{c}}}^{2}D)$	$O(N_{\text{c}}D^{2})$

Appendix

Table 14: Comparison of the computational complexities across the three components of the model’s architecture.

N_{\text{j}}

is the length of subcharacter sequence,

N_{\text{s}}

is the length of subword sequence,

N_{\text{c}}

is the length of character sequence, and

D

is the hidden size.

Panel of Figure 9 — Figure 9: Comparison of computational costs among the base model BERT ${}_{\text{base}}$ , BERT ${}_{\text{base}}$ with SCRIPT , and previous Jamo-based PLM, KOMBO ${}_{\text{base}}^{\text{Jamo}}$ . The results were obtained on the KB-HellaSwag benchmark, a representative Korean NLU task, using a single NVIDIA RTX 3090 GPU. (a) Peak GPU memory usage during training with varying input sequence lengths. (b) Training time per epoch with varying input sequence lengths.

Figure 10: The graphs show the fine-tuning performances of three models, BERT ${}_{\text{base}}$ , BERT ${}_{\text{base}}$ + SCRIPT , and KOMBO ${}_{\text{base}}$ , across nine NLU tasks. The x-axis represents the number of epochs for each task, and the y-axis indicates the accuracy score. Two types of dotted vertical lines are overlaid on the line graphs for each model: the red dotted line marks the epoch at which the model with the proposed SCRIPT module achieves its best performance, while the gray dotted lines indicate the best-performing epochs for the other two models.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

Apr 27, 2026

KOMBO: Korean Character Representations Based on the Combination Rules of Subcharacters

The Korean writing system, Hangeul, has a unique character representation rigidly following the invention principles recorded in Hunminjeongeum.\footnote{Hunminjeongeum is a book published in 1446 that describes the principles of invention and usage of Hangeul, devised by King Sejong \cite{Hunminjeongeum_Guide}.} However, existing pre-trained language models (PLMs) for Korean have overlooked these principles. In this paper, we introduce a novel framework for Korean PLMs called KOMBO, which firstly brings the invention principles of Hangeul to represent character. Our proposed method, KOMBO, exhibits notable experimental proficiency across diverse NLP tasks. In particular, our method outperforms the state-of-the-art Korean PLM by an average of 2.11% in five Korean natural language understanding tasks. Furthermore, extensive experiments demonstrate that our proposed method is suitable for comprehending the linguistic features of the Korean language. Consequently, we shed light on the superiority of using subcharacters over the typical subword-based approach for Korean PLMs. Our code is available at: https://github.com/SungHo3268/KOMBO.

SungHo Kim, Juhyeong Park, Yeachan Kim +1

Department of Artificial Intelligence, Korea University, Seoul, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea

Language Modeling Subword Tokenization

Ask PDF

Feb 24, 2026

Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages

Tokenization and sub-tokenization based models like word2vec, BERT and the GPTs are the state-of-the-art in natural language processing. Typically, these approaches have limitations with respect to their input representation. They fail to fully capture orthographic similarities and morphological variations, especially in highly inflected and under-resource languages. To mitigate this problem, we propose to computes word vectors directly from character strings, integrating both semantic and syntactic information. We denote this transformer-based approach Rich Character Embeddings (RCE). Furthermore, we propose a hybrid model that combines transformer and convolutional mechanisms. Both vector representations can be used as a drop-in replacement for dictionary- and subtoken-based word embeddings in existing model architectures. It has the potential to improve performance for both large context-based language models like BERT and small models like word2vec for under-resourced and morphologically rich languages. We evaluate our approach on various tasks like the SWAG, declension prediction for inflected languages, metaphor and chiasmus detection for various languages. Our experiments show that it outperforms traditional token-based approaches on limited data using OddOneOut and TopK metrics.

Felix Schneider, Maria Gogolev, Sven Sickert +1

Computer Vision Group Friedrich Schiller University Jena

Word Embeddings Low-Resource Language Processing

Ask PDF

Jun 13, 2026

PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino

Large language models (LLMs) process text as sequences of subword tokens, which can obscure the character-level and morphological structure that underlies word formation. This limitation is most acute for languages with non-concatenative morphology, where standard tokenizers systematically misalign token boundaries with morpheme boundaries. We introduce PACUTE, a diagnostic benchmark of 4,600 tasks designed to evaluate morphological understanding in Filipino, a language characterized by productive infixation, reduplication, and diacritic-driven lexical distinctions that are typically absent from written text. PACUTE includes a hierarchical diagnostic framework of six compositional levels that localizes where morphological understanding breaks down. Evaluating open-weight LLMs and frontier commercial models, we find that open-weight models perform near chance on morpheme decomposition regardless of scale. Frontier models perform much better, often recovering individual affixes under contains-match scoring, but remain far below their character-level ceilings on compositional tasks of morpheme transformations and syllabification. These results identify productive morphological composition, rather than character access alone, as the persistent bottleneck for Filipino word-structure understanding.

Jann Railey Montalan, David Demitri Africa, Jimson Paulo Layacan +3

AI Singapore · Nanyang Technological University · UK AI Security Institute +2

LLM Evaluation Language Modeling Low-Resource Language Processing Morphological Analysis

Ask PDF

Abstract

Language Modeling

Explore similar work

KOMBO: Korean Character Representations Based on the Combination Rules of Subcharacters

Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages

PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino