Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text

Lifan Deng; Yongwei Zhang; Sen Sun; Bojun Sun; Jingsong Yu

cs.CLAug 19, 2026

Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text

Authors: Lifan Deng, Yongwei Zhang, Sen Sun, Bojun Sun, Jingsong Yu

Organizations: Key Laboratory of Linguistics, Chinese Academy of Social Sciences (University of Chinese Academy of Social Sciences), Beijing · Rixin College, Tsinghua University, Beijing · Institute of Linguistics, Chinese Academy of Social Sciences, Beijing · Institute of Ethnology and Anthropology, Chinese Academy of Social Sciences, Beijing · School of Software and Microelectronics, Peking University, Beijing

Abstract

Tangut is an extinct language whose script does not explicitly mark word boundaries. We present the first systematic study of Tangut word segmentation using 2,750 expert-annotated segments (31,893 tokens), traditional lexicons, and unlabeled text. Our framework combines a reliability-calibrated lexicon-lattice representation, explicit distributional statistics, and a lightweight character encoder pretrained with MLM. In within-source five-fold cross-validation, the model integrating TangutEncoder, CRF, and external features obtains the numerically highest main-system mean F1 of 0.911 and substantially improves recall beyond the labeled training vocabulary. We further evaluate document-level transfer on 479 segments (4081 tokens) from five works absent from the annotated training corpus. You can access our project at https://github.com/jiangli-va/TangutSeg.

Figures & tables

Figure 1: Surviving fragment of Mahāratnakūṭa-sūtra , Scroll 68.

Category	Segments	Tokens	Types
Buddhist scriptures	234	3717	769
Secular documents	2516	28176	4046
Total	2750	31893	4433

Table 1: Statistics of the expert-annotated Tangut corpus.

Symbol	Feature group	Dim.
BIE	Lexicon-lattice indicators	11
R	Reliability of observed entries	3
P	Prior for unseen entries	3
M	Lexicographic metadata	3
$\mathrm{Dict}_{\mathrm{all}}$	Full lexicon representation	20

Table 2: Composition of the 20-dimensional lexicon representation.

Symbol	Feature group	Dim.
Freq	Bigram frequency	2
Coo	Character association	2
Ent	Neighbor entropy	4
$\mathrm{Dist}_{\mathrm{all}}$	Full distributional representation	8

Table 3: Explicit distributional features extracted from unlabeled text.

Supervised baselines (RQ1)
Model	P	R	F 1	OOV-R	IV-R
Dict-corpus	$0.845{\pm}0.005$	$0.890{\pm}0.003$	$0.867{\pm}0.004$	$0.219{\pm}0.014$	$\textbf{0.954}{\pm}0.002$
CRF	$0.880{\pm}0.004$	$0.888{\pm}0.003$	$0.884{\pm}0.003$	$0.415{\pm}0.026$	$0.933{\pm}0.003$
BiLSTM–CRF	$0.862{\pm}0.008$	$0.873{\pm}0.008$	$0.868{\pm}0.008$	$0.472{\pm}0.014$	$0.912{\pm}0.006$
Transformer-Random	$0.848{\pm}0.008$	$0.863{\pm}0.010$	$0.855{\pm}0.009$	$0.519{\pm}0.020$	$0.896{\pm}0.010$
Lexicon-lattice ablation (RQ2)

Table 4: Overall segmentation results under five-fold cross-validation.

	Secular			Religious
Model	F 1	OOV-R	IV-R	F 1	OOV-R	IV-R
Dict-corpus	$0.869{\pm}0.004$	$0.229{\pm}0.017$	$\textbf{0.956}{\pm}0.003$	$0.855{\pm}0.019$	$0.132{\pm}0.028$	$0.936{\pm}0.010$
CRF	$0.887{\pm}0.010$	$0.430{\pm}0.035$	$0.934{\pm}0.007$	$0.839{\pm}0.011$	$0.262{\pm}0.067$	$0.909{\pm}0.009$
BiLSTM–CRF	$0.875{\pm}0.007$	$0.498{\pm}0.016$	$0.917{\pm}0.006$	$0.810{\pm}0.021$	$0.244{\pm}0.028$	$0.875{\pm}0.021$
Transformer-Random	$0.868{\pm}0.008$	$0.545{\pm}0.018$	$0.906{\pm}0.010$	$0.760{\pm}0.014$	$0.280{\pm}0.060$	$0.816{\pm}0.017$
CRF+ $\text{Dict}_{\text{all}}$	$0.904{\pm}0.005$	$0.481{\pm}0.024$	$0.942{\pm}0.004$	$0.872{\pm}0.004$	$0.301{\pm}0.058$	$0.938{\pm}0.008$

Table 5: Mean performance by document genre.

Model	F 1
CRF	0.686
CRF + $\mathrm{Dict}_{\mathrm{all}}$	0.726
CRF + $\mathrm{Dict}_{\mathrm{all}}$ + $\mathrm{Dist}_{\mathrm{all}}$	0.747
Transformer-Random	0.642
Transformer-Char2Vec	0.652
TangutEncoder	0.722

Table 6: Segmentation results on five unseen works. Full results are provided in Appendix C.4 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Audit unit	All	Religious	Secular
Exact segment ( $\geq 3$ chars)	0	0	0
5-gram overlap (%)	0.37	2.66	0.02
10-gram overlap (%)	0.01	0.08	0.00

Appendix

Table 7: Normalized overlap between annotated and unlabeled text.

Figure 2: Annotation examples from the religious and secular portions of the corpus. Vertical bars indicate expert-annotated word boundaries.

Group	Labels
Core lexical categories	a, c, d, m, n, p, q, r, u, v
Fine-grained lexical labels	b, l, t, nb, nc, nh, nl, no, ns, mc, mo, rd, ri, rp
Morphosyntactic labels	Dir1., Dir2., Erg., Obj., Quot., Nom., Pfv., Fut., Loc., 1sg., 2sg., pl.

Appendix

Table 8: Normalized linguistic-label inventory.

Work	Genre	Seg.	Tokens	Types	OOV
《华严经》	Religious	84	880	369	28.07%
《妙法莲华经》	Religious	128	1380	457	20.22%
《无量寿经》	Religious	90	1043	488	23.49%
《德行集》	Secular	71	406	203	21.92%
《贤智集》	Secular	106	372	257	37.63%
Total	—	479	4081	1289	24.50%

Appendix

Table 9: Composition of the independent cross-document test set. OOV rates are computed over word tokens relative to the complete supervised training corpus. Word types in the total row are deduplicated across the five works.

Parameter	Value
L1 coefficient ( $c_{1}$ )	$1.0$
L2 coefficient ( $c_{2}$ )	$10^{-3}$
Maximum L-BFGS iterations	200
Validation-based stopping	Not used
All possible transitions	Enabled
Character context window	$\pm 2$ characters

Appendix

Table 10: Hyperparameters of the linear CRF. All reported CRF results use a fixed budget of 200 L-BFGS iterations.

Layers	Emb.	Hidden	Batch	Dropout	LR	F 1
2	100	256	512	0.3	$5\times 10^{-4}$	0.8636
2	100	128	512	0.3	$5\times 10^{-4}$	0.8595
2	100	64	512	0.3	$\mathbf{5\times 10^{-4}}$	0.8905
2	100	32	512	0.3	$5\times 10^{-4}$	0.8873
2	100	16	512	0.3	$5\times 10^{-4}$	0.8423
1	100	64	512	0.3	$5\times 10^{-4}$	0.8638

Appendix

Table 11: Preliminary hyperparameter search for the supervised BiLSTM–CRF. Emb. denotes the character-embedding dimension, and Hidden denotes the concatenated output dimension of the two directions. The best segmentation F 1 is highlighted.

Parameter	Selected value
Character embedding size	100
BiLSTM layers	2
Hidden size	32 per direction
BiLSTM output size	64
Emission size	4
Model dropout	0.3

Appendix

Table 12: Selected architectural and training parameters of the BiLSTM–CRF. The maximum number of epochs is only an upper bound; the checkpoint with the lowest development loss is restored after early stopping.

Parameter	Value
Character embedding size	192
Maximum sequence length	128
Transformer layers	3
Attention heads	4
Dimension per head	48
Feed-forward size	768

Appendix

Table 13: Shared architecture of the Transformer–CRF variants. The two external-feature projections are included only in the corresponding fusion models.

Char2Vec parameter	Value
Training objective	Skip-gram
Vector size	192
Context window	5
Negative samples	10
Minimum frequency	1
Training epochs	30

Appendix

Table 14: Training parameters of the static Char2Vec initialization.

MLM parameter	Value
Masking ratio	0.15
Span-masking probability	0.50
Span length	2–4 characters
Replacement strategy	80/10/10
Optimizer	AdamW
Learning rate	$3\times 10^{-4}$

Appendix

Table 15: Masked-language-model pretraining parameters of TangutEncoder. Pretraining completed the full 5,000 steps, and the checkpoint with the lowest validation loss was retained.

Fine-tuning parameter	Value
Frozen-encoder epochs	3
Optimizer	AdamW
Encoder learning rate	$5\times 10^{-5}$
Task-head learning rate	$5\times 10^{-4}$
Weight decay	0.01
Batch size	32 sentences

Appendix

Table 16: Downstream training parameters shared by Transformer–Random, Transformer–Char2Vec and TangutEncoder.

Association features	P	R	F 1	OOV-R	IV-R
Freq + dPMI	$0.906{\pm}0.003$	$0.905{\pm}0.003$	$0.905{\pm}0.003$	$0.486{\pm}0.009$	$0.945{\pm}0.001$
Freq + dPMI + Ent	$0.907{\pm}0.004$	$0.906{\pm}0.004$	$0.907{\pm}0.004$	$0.491{\pm}0.016$	$0.947{\pm}0.002$
Freq + Dice	$0.905{\pm}0.003$	$0.905{\pm}0.004$	$0.905{\pm}0.003$	$0.485{\pm}0.010$	$0.945{\pm}0.002$
Freq + Dice + Ent	$0.907{\pm}0.004$	$0.906{\pm}0.004$	$0.906{\pm}0.004$	$0.489{\pm}0.016$	$0.946{\pm}0.002$
Freq + $t$ -score	$0.905{\pm}0.004$	$0.904{\pm}0.004$	$0.904{\pm}0.004$	$0.483{\pm}0.014$	$0.944{\pm}0.003$
Freq + $t$ -score + Ent	$0.907{\pm}0.003$	$0.906{\pm}0.004$	$0.907{\pm}0.004$	$0.490{\pm}0.013$	$0.946{\pm}0.002$

Appendix

Table 17: Comparison of association measures under five-fold cross-validation.

Model	P	R	F 1	OOV-R	IV-R
Dict-corpus	.845	.890	.867	.219	.954
Dict-dictionary	.787	.728	.756	.666	.733
Dict-all	.821	.740	.779	.665	.748

Appendix

Table 18: Bidirectional maximum matching with different dictionary sources. Values are five-fold means.

Model	External features	P	R	F 1	OOV-R	IV-R
BiLSTM	0	$0.862{\pm}0.008$	$0.873{\pm}0.008$	$0.868{\pm}0.008$	$0.472{\pm}0.014$	$0.912{\pm}0.006$
BiLSTM+BIE	11	$0.885{\pm}0.005$	$0.894{\pm}0.004$	$0.889{\pm}0.004$	$0.546{\pm}0.011$	$0.927{\pm}0.004$
BiLSTM+BIE+R	14	$0.891{\pm}0.003$	$0.898{\pm}0.004$	$0.894{\pm}0.004$	$0.546{\pm}0.009$	$0.931{\pm}0.003$
BiLSTM+ $\mathrm{Dict}_{\mathrm{all}}$	17	$0.893{\pm}0.003$	$0.900{\pm}0.003$	$0.897{\pm}0.003$	$0.551{\pm}0.011$	$0.933{\pm}0.003$
+ $\mathrm{Dict}_{\mathrm{all}}$ + IL	28	$0.896{\pm}0.003$	$0.900{\pm}0.002$	$0.898{\pm}0.002$	$0.557{\pm}0.009$	$0.933{\pm}0.002$
+ $\mathrm{Dict}_{\mathrm{all}}$ + IL + DD	30	$0.897{\pm}0.004$	$0.904{\pm}0.003$	$0.900{\pm}0.003$	$0.554{\pm}0.006$	$0.938{\pm}0.002$

Appendix

Table 19: BiLSTM–CRF variants with progressively richer external features.

	Secular		Religious
Model	F 1	OOV-R	F 1	OOV-R
BiLSTM	$0.875{\pm}0.007$	$0.498{\pm}0.016$	$0.810{\pm}0.021$	$0.244{\pm}0.028$
BiLSTM+BIE	$0.897{\pm}0.005$	$0.573{\pm}0.015$	$0.830{\pm}0.018$	$0.305{\pm}0.043$
BiLSTM+BIE+R	$0.903{\pm}0.004$	$0.574{\pm}0.014$	$0.829{\pm}0.013$	$0.286{\pm}0.053$
BiLSTM+ $\mathrm{Dict}_{\mathrm{all}}$	$0.905{\pm}0.003$	$0.579{\pm}0.014$	$0.835{\pm}0.016$	$0.298{\pm}0.037$
+ $\mathrm{Dict}_{\mathrm{all}}$ + IL	$0.905{\pm}0.002$	$0.586{\pm}0.011$	$0.842{\pm}0.018$	$0.300{\pm}0.043$

Appendix

Table 20: Genre-level results for the BiLSTM–CRF feature variants.

Pretraining	F 1	OOV-R	IV-R
MLM	$0.911$	$0.608$	$0.946$
MLM + WordRank	$0.912$	$0.617$	$0.946$

Appendix

Table 21: Downstream results of lexicon-aware continued pretraining. Both encoders are evaluated with the same dictionary and distributional features.

Overall (479 segments)
Model	P	R	F 1	POS Acc.	POS $N$
CRF	0.6410	0.7381	0.6861	–	–
CRF + $\mathrm{Dict}_{\mathrm{all}}$	0.6850	0.7711	0.7255	–	–
CRF + $\mathrm{Dict}_{\mathrm{all}}$ + $\mathrm{Dist}_{\mathrm{all}}$	0.7114	0.7863	0.7470	–	–
Transformer-Random	0.5915	0.7008	0.6415	–	–
Transformer-Char2Vec	0.6015	0.7121	0.6522	–	–

Appendix

Table 22: Full segmentation and POS-tagging results on five unseen works. POS accuracy is calculated only over words whose predicted boundaries exactly match the gold segmentation; POS

N

denotes the number of such words.

Panel of Figure 3 — Figure 3: The Tangut Digital Intelligence Platform.

Explore similar work

Neural Machine Translation for Low-Resource Tangkhul--English
Jun 24, 2026Chormi Zimik Vashai, Agniva MaitiNeural Machine Translation Low-Resource Languages
Independent Researcher · KIIT University
From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages
Jul 27, 2026Th{é}otime de la SelleHandwritten Text Recognition Transcription
ISC, HiSoMA, CNRS · Institut des Sources Chrétiennes, HiSoMA, CNRS, Lyon, France
Towards a Joint Khmer Text Recognition and Word Segmentation
Aug 31, 2026Marry Kong, Rina Buoy, Sovisal Chenda +3Document Segmentation Handwritten Text Recognition
Techo Startup Center, Ministry of Economy and Finance, Phnom Penh, Cambodia · Osaka Metropolitan University, Osaka, Japan

Jun 24, 2026, cs.CLJ/K move · Enter open · S save

Neural Machine Translation for Low-Resource Tangkhul--English

Chormi Zimik Vashai, Agniva Maiti

Independent Researcher · KIIT University

Neural Machine Translation Low-Resource Languages Natural Language Processing Text Corpora

Open paper AskPDF arXiv

We present a study on low-resource machine translation for the Tangkhul-English (nmf-en) language pair. Tangkhul is a severely under-resourced Tibeto-Burman language spoken primarily in Manipur, India, with virtually no prior natural language processing infrastructure. We describe two systems: (1) a primary system based on ByT5-large fine-tuned on 38,336 Tangkhul-English parallel sentence pairs, and (2) a contrastive system based on mT5-small fine-tuned on the same corpus. Our primary ByT5-large system achieves a corpus BLEU score of 39.97, chrF++ of 58.07, BERTScore F1 of 0.8104, and COMET (wmt22-comet-da) of 0.7302 on a held-out test set of 3,856 sentences. We further discuss the orthographic challenges specific to Tangkhul's Latin-script diacritics, the domain bias of our training corpus (which comprises biblical text, stories, and conversational data), and avenues for future improvement through data diversification and domain adaptation.

Abstract

Explore similar work

Neural Machine Translation for Low-Resource Tangkhul--English

From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages

Towards a Joint Khmer Text Recognition and Word Segmentation