khammoon: From a Linguistic Guideline to a Thai Morpheme Segmenter
Organizations: Typhoon, SCB DataX · AI Singapore
Abstract
Thai is written without spaces between words, making segmentation a foundational step in most Thai NLP pipelines. Existing tools primarily reproduce the word-level conventions of corpora such as BEST; however, because the notion of a "word" in Thai lacks a unified linguistic consensus, these conventions struggle with productive compounding, leading to arbitrary boundaries and ambiguity in downstream representations. Instead, we propose operating at the morphological level, focusing on khammoon, the smallest atomic unit that carries meaning. Segmenting at the morphemic level provides an explicit, consistent morphological representation that eliminates compounding ambiguity. Crucially, this representation benefits downstream learning: models trained on morphemes perform on par with or exceed word-level baselines across part-of-speech tagging, named-entity recognition, and sentiment classification, demonstrating that linguistic consistency improves representation quality without downstream performance penalty. To operationalize this representation, we translate linguistic principles into a formal decision guideline with near-perfect human agreement (), scale annotation with an LLM, and correct systematic errors with human labels. Finally, we train khammoon, a lightweight 1M-parameter convolutional BiLSTM segmenter designed for efficient CPU inference, achieving a boundary F1 of 0.992 on out-of-domain text. The guideline, datasets, model, and library are all publicly released.
Figures & tables
| Reference | \textthai กรรมวิธี — ที่ — ใช้ — แยก — ก๊าซ — ออก — จาก — น้ำ — มัน — ดิบ |
|---|---|
| PyThaiNLP newmm | \textthai กรรมวิธี — ที่ — ใช้ — แยก — ก๊าซ — ออกจาก — น้ำมันดิบ |
| AttaCut | \textthai กรรมวิธี — ที่ — ใช้ — แยก — ก๊าซ — ออก — จาก — น้ำมัน — ดิบ |
| LLM segmenter | \textthai กรรม — วิธี — ที่ — ใช้ — แยก — ก๊าซ — ออก — จาก — น้ำ — มัน — ดิบ |
| khammoon | \textthai กรรมวิธี — ที่ — ใช้ — แยก — ก๊าซ — ออก — จาก — น้ำ — มัน — ดิบ |
| Category | Rule | Keep | Split |
|---|---|---|---|
| 1. Mixed origin | Parts from different sources (Thai, Pali–Sanskrit, Khmer, foreign) are split. | — | \textthai ผล—ไม้, \textthai ราช—วัง, \textthai พล—เมือง |
| 2. All Pali–Sanskrit | Keep when the meaning is read back to front, a true \textthai สมาส or \textthai สนธิ; split when the parts are in Thai order or form a couplet. | \textthai ประวัติศาสตร์, \textthai พุทธศาสนา | \textthai ศาสนา—พุทธ, \textthai ประชา—ชน |
| 3. Loanwords | Keep when read as in the source language, with the head last; split when the parts follow Thai order, with the head first. | \textthai วิปครีม, \textthai คอนกรีตบล็อก | \textthai คุกกี้—ช็อกโกแลต |
| 4. Native Thai | A: keep sound-reduced forms. B: split meaningless affix syllables, except the fused prefixes \textthai ระ-, \textthai กระ-, \textthai ประ-. C: keep the base form of an onomatopoeia, split explicit doubling. D: otherwise apply the substitution and insertion tests. | \textthai ตะวัน, \textthai กระโดด, \textthai คิกคัก, \textthai นาฬิกา | \textthai ตา—เตอ, \textthai คิก—คิก—คัก—คัก, \textthai รถ—ไฟ, \textthai มือ—ถือ |
| 5. Unknown or still uncertain | Pronunciation test: keep when a part changes its pronunciation in the combination, split when it does not. | \textthai อนึ่ง | \textthai ขะมัก—เขม้น, \textthai พระ—สงฆ์ |