A Lattice-Based HMM Approach to Aruz Scansion of Ottoman Poetry
Organizations: Johns Hopkins University
Abstract
Ottoman Divan poetry is composed in aruz, a quantitative meter based on the weights of syllables. The meter is sensitive to specific characters recorded in the original writing in the Arabic script. This work is concerned with automatic meter finding and scansion alignment for Ottoman Divan poetry, especially on modern Turkish spelling without the special diacritics that scholarly transliterations of Ottoman preserve. We define two tasks on this input, first identifying the meter used in the poem, and second aligning the meter's weights with the syllabification of the poem, identifying the flaws and freedoms the poet has taken along the way. We build a corpus of 52,000 lines split by poet, with scansion annotations for 500 test lines, and a robustness benchmark generated at the deviation rates measured on real web and OCR copies. Our model is a hidden Markov model over the positions of each meter, decoded over a lattice of every candidate path, and trained by EM from the printed meter, without scansion labels. On a held-out set it scans 77.6% of the test lines fully correctly, against 26.8% for the available rule-based scanner, 55.0% for a neural tagger trained with the same supervision and 63.6% for Claude Opus 5.5 few-shot. The code is open-sourced and the tool is released on the web.
Figures & tables
| input | Ey | ki | düş | dün | Mih | ri | her | bir | bi | ve | fa | nun | pa | yı | na |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| read | Ey | ki | düş | düñ | Mih | ri | her | bir | bī | ve | fā | nuñ | pā | yı | na |
| pos. | – | – | – | – | – | – | – | – | – | – | – | ||||
| flaw | zihaf | ||||||||||||||
| input | Sen | da | hi | bi | a | ri | miş | sin | böy | le | bil | mez | düm | se | ni |
| read | Sen | da | ḫī | bī | ʿā‿ | ri | miş | sin | böy | le | bil | mez | düm | se | ni |
| pos. | – | – | – | – | – | – | – | – | – | – | – |
| split | poets (century) | poems | lines |
|---|---|---|---|
| train | Bahtî (17), Sürûrî (16), Dürrî (18), Esrâr (18), Nâfiz (18), Hâmî (18) | 1,644 | 18,282 |
| dev | Tâcîzâde (15), Yahyâ (16), Şehrî (17), Vehbî (18) | 1,133 | 18,090 |
| test | Fuzûlî (16), Mihrî Hatun (15), Murâdî (16) | 1,731 | 15,992 |
| Task 1: meter | Task 2: scansion | ||||
|---|---|---|---|---|---|
| system | poem | line | full | flaw F1 | syll. F1 |
| syllable count | 56.6 | 57.2 | – | – | – |
| BiLSTM tagger (5 seeds) | 97.8 | 92.0 | 55.0 | 80.4 | 96.0 |
| Kurt & Kara, plain letters | 91.0 | 71.4 | 2.4 | 42.0 | 85.8 |
| restorer + Kurt & Kara | 98.9 | 84.1 | 26.8 | 54.9 | 90.5 |
| ours : HMM + restorer | 99.9 | 98.2 | 77.6 | 89.6 | 98.0 |
| Task 1: meter | Task 2: scansion | ||||
| ablation | poem | line | full | flaw F1 | syll. F1 |
| full model | 99.9 | 98.2 | 77.6 | 89.6 | 98.0 |
| no HMM (aligner) | 99.9 | 96.3 | 33.4 | 56.6 | 95.7 |
| no EM | 96.8 | 85.0 | 59.4 | 77.1 | 94.7 |
| no O/K symbols | 99.9 | 97.4 | 35.2 | 64.7 | 91.0 |
| flat candidate probs. | 99.7 | 95.1 | 28.4 | 53.6 | 94.5 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| meter, as the editions print it | template(s) | train poems % | train lines % | |
|---|---|---|---|---|
| fāʿilātün fāʿilātün fāʿilātün fāʿilün | x – – – – – – – – – – | 1 | 31.4 | 30.1 |
| mefāʿīlün mefāʿīlün mefāʿīlün mefāʿīlün | – – – – – – – – – – – – | 1 | 19.8 | 18.7 |
| mefʿūlü fāʿilātü mefāʿīlü fāʿilün | – – – – – – – – | 1 | 13.0 | 11.5 |
| mefʿūlü mefāʿīlü mefāʿīlü feʿūlün | – – – – – – – – | 1 | 11.1 | 8.7 |
| feʿilātün feʿilātün feʿilātün feʿilün | x – – – – – – – – , x – – – – – – – | 2 | 9.7 | 14.0 |
| mefāʿilün feʿilātün mefāʿilün feʿilün | – – – – – – – – , – – – – – – – | 2 | 6.1 | 5.0 |
| position kind | L:tr | S:tr | H:tr | O:tr | K:tr | L:ar | S:ar | H:ar | O:ar | K:ar |
|---|---|---|---|---|---|---|---|---|---|---|
| long | .362 | .141 | .429 | .066 | .002 | |||||
| short | .571 | .006 | .005 | .383 | .035 | |||||
| free | .434 | .157 | 0 | .352 | .056 | 0 | .001 | |||
| insert | .014 | .071 | .014 | .014 | .014 | .026 | .579 | .240 | .014 | .014 |
| deviation (after the alphabet fold) | anthology | OCR edition | OCR popular book | changes weights | example |
|---|---|---|---|---|---|
| vowel changed in the ending | 4.08 | 9.65 | 9.33 | 0 | istersen istersin |
| Turkish letter lost or swapped | 1.73 | 5.63 | 1.72 | 0 | zulmı zulmi |
| letter edit, non-word | 1.55 | 3.75 | 5.16 | 28 | çekmekden çekmekten |
| letter edit, another word | 2.78 | 2.86 | 3.38 | 17 | dideyi didei |
| vowel changed | 1.33 | 2.57 | 2.84 | 1 | ıkdı akdi |
| vowel changed in the stem | 2.16 | 1.34 | 2.37 | 0 | etseler itseler |
| lines | dev./w. | letters | syll. | weights | CER | |
|---|---|---|---|---|---|---|
| web copies, measured | 512 | 20.6 | 77.3 | 6.8 | 18.6 | 4.8 |
| generated, 1 | 14,516 | 19.5 | 72.1 | 5.1 | 18.1 | 5.2 |
| generated, 2 | 13,018 | 25.6 | 81.7 | 9.2 | 30.6 | 7.7 |
| OCR’d books, measured | 13,532 | 28.9 | 88.6 | 10.3 | 23.4 | 6.3 |
| generated, 1 | 15,992 | 27.6 | 85.8 | 13.6 | 25.0 | 7.3 |
| generated, 2 | 15,992 | 42.2 | 95.8 | 24.0 | 42.9 | 12.4 |
| input | faulty lines | AUROC | precision | recall | alarms, scansion kept | top-1 word | kind | word offered |
|---|---|---|---|---|---|---|---|---|
| web | 618/3438 | 0.714 | 51.3 | 42.4 | 10.1 | 57.0 | 83.9 | 3.4 |
| web x2 | 888/3038 | 0.724 | 62.3 | 47.0 | 13.5 | 53.2 | 83.9 | 4.2 |
| web forms | 0/3742 | – | – | – | 10.0 | – | – | – |
| web damage | 632/3742 | 0.711 | 57.9 | 40.5 | – | 60.7 | 90.8 | 4.4 |
| web clutter | 0/3742 | – | – | – | – | – | – | – |
| web structure | 0/3428 | – | – | – | – | – | – | – |
Computational Literary Studies
Computational Literary Studies uses data analysis and language technologies to examine literary style, themes, narratives, influence, and change.
See all 53 papers