Pingala: Prosody-Aware Decoding for Sanskrit Poetry Generation
Organizations: Indian Institute of Technology, Kharagpur · Oriflow
Abstract
Poetry generation in Sanskrit typically requires the verse to be semantically coherent and adhere to strict prosodic rules. In Sanskrit prosody, every line of a verse is typically a fixed length sequence of syllables adhering to prescribed binary patterns of syllable weights. We observe that instead of treating a verse as a monolithic sequence, segmenting them as grouped-lines leads to significant improvement in semantic coherence by 10% with comparable metrical adherence. Specifically, Pingala, our proposed decoding approach is designed to encourage every line to have well-formed words and our token selection biases the model towards it by preferring longer tokens. Writing in Sanskrit follows phonemic orthography, hence using a phonetically aware transliteration scheme, SLP1, increased the metrical alignment by 46% with comparable semantic similarity, for a instruction fine-tuned large language models like Phi-4. We also introduce a new approach for reference-free evaluation using cross-encoders which achieved better alignment with true poetry instances.
Figures & tables
| Eval Metric | Ground Truth | Pingala | CD |
|---|---|---|---|
| Semantic Similarity ∗ | 73.46 | 69.05 | 64.96 |
| Human Ratings | - | 2.70 | 2.13 |
| Approach | Positive | Negative(Avg.) | Spread |
|---|---|---|---|
| bge-m3 | 73.46% | 50.97% | 22% |
| cross | 90.02% | 1.67% | 88.33% |
| Mean Acc. | ||
|---|---|---|
| Batch Size | bge-m3 | cross-encoder |
| Full | 46.78% | 74.74% |
| 16 | 82.35% | 84.71% |
| 4 | 91.53% | 92.87% |
| Model | Raw | Isotonic Calibrated | ||
|---|---|---|---|---|
| Pearson | Spearman | Pearson | Spearman | |
| Bi-Encoder | 0.6075 | 0.6094 | 0.6559 | 0.6459 |
| Cross-Encoder | 0.6767 | 0.5833 | 0.7248 | 0.7175 |
| Model | FT | CD | Pingala | Full % | Partial % | Sim |
|---|---|---|---|---|---|---|
| Raw Translation Models | ||||||
| NLLB-dist-1.3B ∗ | - | ✓ | - | 82.97 | 90 | 70.75 |
| NLLB-dist-1.3B | - | - | ✓ | 0.14 | 0.25 | 35.62 |
| NLLB-dist-1.3B ∗ | ✓ | ✓ | - | 99.09 | 99.1 | 67.99 |
| NLLB-dist-1.3B | ✓ | ✓ | - | 98.8 | 99.09 | 69.01 |
| NLLB-dist-1.3B | ✓ | - | ✓ | 100 | 100 | 78.89 |
| Model | Full % | Partial % | Sim |
|---|---|---|---|
| NLLB-dist-1.3B | 100 | 100 | 69.76 |
| Phi-4-14B | 61.54 | 74.23 | 74.62 |
| Phi-4-14B(SLP1) | 92.31 | 95.85 | 67.87 |
| Quality | NLLB | Phi-4 | Phi-4(SLP1) |
|---|---|---|---|
| Excellent ( ) | 970 | 912 | 960 |
| Good (0.6–0.8) | 144 | 166 | 166 |
| Fair (0.4–0.6) | 95 | 89 | 111 |
| Poor (0.2–0.4) | 72 | 93 | 54 |
| Very poor ( 0.2) | 140 | 161 | 130 |
| Syllable Deviation Category | NLLB | Phi-4 | Phi-4(SLP1) |
|---|---|---|---|
| Perfect (32) | 1421 | 1105 | 1397 |
| Near-perfect (28–36) | 0 | 206 | 23 |
| Moderate (24–40) | 0 | 104 | 0 |
| Poor | 0 | 6 | 1 |
| Annotator | Semantic Coherence |
|---|---|
| A1 | 3.08 |
| A2 | 2.32 |
| A3 | 2.5 |
| A4 | 1.96 |
| A5 | 2.28 |
| Krippendorff’s alpha | 0.059 |
| Syntactic (%) | Semantic (%) | |
|---|---|---|
| 0 | 92.61 | 64.96 |
| 1 | 100.00 | 77.06 |
| 3 | 100.00 | 77.56 |
| 5 | 100.00 | 78.89 |
| 7 | 99.37 | 73.84 |
| 10 | 92.75 | 61.42 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Generated Sanskrit Poetry |
|---|---|
| NLLB-dist-1.3B | \devanagarifont तस्याहं न सकामां कृत्स् संस्काराणां महायशाः ।दुःखोपजीवितायास्त्वत् कालयोगेन केनचिन् ॥ |
| Phi-4-14B | \devanagarifont अक्षमश्च मया कार्षी श्शोकपरा महायशाः । कृतं कर्म महत्कार्यं किं पुनर्जन्म जीवित ।। |
| Phi-4-14B(SLP1) | na ca tasya mahAtmAnaH kriyatAmitaraM kriyam . mama duzkftakAlasya jIvitA kiM Bavizyati .. |
| Approach | Positive | Negative(Avg.) | Spread |
|---|---|---|---|
| Devanagri | 90.02% | 1.67% | 88.33% |
| SLP1 | 95.81% | 6.07% | 89.74% |
| Mean Acc. | ||
|---|---|---|
| Batch Size | Devanagari | SLP1 |
| Full | 74.74% | 77.07% |
| 16 | 84.71% | 78.91% |
| 4 | 92.87% | 90.07% |
| Model | Raw | Isotonic Calibrated | ||
|---|---|---|---|---|
| Pearson | Spearman | Pearson | Spearman | |
| Cross-Encoder | 0.6767 | 0.5833 | 0.7248 | 0.7175 |
| Cross-Encoder | 0.542 | 0.598 | 0.679 | 0.653 |
Text Generation
Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.
See all 421 papers