Poetry generation in Sanskrit typically requires the verse to be semantically coherent and adhere to strict prosodic rules. In Sanskrit prosody, every line of a verse is typically a fixed length sequence of syllables adhering to prescribed binary patterns of syllable weights. We observe that instead of treating a verse as a monolithic sequence, segmenting them as grouped-lines leads to significant improvement in semantic coherence by 10% with comparable metrical adherence. Specifically, Pingala, our proposed decoding approach is designed to encourage every line to have well-formed words and our token selection biases the model towards it by preferring longer tokens. Writing in Sanskrit follows phonemic orthography, hence using a phonetically aware transliteration scheme, SLP1, increased the metrical alignment by 46% with comparable semantic similarity, for a instruction fine-tuned large language models like Phi-4. We also introduce a new approach for reference-free evaluation using cross-encoders which achieved better alignment with true poetry instances.
Figures & tables
Eval Metric
Ground Truth
Pingala
CD
Semantic Similarity ∗
73.46
69.05
64.96
Human Ratings
-
2.70
2.13
Table 1: Comparison of semantic similarity scores and human evaluation ratings( on standard constrained decoding(CD) and Pingala outputs. The bi-encoder assigns similar scores to translations with vastly different human-perceived quality. ∗ as used in \ttfamily Chandomitra. Human ratings for Pingala is average of the ratings specified in Table 1 while for the CD is from \ttfamily Chandomitra.
Approach
Positive
Negative(Avg.)
Spread
bge-m3
73.46%
50.97%
22%
cross
90.02%
1.67%
88.33%
Table 2: Semantic similarity scores based on the cross-encoder
Mean Acc.
Batch Size
bge-m3
cross-encoder
Full
46.78%
74.74%
16
82.35%
84.71%
4
91.53%
92.87%
Table 3: Mean Accuracy: bi-encoder model( \ttfamily Chandomitra) vs proposed cross-encoder model
Model
Raw
Isotonic Calibrated
Pearson
Spearman
Pearson
Spearman
Bi-Encoder
0.6075
0.6094
0.6559
0.6459
Cross-Encoder
0.6767
0.5833
0.7248
0.7175
Table 4: Correlation between human semantic ratings (N=200) and automatic scorers.
Figure 1: An infographic comparison of working of Pingala vs CD from \ttfamily Chandomitra.
Model
FT
CD
Pingala
Full %
Partial %
Sim
Raw Translation Models
NLLB-dist-1.3B ∗
-
✓
-
82.97
90
70.75
NLLB-dist-1.3B
-
-
✓
0.14
0.25
35.62
NLLB-dist-1.3B ∗
✓
✓
-
99.09
99.1
67.99
NLLB-dist-1.3B
✓
✓
-
98.8
99.09
69.01
NLLB-dist-1.3B
✓
-
✓
100
100
78.89
Table 5: Evaluation on the Chandomitra test set. “FT” denotes fine-tuning on the \ttfamily Chandomitratraining split; “CD” denotes standard constrained decoding as used in \ttfamily Chandomitra; “ \ttfamily Pingala” denotes our custom-constrained decoding framework. “Full %” is fraction of outputs fully conformant to Anuṣṭubh; “Partial %” is fraction partially conformant. “Sim” is automatic semantic similarity (higher is better). ∗ same settings as in \ttfamily Chandomitra
Model
Full %
Partial %
Sim
NLLB-dist-1.3B
100
100
69.76
Phi-4-14B
61.54
74.23
74.62
Phi-4-14B(SLP1)
92.31
95.85
67.87
Table 6: Evaluation on the OOD set of \ttfamily Chandomitra. For the models that has been fine-tuned on training split of \ttfamily Chandomitra, and using Pingala. “Full %” is fraction of outputs fully conformant to Anuṣṭubh; “Partial %” is fraction partially conformant. “Sim” is automatic semantic similarity (higher is better).
Quality
NLLB
Phi-4
Phi-4(SLP1)
Excellent ( ≥0.8 )
970
912
960
Good (0.6–0.8)
144
166
166
Fair (0.4–0.6)
95
89
111
Poor (0.2–0.4)
72
93
54
Very poor ( < 0.2)
140
161
130
Table 7: Semantic similarity distribution for NLLB, Phi-4, and Phi-4(SLP1) outputs. Dataset size: N=1421 .
Table 10: Ablation results for constraint weight β .
Figure 2: Impact of constraint weight β on syntactic and semantic quality. Semantic quality peaks at β=5 , illustrating the optimal balance between structural enforcement and meaning preservation.
na ca tasya mahAtmAnaH kriyatAmitaraM kriyam . mama duzkftakAlasya jIvitA kiM Bavizyati ..
Appendix
Table 11: Sample of generated Sanskrit poetry for the input "I could not even perform the last rites of that magnanimous one who died of grief on account of me. What use is my life with an unlucky birth?"
Table 13: Mean Accuracy: comparison of cross-encoder model trained on Devanagari vs SLP1
Model
Raw
Isotonic Calibrated
Pearson
Spearman
Pearson
Spearman
Cross-Encoder
0.6767
0.5833
0.7248
0.7175
Cross-Encoder
0.542
0.598
0.679
0.653
Appendix
Table 14: Comparison of correlation with human semantic ratings (N=200) between cross-encoder trained using Sanskrit Devanagari vs SLP1 transliteration scheme