Classical Sanskrit poses three challenges unresolved by single-call monolithic models: morphophonemic fusion Sandhi, dense nominal compounding Samāsa, and discourse-level zero-anaphora forced by poetic meter. We present MITRA-Swarm, a multi-agent architecture injecting explicit linguistic structure into an otherwise monolithic pipeline: a Sandhi-splitting agent, a parallel array of generalist gemma-4-26B-A4B-it and domain-specialized gemma-2-mitra-it translators, and a discourse-aware Ellipsis Restorer that consults corpus-derived register tags to avoid inventing active-voice subjects in impersonal legal and philosophical registers. Two separated judges evaluate quality: a routing judge (Jroute, same model family as the generators) driving an internal critique loop, and an independent reporting judge (Jreport=gpt-4o-mini, no shared vendor lineage) scoring every output exactly once. On a stratified benchmark of 1,000 verses across five registers (23,990 scored verdicts, 12-condition pre-registered ablation), the full swarm reaches 85.90 MQM against a monolithic baseline of 85.27 (Δ=+0.63, padj=0.0018, 95% CI [+0.30,+0.97]), and isolated discourse restoration alone accounts for +0.44 MQM (padj=0.0288), outperforming a call-count-matched self-refinement baseline by +0.51 MQM (padj=0.0288). Severe per-verse failures (MQM<80) fall from 9.0% to 7.6%, at roughly 9× the per-verse inference cost of the baseline. Additionally, Jroute scores its own family's outputs 2-7 MQM higher than Jreport on every one of the 12 conditions, empirically motivating the two-judge split; and a preflight screen of a 192,151-entry retrieval index against all 12 evaluation works verifies 0 reference leaks across 269 runtime checks. These are small gains on a metric saturating near 85-86.
Figures & tables
Figure 1: System architecture of the MITRA-Swarm pipeline. Jroute (same model family as the generators) drives the internal critique loop; Jreport (independent vendor, gpt-4o-mini ) scores the finished output exactly once, off the critical path, and is the only judge whose scores are reported as results (Section 5.2 ).
Register
Description
Pop.
Sample
Voice-preserv.
epic
Itihāsa/Purāṇa narrative
1,720
310
–
sutra
Philosophical sūtra, commentary
1,488
268
✓
vedic
Vedic saṃhitā/upaniṣad; archaic
1,000
180
–
kavya
Ornate metrical poetry
844
152
–
legal
Dharmaśāstra, nibandha
500
90
✓
Total
5,552
1,000
Table 1: Register composition of the N=1,000 evaluation sample, proportional to its population in the 5,552-verse test partition. Voice-preserving marks registers routed to the impersonal-voice guard in the Ellipsis Restorer (Section 3.4 ).
Configuration
Prompt tok.
Compl. tok.
Total/verse
Latency (s)
baseline
91
49
140
2.5
self_refine
164
89
253
4.9
bo_n
455
258
713
12.8
full_swarm
828
462
1290
7.1
Table 2: Per-verse compute, key conditions (measured, not estimated). Full swarm costs 9.2× baseline tokens for +0.63 MQM.
Configuration
N
MQM
SD
P10
P25
Min
% < 80
baseline
1000
85.27
5.67
80.0
84.0
35
9.0
baseline_dup
1000
85.27
5.53
80.0
84.0
35
8.5
placebo_rejudge
1000
85.26
5.71
80.0
84.0
35
8.9
bo_n
1000
85.58
5.51
80.0
84.0
40
8.7
self_refine
1000
85.39
5.56
80.0
84.0
35
8.3
guide_only
1000
85.23
5.65
80.0
84.0
40
8.9
Table 3: Ablation results on the N=1,000 stratified MitraSamgraha benchmark. MQM from Jreport ( gpt-4o-mini ); P10/P25 are the 10th/25th percentiles; % <80 is the severe-failure rate.
Contrast
N
Δ MQM
95% CI
padj
dz
W/L/T
Full swarm vs monolithic baseline
999
+0.63 ∗
[+0.30, +0.97]
0.002
+0.12
397/294/308
Re-judging alone (placebo)
1000
-0.01 †
[-0.07, +0.01]
1.000
-0.02
2/2/996
Full swarm vs best-of-N (compute matched)
999
+0.32
[-0.05, +0.67]
0.460
+0.06
363/333/303
Full swarm vs self-refine (specialisation)
999
+0.51 ∗
[+0.16, +0.85]
0.029
+0.09
389/301/309
Morphosemantic guide
999
-0.01 †
[-0.27, +0.23]
1.000
-0.00
170/179/650
Ellipsis restoration
999
+0.44 ∗
[+0.12, +0.73]
0.029
+0.09
323/232/444
Table 4: Pre-registered contrasts, paired approximate randomization test, R=10,000 , Holm–Bonferroni adjusted over the family of 10. ∗ significant and exceeds the 0.21 MQM noise floor. † below the noise floor (not distinguishable from run-to-run variation regardless of nominal p -value).
Judge diagnostic
Value
Reporting judge
openai:gpt-4o-mini
Judge failure rate
0.41%
Arithmetic disagrees with rubric
96.4% (mean ∣Δ∣ = 16.37)
Test–retest SD (repeated calls)
1.45
Self-preference range (min–max, 12 conditions)
+1.94 to +6.79
Table 5: Judge validation summary under the zero-shot judge prompt (the few-shot comparison above is a separate, n=100 -per-condition subsample). Self-preference range is Jroute−Jreport over the 12 conditions.
Tool
Calls
Hit rate
Errors
Latency (ms)
TRANSLATION_MEMORY
5513
0.05
0
513
Table 6: Tool utility, aggregated over all conditions in which the tool inventory was active.
Table 9: Full 12-condition register breakdown, MQM by register and configuration ( Jreport ); the condensed discussion in Section 5 is derived from this.
Configuration
Prompt tok.
Compl. tok.
Total/verse
Latency (s)
baseline
91
49
140
2.5
baseline_dup
91
48
140
2.4
bo_n
455
258
713
12.8
full_swarm
828
462
1290
7.1
guide_ellipsis
738
430
1169
5.1
guide_only
482
389
871
2.5
Appendix
Table 10: Full per-verse token budget and latency by configuration (measured); Table 2 in the main text shows the three key conditions. Referenced from Section 4.4.
Configuration
addition
mistranslation
omission
untranslated
awkward
grammar
baseline
0.07
1.62
1.35
0.02
0.02
0.13
baseline_dup
0.07
1.62
1.36
0.02
0.02
0.13
bo_n
0.07
1.63
1.37
0.01
0.02
0.13
full_swarm
0.06
1.68
1.31
0.01
0.02
0.13
guide_ellipsis
0.06
1.69
1.31
0.01
0.02
0.13
guide_only
0.06
1.66
1.32
0.02
0.02
0.15
Appendix
Table 11: Errors per segment by MQM category, Jreport . Referenced from Section 5 .
Table 12: Full per-condition judge-validation diagnostics; self-preference is Jroute−Jreport . Table 5 in the main text summarizes this. Referenced from Section 5.2 .
Figure 2: Tail of the per-verse MQM distribution, full swarm vs. monolithic baseline. Referenced from Section 5 .
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
We introduce grounded glossary generation, a structured task requiring models to recover semantically meaningful Sanskrit phrases and produce translation-grounded meanings from a sloka-translation pair, formalizing the traditional patha commentary practice as an evaluable NLP objective. We construct a benchmark of 31,316 sloka-translation-glossary triples from the Valmiki Ramayana and Srimad Bhagavatam, paired with two metrics: Jaccard for phrase recovery and Meaning Faithfulness for semantic consistency. Across zero-shot, few-shot, and instruction fine-tuned variants of Gemma-3n-E4B, Gemma-3-12B, Phi-4, and Qwen3.5-9B, instruction fine-tuning substantially outperforms prompting, while explicit segmentation yields gains. Error analysis identifies over-segmentation of sandhi and samasa compounds as the dominant failure mode, pointing to morphological modeling as the key bottleneck for faithful Sanskrit lexical decomposition.
Manoj Balaji Jagadeeshan, Sai Pragnaan Marala, Pawan Goyal
Digital humanities projects increasingly rely on machine translation and large language models to widen access to classical, religious, and otherwise under-translated textual traditions. Yet standard translation benchmarks are poorly suited to such materials: they typically compare a system output against a single reference translation, even though classical texts often support multiple faithful renderings that differ in terminology, register, and interpretation. This article introduces PaliBench, both a benchmark for Pali-to-English translation and a reusable method for constructing multi-reference translation benchmarks for classical languages. The Pali case study draws on passages from the Sutta Pitaka aligned with independent English translations by Bhikkhu Sujato, Bhikkhu Thanissaro, and Bhikkhu Bodhi. The workflow combines LLM-assisted alignment of independently segmented translations, automated verification against source files, passage-level quality filtering, deduplication of formulaic repetitions, and multi-metric evaluation against multiple human references. The resulting benchmark contains 1,700 passages spanning 8,389 segments and approximately 345,000 tokens. We use it to evaluate ten contemporary large language models with complementary metrics, finding strong cross-metric concordance in system rankings alongside substantial variation in reliability and semantic outlier rates. The broader contribution is methodological: PaliBench shows how existing scholarly translations can be transformed into evaluation infrastructure for interpretive textual traditions without treating any single translation as definitive. Although developed for Pali Buddhist texts, the approach could be portable to other classical corpora where sufficient independent reference translations exist.
Máté Metzger, Nadnapang Phophichit
Independent Researcher · International Buddhist Studies College · Mahachulalongkornrajavidyalaya University
Classical texts aligned with their translations support machine translation, retrieval and computational research, but evidence comparing alignment workflows is scattered. This study compares seven systems on 452 texts in Pali, Sanskrit, Mishnaic Hebrew and Tibetan, comprising 9,833 human-aligned units: four embedding pipelines, a direct LLM call, an autonomous agent, and the agent revised by an independent auditor. Generative workflows recover 93-94% of reference correspondences, against at most 77% for embeddings. A ceiling analysis shows that sentence boundaries make some references unrepresentable by the embedding pipelines. Reference recovery is similar across generative workflows: the agent's advantage is 0.5 percentage points (95% CI -0.02 to 1.17), and auditing adds no established benefit. Agents nevertheless produce structurally valid output for all 452 texts, against 437 for direct calls. A blinded three-LLM panel assesses every generative mismatch against the source and human reference. Most mismatches are labelled defensible editorial variation; consensus major-error labels cover only 0.06-0.14% of units. The panel labels significantly fewer residual defects for agents than direct calls (0.7% versus 1.4%), suggesting that reference recovery alone understates alignment quality. On ten long Pali discourses taken as published online, agents and audited agents raise recovery from the direct call's 71% to 84% and 92%. Identical reference-located chunks bring all three to 93%. Agents thus improve structural reliability and reduce judged defects on short passages, while their large recovery advantage on long documents disappears after chunking. In this setting, independent auditing offers little measurable additional benefit on prepared passages.