NepTTS-Bench: Native-Speaker Benchmarking and NepaliMOS for Nepali Text-to-Speech
Organizations: Ampixa Labs · Kathmandu University · Central Department of Linguistics Tribhuvan University
Abstract
Nepali TTS has reached a point where multiple systems are available, but there is no shared evaluation to compare them. NepTTS-Bench was created to address what existing automated and manual evaluation systems lack for Nepali TTS. A controlled benchmark with phonological grounding which contains 205 sentences (77 minimal pairs targeting aspiration, retroflex/dental contrasts, nasalization, gemination, and schwa deletion) which is rated by 180 speakers. There are 6,962 ratings for 10 TTS configurations and two human references. Along with this 7 ASR systems were evaluated for round-trip CER. SCOREQ was also run as an automated baseline. The key finding is that automated evaluation like SCOREQ prefers proprietary systems like ElevenLabs but native listeners prefer Piper. This is contributed by the fact that most of the systems not properly tuned for Nepali tend to show Hindi phonological behaviour and our minimal pairs are designed to surface such contrasts. We also present NepaliMOS predictor which is fine tuned on collected ratings. NepaliMOS achieves system-level rho = 0.90 on the dense panel but drops to utterance-level rho = 0.43 under leave-one-system-out evaluation. This is statistically significant and confirms that NepaliMOS captures genuine signal but cross-provider transfer is limited. A human calibration is recommended for new systems for which we release a Python evaluation package for practitioners to run the benchmark against new systems.
Figures & tables
| System | Access | MOS | 95% CI | System | Access | MOS | 95% CI | ||
|---|---|---|---|---|---|---|---|---|---|
| Natural speech | Ref. | 3.95 | [3.79,4.15] | 597 | Gemini (Hindi) | Prop. | 3.34 | [3.15,3.53] | 514 |
| Human (read) | Ref. | 3.64 | [3.44,3.82] | 552 | Gemini Flash | Prop. | 3.25 | [2.90,3.68] | 618 |
| TingTing Asmita | Prop. | 3.55 | [3.27,3.88] | 651 | TingTing Sambriddhi | Prop. | 3.20 | [2.91,3.55] | 617 |
| Piper | Open/local | 3.50 | [3.31,3.69] | 515 | gTTS | Prop. | 2.60 | [2.38,2.83] | 510 |
| TingTing Subina | Prop. | 3.47 | [3.24,3.79] | 644 | |||||
| ElevenLabs v3 | Prop. | 3.47 | [3.28,3.66] | 516 |
| Metric | vs. human MOS | Raw |
|---|---|---|
| NepaliMOS (ours) | 0.90 | 0.001 |
| Chirp 2 CER | 0.60 | 0.09 |
| MMS CER | 0.58 | 0.10 |
| XLS-R Nepali CER | 0.53 | 0.14 |
| SCOREQ | 0.40 | 0.29 |
| NepConformer CER | 0.30 | 0.43 |
TTS Synthesis
Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.
See all 294 papers