Cascaded speech-to-text translation (S2TT) systems for low-resource languages can suffer from structural noise, particularly the loss of punctuation during the Automatic Speech Recognition (ASR) phase. This research investigates the impact of such noise on Nepali-to-English translation and proposes an optimized pipeline to mitigate quality degradation. We first establish highly proficient ASR and NMT components: a Wav2Vec2-XLS-R-300m model achieved a state-of-the-art 2.72% CER on OpenSLR-54, and a multi-stage fine-tuned MarianMT model reached a 28.32 BLEU score on the FLORES-200 benchmark. We empirically investigate the influence of punctuation loss, demonstrating that unpunctuated ASR output significantly degrades translation quality, causing a massive 20.7% relative BLEU drop on the FLORES benchmark. To overcome this, we propose and evaluate an intermediate Punctuation Restoration Module (PRM). The final S2TT pipeline was tested across three configurations on a custom dataset. The optimal configuration, which applied the PRM directly to ASR output, achieved a 4.90 BLEU point gain over the direct ASR-to-NMT baseline (BLEU 36.38 vs. 31.48). This improvement was validated by human assessment, which confirmed the optimized pipeline's superior Adequacy (3.673) and Fluency (3.804) with inter-rater reliability (Krippendorff's α≥ 0.723). This work validates that targeted punctuation restoration is the most effective intervention for mitigating structural noise in the Nepali S2TT pipeline. It establishes an optimized baseline and demonstrates a critical architectural insight for developing cascaded speech translation systems for similar low-resource languages.
Figures & tables
Figure 1. Cascaded Nepali-to-English Speech-To-Text-Translation System Architecture
Table 1. Nepali ASR datasets used for fine-tuning.
Stage
Corpus Type
Size (Pairs)
Primary Source / Filtering
Rationale
Stage 1: Foundational
Filtered Web crawled NLLB a
708,000
Initial 19.6M web-crawled NLLB corpus pairs filtered by LabSE cosine similarity ( >0.80 ), and removal of Nepali numerals (to maintain consistency).
Knowledge Transfer: To learn fundamental translation nuances from filtered, noisy web data.
Stage 2: Expansion
Synthetic Corpora b,c
5.02M
Combination of 1.6M synthetic pairs ( Duwal and Bal, 2019 ) and 3.42M pre-training pairs ( Duwal et al., 2024 ) filtered using a chrF++ cut-off of 50 (translated with 8-bit NLLB/IndicTrans2).
Coverage Expansion: Increase exposure to diverse structures and vocabularies using high-volume, quality-filtered synthetic data.
Stage 3: Refinement
Manually translated high-Quality corpora d,e
210,875
OPUS (GNOME/KDE/Ubuntu), Bible, Global Voices, Penn Treebank + ( Acharya and Bal, 2018 ) , Nepal Law Commission, Easy Bible, Nepal Budget Speech 2081/82, Shirish ko Fool Translation.
Quality Refinement: Final tuning for translation fluency and accuracy on high-fidelity human-translated pairs.
Table 2. Corpora used at each stage of NMT training and their filtering/rationale.
Hyperparameter
Value
Learning Rate (Final)
2×10−5
Batch Size (Per Device)
16
Gradient Accumulation
2
Warmup Steps
500
Table 3. Key Hyperparameters for Wav2Vec2.0 Fine-Tuning
Hyperparameter
Value
Learning Rate
2×10−5
Batch Size (Train)
8
Weight Decay
0.1
Table 4. Key Hyperparameters for mT5 Fine-Tuning
Hyperparameter
Pre-training
Fine-tuning
Learning Rate
5×10−5
2×10−5
Batch Size (Train/Eval)
16
16
Weight Decay
0.01
0.01
Warmup Steps
5% of total steps
5% of total steps
Table 5. Hyperparameters for MarianMT Pre-training and Fine-tuning
Evaluation Type
Metrics
ASR
WER, CER
NMT Benchmarking
BLEU, chrF++, METEOR
Punctuation Impact & S2TT
Δ BLEU, Δ chrF++
Human Evaluation
1–5 Likert Scale
Table 6. Evaluation Types and Corresponding Metrics
Metric
Result
Word Error Rate (WER)
16.82%
Character Error Rate (CER)
2.72%
Table 7. ASR Performance on OpenSLR-54 Test Set
Test Set
Metric
Our Score
Previous Best
FLORES-200 DevTest
BLEU
29.04
20.76
chrF++
58.14
N/A
METEOR
0.6314
N/A
FLORES-200 Dev
BLEU
28.48
19.37
chrF++
58.07
N/A
METEOR
0.5676
N/A
Table 8. Benchmarking of the Final MarianMT Model Against Standard Test Sets and Previous Best Results
Test Set
Condition
BLEU
chrF++
Δ BLEU
Δ chrF++
FLORES-200 DevTest
Punctuated
29.04
58.14
—
—
Unpunctuated (ASR)
23.13
55.25
-5.91
-2.89
FLORES-200 Dev
Punctuated
28.48
58.07
—
—
Unpunctuated (ASR)
24.12
56.11
-4.36
-1.97
Tatoeba
Punctuated
39.66
55.73
—
—
Unpunctuated (ASR)
28.40
51.16
-11.26
-4.57
Table 9. Impact of Punctuation on Translation Quality Across Test Sets (MarianMT Model Evaluation)
Figure 2. Comparative performance of NMT on punctuated and unpunctuated (Simulated ASR) inputs across three test sets. A consistent degradation is observed, confirming that punctuation loss substantially impacts translation quality.