Cascaded speech-to-text translation (S2TT) systems for low-resource languages can suffer from structural noise, particularly the loss of punctuation during the Automatic Speech Recognition (ASR) phase. This research investigates the impact of such noise on Nepali-to-English translation and proposes an optimized pipeline to mitigate quality degradation. We first establish highly proficient ASR and NMT components: a Wav2Vec2-XLS-R-300m model achieved a state-of-the-art 2.72% CER on OpenSLR-54, and a multi-stage fine-tuned MarianMT model reached a 28.32 BLEU score on the FLORES-200 benchmark. We empirically investigate the influence of punctuation loss, demonstrating that unpunctuated ASR output significantly degrades translation quality, causing a massive 20.7% relative BLEU drop on the FLORES benchmark. To overcome this, we propose and evaluate an intermediate Punctuation Restoration Module (PRM). The final S2TT pipeline was tested across three configurations on a custom dataset. The optimal configuration, which applied the PRM directly to ASR output, achieved a 4.90 BLEU point gain over the direct ASR-to-NMT baseline (BLEU 36.38 vs. 31.48). This improvement was validated by human assessment, which confirmed the optimized pipeline's superior Adequacy (3.673) and Fluency (3.804) with inter-rater reliability (Krippendorff's α≥ 0.723). This work validates that targeted punctuation restoration is the most effective intervention for mitigating structural noise in the Nepali S2TT pipeline. It establishes an optimized baseline and demonstrates a critical architectural insight for developing cascaded speech translation systems for similar low-resource languages.
Figures & tables
Figure 1. Cascaded Nepali-to-English Speech-To-Text-Translation System Architecture
Table 1. Nepali ASR datasets used for fine-tuning.
Stage
Corpus Type
Size (Pairs)
Primary Source / Filtering
Rationale
Stage 1: Foundational
Filtered Web crawled NLLB a
708,000
Initial 19.6M web-crawled NLLB corpus pairs filtered by LabSE cosine similarity ( >0.80 ), and removal of Nepali numerals (to maintain consistency).
Knowledge Transfer: To learn fundamental translation nuances from filtered, noisy web data.
Stage 2: Expansion
Synthetic Corpora b,c
5.02M
Combination of 1.6M synthetic pairs ( Duwal and Bal, 2019 ) and 3.42M pre-training pairs ( Duwal et al., 2024 ) filtered using a chrF++ cut-off of 50 (translated with 8-bit NLLB/IndicTrans2).
Coverage Expansion: Increase exposure to diverse structures and vocabularies using high-volume, quality-filtered synthetic data.
Stage 3: Refinement
Manually translated high-Quality corpora d,e
210,875
OPUS (GNOME/KDE/Ubuntu), Bible, Global Voices, Penn Treebank + ( Acharya and Bal, 2018 ) , Nepal Law Commission, Easy Bible, Nepal Budget Speech 2081/82, Shirish ko Fool Translation.
Quality Refinement: Final tuning for translation fluency and accuracy on high-fidelity human-translated pairs.
Table 2. Corpora used at each stage of NMT training and their filtering/rationale.
Hyperparameter
Value
Learning Rate (Final)
2×10−5
Batch Size (Per Device)
16
Gradient Accumulation
2
Warmup Steps
500
Table 3. Key Hyperparameters for Wav2Vec2.0 Fine-Tuning
Hyperparameter
Value
Learning Rate
2×10−5
Batch Size (Train)
8
Weight Decay
0.1
Table 4. Key Hyperparameters for mT5 Fine-Tuning
Hyperparameter
Pre-training
Fine-tuning
Learning Rate
5×10−5
2×10−5
Batch Size (Train/Eval)
16
16
Weight Decay
0.01
0.01
Warmup Steps
5% of total steps
5% of total steps
Table 5. Hyperparameters for MarianMT Pre-training and Fine-tuning
Evaluation Type
Metrics
ASR
WER, CER
NMT Benchmarking
BLEU, chrF++, METEOR
Punctuation Impact & S2TT
Δ BLEU, Δ chrF++
Human Evaluation
1–5 Likert Scale
Table 6. Evaluation Types and Corresponding Metrics
Metric
Result
Word Error Rate (WER)
16.82%
Character Error Rate (CER)
2.72%
Table 7. ASR Performance on OpenSLR-54 Test Set
Test Set
Metric
Our Score
Previous Best
FLORES-200 DevTest
BLEU
29.04
20.76
chrF++
58.14
N/A
METEOR
0.6314
N/A
FLORES-200 Dev
BLEU
28.48
19.37
chrF++
58.07
N/A
METEOR
0.5676
N/A
Table 8. Benchmarking of the Final MarianMT Model Against Standard Test Sets and Previous Best Results
Test Set
Condition
BLEU
chrF++
Δ BLEU
Δ chrF++
FLORES-200 DevTest
Punctuated
29.04
58.14
—
—
Unpunctuated (ASR)
23.13
55.25
-5.91
-2.89
FLORES-200 Dev
Punctuated
28.48
58.07
—
—
Unpunctuated (ASR)
24.12
56.11
-4.36
-1.97
Tatoeba
Punctuated
39.66
55.73
—
—
Unpunctuated (ASR)
28.40
51.16
-11.26
-4.57
Table 9. Impact of Punctuation on Translation Quality Across Test Sets (MarianMT Model Evaluation)
Figure 2. Comparative performance of NMT on punctuated and unpunctuated (Simulated ASR) inputs across three test sets. A consistent degradation is observed, confirming that punctuation loss substantially impacts translation quality.
Current speech translation systems, including SpeechLLMs, are trained on cleaned text and tend to strip disfluencies like filled pauses and false starts rather than translate them. We show this comes at a cost: disfluencies carry meaning that gets lost when speech is cleaned up. To study this systematically, we introduce Uh-Mazing, a benchmark of human-translated, disfluency-annotated Switchboard speech covering English into eight target languages. Across these languages and several architectures, we find that false starts and self-repairs, not filled pauses or discourse markers, drive most of the translation-quality loss, and that models which fail to preserve a disfluency tend to omit it rather than mistranslate it. We show inference-time decoding can mitigate this without retraining, and release the benchmark and code.
Maike Züfle, Maria Teleki, Fabian Retkowski +5
Karlsruhe Institute of Technology · Texas A&M University · ETH Zurich +1
Speech translation for low-resource languages remains fundamentally limited by the scarcity of high-quality, diverse parallel speech data, a challenge that is especially pronounced in African linguistic contexts. To address this, we introduce NaijaS2ST, a parallel speech translation dataset spanning Igbo, Hausa, Yorùbá, and Nigerian Pidgin paired with English. The dataset comprises approximately 50 hours of speech per language and captures substantial variation in speakers and accents, reflecting realistic multilingual and multi-accent conditions. With NaijaS2ST, we conduct a comprehensive benchmark of cascaded, end-to-end (E2E), and AudioLLM-based approaches across bidirectional translation settings. Our results show that audio LLMs with few-shot examples are more effective for speech-to-text translation than cascaded and end-to-end methods trained on fine-tuned data. However, for speech-to-speech translation, the cascaded and audio LLM paradigms yield comparable performance, indicating that there is still considerable room for improvement in developing targeted, task-specific models for this setting. By providing both a high-quality dataset and a systematic benchmark, we hope that NaijaS2ST will serve as a strong foundation for advancing research in low-resource, multilingual speech translation.
Marie Maltais, Yejin Jeon, Min Ma +7
Mila - Quebec AI Institute · McGill University, Canada · Google DeepMind +6
Punctuation restoration is essential for improving the readability and downstream utility of automatic speech recognition (ASR) outputs, yet remains underexplored for Persian despite its importance. We introduce PersianPunc, a large-scale, high-quality dataset of 17 million samples for Persian punctuation restoration, constructed through systematic aggregation and filtering of existing textual resources. We formulate punctuation restoration as a token-level sequence labeling task and fine-tune ParsBERT to achieve strong performance. Through comparative evaluation, we demonstrate that while large language models can perform punctuation restoration, they suffer from critical limitations: over-correction tendencies that introduce undesired edits beyond punctuation insertion (particularly problematic for speech-to-text pipelines) and substantially higher computational requirements. Our lightweight BERT-based approach achieves a macro-averaged F1 score of 91.33% on our test set while maintaining efficiency suitable for real-time applications. We make our dataset (https://huggingface.co/datasets/MohammadJRanjbar/persian-punctuation-restoration) and model (https://huggingface.co/MohammadJRanjbar/parsbert-persian-punctuation) publicly available to facilitate future research in Persian NLP and provide a scalable framework applicable to other morphologically rich, low-resource languages.
Mohammad Javad Ranjbar Kalahroodi, Heshaam Faili, Azadeh Shakery
University of Tehran, Tehran, Iran · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran