Development and multi-center evaluation of domain-adapted speech recognition for human-AI teaming in real-world gastrointestinal endoscopy
Authors: Ruijie Yang, Yan Zhu, Peiyao Fu, Te Luo, Zhihua Wang, Xian Yang, Quanlin Li, Pinghong Zhou, Shuo Wang
Organizations: Zhejiang University, Hangzhou, China. · Shanghai Institute for Advanced Study, Zhejiang University, Shanghai, China. · Shanghai Key Laboratory of MICCAI, Shanghai, China. · Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China. · Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai, China. · Shanghai Collaborative Innovation Center of Endoscopy, Shanghai, China. · Alliance Manchester Business School, The University of Manchester, Manchester, UK.
Automatic speech recognition (ASR) is a critical interface for human-AI interaction in gastrointestinal endoscopy, yet its reliability in real-world clinical settings is limited by domain-specific terminology and complex acoustic conditions. Here, we present EndoASR, a domain-adapted ASR system designed for real-time deployment in endoscopic workflows. We develop a two-stage adaptation strategy based on synthetic endoscopy reports, targeting domain-specific language modeling and noise robustness. In retrospective evaluation across six endoscopists, EndoASR substantially improves both transcription accuracy and clinical usability, reducing character error rate (CER) from 20.52% to 14.14% and increasing medical term accuracy (Med ACC) from 54.30% to 87.59%. In a prospective multi-center study spanning five independent endoscopy centers, EndoASR demonstrates consistent generalization under heterogeneous real-world conditions. Compared with the baseline Paraformer model, CER is reduced from 16.20% to 14.97%, while Med ACC is improved from 61.63% to 84.16%, confirming its robustness in practical deployment scenarios. Notably, EndoASR achieves a real-time factor (RTF) of 0.005, significantly faster than Whisper-large-v3 (RTF 0.055), while maintaining a compact model size of 220M parameters, enabling efficient edge deployment. Furthermore, integration with large language models demonstrates that improved ASR quality directly enhances downstream structured information extraction and clinician-AI interaction. These results demonstrate that domain-adapted ASR can serve as a reliable interface for human-AI teaming in gastrointestinal endoscopy, with consistent performance validated across multi-center real-world clinical settings.
Figures & tables
Figure 1: Overall framework of EndoASR for real-world deployment in gastrointestinal endoscopy. The figure illustrates the conceptual and methodological design of EndoASR, spanning system motivation, model development, and clinical validation. Panel (a) presents the paradigm shift from algorithm-centric medical AI toward agent-based human–AI teaming, in which ASR serves as a real-time speech interface in hands-busy endoscopic workflows. Panel (b) summarizes the key challenges of speech recognition in endoscopy rooms, including specialized medical terminology, complex procedural acoustics, and real-time operational constraints. Panel (c) depicts the two-stage domain adaptation strategy, in which synthetic speech derived from structured endoscopy reports is used for domain-specific language adaptation, followed by noise-aware fine-tuning to improve robustness under realistic operating-room conditions. Panel (d) shows the progressive evaluation design, combining retrospective single-center validation across multiple endoscopists with prospective multi-center validation across diverse clinical content categories, enabling assessment of both controlled performance and real-world generalization.
Model
CER (%) ↓
BLEU-1 (%) ↑
BERTScore (%) ↑
RTF ↓
Whisper-large-v3
32.75 ± 16.27
68.41 ± 15.38
84.21 ± 6.74
0.0531
Whisper-turbo
31.63 ± 14.92
69.76 ± 13.27
83.72 ± 6.64
0.0182
Conformer-Transducer
50.51 ± 16.11
50.56 ± 16.03
72.45 ± 7.93
0.0230
Wav2Vec2-XLSR-53
58.77 ± 14.23
44.83 ± 11.88
65.15 ± 6.99
0.0018
Paraformer
20.52 ± 13.06
80.92 ± 10.26
90.11 ± 5.51
0.0051
Table 1: Performance comparison of general-purpose ASR models on the retrospective endoscopy dataset. Models were evaluated on 600 real intraoperative recordings from six endoscopists without domain-specific adaptation. Character error rate (CER), BLEU-1, and BERTScore are reported as mean ± standard deviation. Real-time factor (RTF) is included to illustrate model inference efficiency. Lower CER and RTF indicate better performance, while higher BLEU-1 and BERTScore reflect improved semantic consistency.
Condition
Model
CER (%) ↓
BLEU-1 (%) ↑
BERTScore (%) ↑
Mandarin audio
Paraformer
8.15 ± 4.43
91.99 ± 4.32
96.03 ± 2.03
EndoASR
3.71 ± 3.82
96.44 ± 3.60
98.60 ± 1.48
EndoASR-noise
3.21 ± 3.14
96.85 ± 3.06
98.81 ± 1.23
Audio with noise
Paraformer
9.10 ± 4.89
91.05 ± 4.79
95.49 ± 2.30
EndoASR
3.75 ± 3.77
96.39 ± 3.56
98.55 ± 1.47
EndoASR-noise
3.29 ± 3.28
96.79 ± 3.16
98.78 ± 1.25
Table 2: Performance of the two-stage fine-tuned ASR model on the synthetic endoscopic dataset. The baseline Paraformer model and the proposed model after stage-1 and stage-2 fine-tuning were evaluated on the synthetic test set, including both clean Mandarin speech and speech mixed with simulated endoscopy-room ambient noise. Results are reported as mean ± standard deviation. Lower CER and higher BLEU-1 and BERTScore indicate better performance.
Figure 2: Retrospective evaluation of ASR performance, inter-speaker variability, and noise-related robustness analysis. Panel (a) compares the performance of different ASR models on the retrospective clinical dataset across four metrics, including 1–CER, BLEU-1, BERTScore, and medical terminology accuracy. Panel (b) reports model performance stratified by individual endoscopists (P1–P6) on the same retrospective dataset, highlighting speaker-dependent variations across evaluation metrics. Panel (c) further visualizes inter-speaker variability by summarizing performance differences across endoscopists, illustrating the heterogeneity of intra-procedural speech characteristics. Panel (d) illustrates the relationship between model parameter size and medical terminology accuracy across different methods. Models closer to the upper-left corner achieve a more favorable efficiency–accuracy trade-off, delivering higher terminology accuracy with fewer parameters.
Model
CER ↓
BLEU-1 ↑
BERTScore ↑
Med ACC ↑
P1
Paraformer
25.06 ± 18.86
78.14 ± 10.94
88.56 ± 5.51
49.41 ± 36.33
EndoASR
20.88 ± 22.13
83.61 ± 12.38
92.43 ± 5.43
79.71 ± 29.78
EndoASR-noise
18.51 ± 19.57
84.99 ± 11.09
92.97 ± 5.03
82.91 ± 26.14
P2
Paraformer
15.93 ± 9.44
84.62 ± 8.85
91.96 ± 4.87
59.42 ± 37.22
EndoASR
10.02 ± 6.48
90.55 ± 5.95
95.28 ± 3.39
88.88 ± 21.67
EndoASR-noise
9.77 ± 6.25
90.79 ± 5.71
95.38 ± 3.29
89.89 ± 19.75
Table 3: Retrospective evaluation of the baseline and domain-adapted ASR models across six endoscopists. Performance of the baseline Paraformer model and the proposed models after stage-1 and stage-2 fine-tuning was evaluated on 600 real intraoperative recordings from six endoscopists (P1–P6). Results are reported as mean ± standard deviation for CER, BLEU-1, BERTScore, and medical term accuracy (Med ACC), all in %. Lower CER and higher BLEU-1, BERTScore, and Med ACC indicate better performance.
Figure 3: Prospective multi-center evaluation across clinical content categories. Panel (a) summarizes ASR performance on the prospective multi-center dataset across evaluation metrics, reflecting overall recognition quality under real-world clinical conditions. Panel (b) presents medical terminology accuracy stratified by center and content category, highlighting variation in domain-specific term recognition. Panel (c) compares 1–CER performance of different ASR models across centers and clinical categories, illustrating cross-institutional and content-dependent performance patterns in prospective evaluation.
Figure 4: Acoustic variability across centers and the impact of synthetic data scaling on medical term recognition. (a) t-SNE visualization of MFCC-based acoustic embeddings from the prospective multi-center dataset. Each point represents a speech segment, with colors indicating different clinical centers. Distinct clustering patterns are observed across centers, reflecting substantial inter-center variability in acoustic characteristics, likely due to differences in recording environments, speakers, and procedural conditions. (b) Effect of synthetic training data scale on medical term recognition accuracy. Performance is plotted against the size of synthetic data used for domain adaptation, where the point at size 0 corresponds to the baseline model without adaptation (Paraformer). Results show a consistent improvement in accuracy as the amount of synthetic data increases.
Model
CER ↓
BLEU-1 ↑
BERTScore ↑
Med ACC ↑
C1
Paraformer
15.24 ± 12.28
85.91 ± 9.36
92.56 ± 4.80
66.72 ± 36.53
EndoASR
14.50 ± 11.42
86.89 ± 8.67
93.26 ± 4.10
86.22 ± 25.38
EndoASR-noise
14.75 ± 10.81
86.49 ± 8.34
92.90 ± 4.05
89.05 ± 21.87
C2
Paraformer
14.76 ± 11.16
85.78 ± 10.59
92.84 ± 5.11
67.81 ± 38.41
EndoASR
14.30 ± 8.62
86.37 ± 7.96
93.16 ± 4.35
87.16 ± 22.44
EndoASR-noise
14.83 ± 9.28
85.78 ± 8.74
92.64 ± 4.73
89.72 ± 21.10
Table 4: Prospective multi-center evaluation of the domain-adapted ASR models in real-world endoscopic practice. Performance of the baseline Paraformer model and the proposed domain-adapted models after stage-1 and stage-2 fine-tuning was evaluated on a prospective multi-center dataset collected from five independent endoscopy centers (C1–C5). Each center contributed 60 intraoperative recordings spanning six clinical content categories. Results are reported as mean ± standard deviation for CER, BLEU-1, BERTScore, and medical term accuracy (Med ACC), all in %. Lower CER and higher BLEU-1, BERTScore, and Med ACC indicate better performance.
Figure 5: Data construction, transcription quality, runtime efficiency, and downstream usability of EndoASR. Panel (a) illustrates the construction of the training data, including synthetic speech generated from structured clinical text and noise-augmented speech informed by real endoscopy-room acoustics. Panel (b) compares representative transcription outputs produced by Whisper-large-v3, Paraformer, and EndoASR-noise, highlighting differences in domain-specific terminology recognition and transcription errors (shown in red). Panel (c) presents a comparison of real-time factor (RTF) across different ASR models, reflecting their runtime efficiency under practical deployment settings. Panel (d) demonstrates downstream usage by integrating ASR outputs with a large language model for structured clinical information extraction, comparing the performance of EndoASR and Whisper-large-v3 in supporting accurate and usable endoscopy report generation.
This paper presents an end-to-end study of automatic speech recognition (ASR) for adolescent health communication in three Ghanaian languages (Twi, Dagbani, and Ewe). The work proceeds in three connected stages; First, we benchmark five ASR systems (three language-specific Wav2Vec2 models and two multimodal LLMs, Gemma 3n and Gemma 4) on a general-domain Bible corpus and a Youth Adolescent Sexual and Reproductive Health (ASRH) Domain ASR dataset, using Character and Word Error Rate (CER, WER). Second, guided by the benchmark, we perform supervised domain adaptation: although Gemma 4 was the strongest zero-shot candidate, fine-tuning it proved computationally infeasible, so we pivoted to the compact Qwen3-ASR-0.6B, fine-tuned on a large Ghana Bible corpus (~90k samples) and evaluated strictly on held-out human-collected in-domain audio. Fine-tuning reduced WER on every language, most dramatically for Ewe (WER from 109.3% to 64.8%, a drop of 44.5 pp; CER from 65.1% to 24.9%). Third, we validate the work through KasaHealth, a live voice-first ASRH application deployed in all three languages, complemented by Senti-Check, a technical evaluation harness. KasaHealth was tested by 50 community respondents and achieved a 100% chat-approval rate, a 72% Good-or-Excellent translation rating, and a 92% would-recommend rate, while surfacing the domain gaps that most constrain real-world use. Across all three stages the evidence converges: for these languages the binding constraint is validated in-domain data, not model capability or computation.
Stephen E. Moore, Akwasi Asare, Mich-Seth Owusu +3
Department of Mathematics, University of Cape Coast, Ghana · Ghana Natural Language Processing, Cape Coast, Ghana
Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR architectures connect a speech encoder to a large language model via a projection module, enabling adaptation with text-only data. However, this introduces a modality gap, as the LLM is not exposed to the noisy representations produced by the speech projector. We investigate whether small amounts of speech can mitigate this mismatch. We compare three strategies: text-only adaptation, paired speech-text adaptation, and mixed batching (MB), which combines both. Experiments in in-domain and out-of-domain settings show that even limited speech consistently improves performance. Notably, MB using only 10% of the target-domain (less than 4 hours) speech achieves word error rates comparable to, or better than, conventional ASR fine-tuning with the full dataset, indicating that small amounts of speech provide a strong modality-alignment signal.
Domain-specific speech remains a persistent challenge for automatic speech recognition (ASR), even for state-of-the-art systems like OpenAI's Whisper. We introduce Whisper: Courtside Edition, a novel multi-agent large language model (LLM) pipeline that enhances Whisper transcriptions without retraining. The pipeline intercepts Whisper's initial transcript, applies specialized LLM agents for domain context identification, named entity recognition, and jargon detection, and generates compact prompts that guide Whisper's decoder. Evaluated on 421 NBA basketball commentary segments (a domain characterized by dense proper nouns and technical terminology) our best pipeline achieves a statistically significant 17.0% relative reduction in word error rate (WER; from 0.217 to 0.180, p<0.001). Improvements are observed in 40.1% of segments with degradation in only 7.1%, substantially outperforming direct transcript post-editing. These results demonstrate that prompt-based augmentation can deliver scalable domain adaptation for ASR, offering a practical alternative to costly model fine-tuning.