Fine-Tuning a Transducer for Impaired Speech: Where the Gains Appear and What They Mean for Evaluation

Kien Nguyen; Tuan Dinh; Bao Phan

cs.CLNov 5, 2026

Fine-Tuning a Transducer for Impaired Speech: Where the Gains Appear and What They Mean for Evaluation

Authors: Kien Nguyen, Tuan Dinh, Bao Phan

Organizations: Eloky

Abstract

Speech recognition still works poorly for many people whose speech is affected by conditions such as ALS, Parkinson's disease, cerebral palsy, Down syndrome or stroke. We took part in the Speech Accessibility Project Challenge 2 (SAPC2) with a public speech recognizer that we fine-tuned on the challenge data. Our best entry scored 7.81% character error rate on Test1, the first of two hidden test sets (official baseline 15.26%). Our live entries ran on CPUs with a combined median latency of about 0.9 seconds at 7.96-8.07% error. On new speakers, fine-tuning reduced errors most for the speakers that an off-the-shelf recognizer finds hardest to understand. Sentences already seen in training had far lower error rates than new ones (2.58% vs 6.42%), a larger difference than being a familiar speaker made. We close with lessons for evaluating such systems.

Figures & tables

System	Trained on selection speakers	CER	WER
Official challenge baseline (organisers)	–	15.26	21.53
Earlier entry	no	8.30	11.95
Best entry trained without the selection speakers	no	7.87	11.78
Best scored entry, trained on all labelled speakers	yes	7.81	11.69

Table 1: Official Track 1 results on the hidden Test1 set (%). Results for the second test set were not released when this version was written.

Condition	Speakers	Recordings	Ours (averaged)	Parakeet TDT 1.1B	Whisper large-v3
			fine-tuned	before fine-tuning	off the shelf
Parkinson’s disease	11	1,356	4.76 [2.37, 7.47]	7.51 [3.21, 13.29]	8.00 [3.72, 13.18]
ALS	10	522	4.95 [2.02, 11.67]	8.60 [3.82, 19.38]	11.60 [6.32, 21.39]
Cerebral palsy	9	1,058	8.49 [5.01, 14.34]	20.37 [10.07, 40.11]	19.59 [11.88, 30.79]
Down syndrome	7	1,066	7.00 [5.01, 9.72]	14.73 [10.12, 19.74]	15.12 [10.46, 20.65]
Stroke †	4	508	8.56 [3.00, 17.21]	20.59 [7.44, 50.68]	20.76 [6.83, 41.66]

Table 2: CER (%) on the selection panel (41 new speakers) by condition, with 95% speaker intervals. “Ours” is the model of our best entry trained without the selection speakers. † Only four speakers.

Group (Whisper CER)	Spk.	Fine-tuned	Original	Removed	Spk.	Fine-tuned	Original
	New sentences only				All recordings
Mild ( $<$ 5)	37	1.91 [1.53, 2.30]	2.38 [1.99, 2.77]	20%	41	1.22	2.76
Moderate (5–15)	38	4.04 [3.15, 5.05]	5.57 [4.46, 6.85]	27%	34	2.55	6.49
Severe (15–30)	23	10.89 [8.93, 13.37]	17.00 [11.96, 23.86]	36%	22	7.09	18.08
Very severe ( $\geq$ 30)	26	30.62 [24.98, 37.46]	51.88 [41.62, 62.15]	41%	27	16.38	56.44

Table 3: CER (%) of the 124 new speakers grouped by their CER under Whisper large-v3, with 95% speaker intervals. Groups are Whisper CER ranges in %. “Fine-tuned” is the analysis model; “Original” is Parakeet before fine-tuning; “Removed” is the share of its errors that fine-tuning removed. “New sentences only” keeps recordings whose text is not in the analysis model’s training data; groups are recomputed on those recordings, so speaker counts differ.

Group (speaker / sentence)	Rec.	Spk.	Fine-tuned	Parakeet	Whisper
			(analysis)	original	off the shelf
1: known / known	2,362	300	1.45 [1.05, 1.87]	13.87 [11.84, 16.11]	13.07 [11.07, 15.19]
1a: same speaker	1,105	293	0.75 [0.47, 1.08]	12.90 [10.79, 15.29]	12.29 [10.07, 14.72]
1b: other speakers only	1,257	300	1.86 [1.38, 2.44]	14.45 [12.35, 16.77]	13.54 [11.51, 15.74]
2: known / new	1,500	300	5.02 [4.34, 5.78]	9.32 [7.84, 11.01]	8.67 [7.50, 9.95]
3: new / known	37,281	124	2.58 [1.71, 3.60]	16.18 [12.68, 20.03]	15.20 [11.90, 18.86]

Table 4: CER (%) by group, with 95% speaker intervals. Groups are labelled speaker / sentence. Group 1 combines 1a (the same speaker said the sentence in training) and 1b (only other speakers did). “Fine-tuned” is the analysis model; “Parakeet” is Parakeet TDT 1.1B before fine-tuning; “Whisper” is Whisper large-v3 off the shelf. “Known” refers to the analysis model’s training data; for the other two models these are simply the same recordings.

Track 2 entry	CER (%)	First stable word (s)	Final text (s)	Sum (s)
Entry 1	8.05	7.35	1.20	8.56
Entry 2	7.98	0.82	1.51	2.34
Entry 3	8.02	0.41	0.53	0.94
Entry 4	7.96	0.41	0.52	0.93
Entry 5	7.97	0.42	0.50	0.92
Entry 6	8.07	0.42	0.44	0.85

Table 5: Our six scored Track 2 entries on Test1, as reported by the organisers: CER and the two median latencies in seconds. “Sum” adds the two medians.

Dysarthric Speech Recognition

Dysarthric speech recognition develops systems that transcribe speech affected by impaired motor control and articulation.

See all 15 papers

Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors
Oct 17, 2026Sankalp Parashar, Preethi Jyothi, Sunita SarawagiAutomatic Speech Recognition Streaming ASR
IIT Bombay
Adapting Foundation ASR Models to Dysarthric Speech: A Case Study
Jun 30, 2026Christian Huber, Laura Kernahan, Alexander WaibelPersonalized Speech Recognition Automatic Speech Recognition
Interactive Systems Lab Karlsruhe Institute of Technology Karlsruhe, Germany · Interactive Systems Lab Carnegie Mellon University Pittsburgh PA, USA
When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition
May 4, 2026Pehuén Moure, Niclas Pokel, Bilal Bounajma +4Audio-Language Model Evaluation ASR Evaluation
Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland · Department of Mathematics, ETH Zurich, Switzerland · Department of Computational Linguistics, University of Zurich, Switzerland

Oct 17, 2026J/K move · Enter open · S save

Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors

Sankalp Parashar, Preethi Jyothi, Sunita Sarawagi

IIT Bombay

Automatic Speech Recognition Streaming ASR

Open paper AskPDF arXiv

Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during streaming ASR permits a recourse mechanism where the speaker can be asked to repeat the affected content under cleaner acoustic conditions. We propose PARDON, a new framework for online error localization, that identifies spans likely to benefit from this intervention. Our supervision uses clean audio to estimate how the recognizer's output might change under improved acoustic conditions. We probabilistically align noisy and clean decodes using a pair-HMM to obtain soft supervision. These soft targets train a counterfactual gain scorer by combining a suite of acoustic and language features to produce token-level scores, that are further grouped into spans for targeted recourse. An LLM uses the ASR transcript of the repetition to revise only the selected span. At a precision of 70%, PARDON achieves the highest recall by identifying up to 52% more erroneous tokens compared to the strongest baseline. Targeted repetition yields relative WER reductions of up to 24% under a budget that accounts for both interruption frequency and the lengths of re-uttered spans.

Abstract

Dysarthric Speech Recognition

Explore similar work

Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors

Adapting Foundation ASR Models to Dysarthric Speech: A Case Study

When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition