Advancing Sentence-Level Dysarthric Audio-Visual Speech Recognition via Test-Time Adaptation

Shiyao Wang; Hui Wang; Jiaming Zhou; Shiwan Zhao; Yong Qin

cs.SDOct 16, 2026

Advancing Sentence-Level Dysarthric Audio-Visual Speech Recognition via Test-Time Adaptation

Authors: Shiyao Wang, Hui Wang, Jiaming Zhou, Shiwan Zhao, Yong Qin

Organizations: Academy for Advanced Interdisciplinary Studies, Nankai University, Tianjin, China

Abstract

Dysarthric audio-visual speech recognition (DAVSR) enhances performance by integrating complementary visual cues. However, existing research is largely confined to word-level tasks with limited vocabularies, which severely restricts real-world communication for dysarthric speakers. Furthermore, performance remains heavily dependent on supervised fine-tuning with speaker-specific data; significant variations in age, etiology, and severity cause traditional methods to struggle with unseen speakers. To address these domain shifts and the scarcity of continuous multimodal resources, we first introduce a "source speech search-segmentation" pipeline to construct a structured sentence-level corpus from raw CDSD videos. We then propose the use of unsupervised Test-Time Adaptation (TTA) for sentence-level DAVSR using Sequential-level Generalized Entropy Minimization (SGEM). Our evaluations across the state-of-the-art AV-SUPERB and ViSpeR frameworks demonstrate that TTA consistently alleviates Character Error Rate (CER) degradation for unseen pathological speakers. Paired t-tests verify that these improvements are statistically significant (p < 0.05), highlighting TTA as a promising paradigm for advancing speaker-adaptive assistive speech interfaces.

Figures & tables

Figure 1: The workflow of the source speech search-segmentation procedure. The framework processes raw audio-visual streams through three core stages: (1) monotonic segmentation to ensure chronological consistency; (2) Fbank feature extraction and distance calculation to locate candidates via minimum Euclidean distance; and (3) multi-modal manual result verification and refinement to filter out no-match cases and finalize the aligned sentence-level corpus.

ID	Segmented Speech		Raw Video		Aligned	Availability
	Count	Range (s)	Count	Range (s)	Data	Rate
01	1107	[0.5, 10.8]	22	[136.2, 279.6]	1107	100.0%
02	988	[0.4, 12.6]	12	[237.6, 348.6]	968	98.0%
03	926	[0.4, 15.7]	12	[154.8, 585.6]	912	98.5%
04	1035	[0.5, 10.4]	14	[252.6, 307.2]	1035	100.0%
06	969	[0.6, 9.2]	9	[297.6, 744.0]	967	99.8%

Table 1: Statistics of the CDSD dataset.

Figure 2: TTA was evaluated on two audio-visual speech recognition frameworks: AV-SUPERB and ViSpeR. Colors indicate different TTA setups, identifying the audio feature extractor, video feature extractor, cross-modal fusion layer, encoder, layer normalization (LayerNorm) layers, and decoder, respectively. Color viewing is recommended.

Figure 3: Qualitative examples of speech recognition transcripts from speakers 06 and 44 under their optimal model (ViSpeR), demonstrating the error-correction capability of the TTA.

\Delta\text{CER}

quantifies the absolute reduction in CER.

Figure 4: Distribution of testing utterances per speaker, categorized by the level of CER reduction achieved through TTA across four distinct evaluation frameworks: AS-1, AS-2, AS-3, and ViSpeR. To compensate for visual limitations, the samples are binned into four precise CER improvement intervals (

\Delta\text{CER}=\text{CER}_{\text{unadapted}}-\text{CER}_{\text{TTA}}

): marginal-to-low correction (

25\%>\Delta\text{CER}\geq 0\%

, blue bars), low-to-moderate correction (

50\%>\Delta\text{CER}\geq 25\%

, orange bars), moderate-to-high correction (

75\%>\Delta\text{CER}\geq 50\%

, green bars), and significant correction (

\Delta\text{CER}\geq 75\%

, red bars).

Speaker ID	Model	Baseline	TTA-1	TTA-3	TTA-5	TTA-10	$\Delta$
	AS-1	88.4	87.6	87.1	87.8	87.7	1.3
01	AS-2	74.4	72.1	71.0	71.7	72.2	3.4
	AS-3	75.0	73.0	72.5	72.7	73.3	2.5
	ViSpeR	58.9	55.7	48.1	47.3	47.1*	11.8
	AS-1	85.2	85.5	84.3	84.3	84.4	0.9
02	AS-2	88.2	87.6	87.4	87.4	88.1	0.8

Table 2: TTA Performance (CER, %) on unseen speakers across various adaptation iterations. The absolute CER reduction (

\Delta

) is computed as the baseline error minus the optimal adapted error (

\text{CER}_{\text{Baseline}}-\text{CER}_{\text{Optimal}}

), with the latter denoted by the bolded values in each row. The asterisk (

*

) highlights the best recognition performance achieved per speaker across all evaluated configurations.

Model	Setting	Baseline	TTA-1	TTA-3	TTA-5	TTA-10	$\Delta$
	LayerNorm	88.4	87.7	87.6	87.7	88.5	0.8
	Feature	88.4	87.6	87.1	87.8	87.7	1.3
AS-1	Encoder	88.4	87.7	87.5	87.9	88.7	0.9
	Decoder	88.4	87.8	87.6	87.9	88.7	0.8
	All	88.4	87.6	87.6	87.7	88.2	0.8
	LayerNorm	74.4	72.4	72.1	73.1	73.2	2.3

Table 3: The performance of different TTA settings for speaker 01 under various frameworks (CER, %).

Dysarthric Speech Recognition

Dysarthric speech recognition develops systems that transcribe speech affected by impaired motor control and articulation.

See all 14 papers

End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
Mar 2, 2026Minghui Wu, Haitao Tang, Jiahuan Fan +2Clinical Speech Processing TTS Synthesis
University of Science and Technology of China, China · iFlytek Co., Ltd., China
DARS: Dysarthria-Aware Rhythm-Style Synthesis for ASR Enhancement
Mar 2, 2026Minghui Wu, Xueling Liu, Jiahuan Fan +3Synthetic Data Augmentation for Speech Recognition TTS Synthesis
University of Science and Technology of China, China · iFlytek Co., Ltd., China · Huawei Technology, China
Adapting Foundation ASR Models to Dysarthric Speech: A Case Study
Jun 30, 2026Christian Huber, Laura Kernahan, Alexander WaibelPersonalized Speech Recognition Automatic Speech Recognition
Interactive Systems Lab Karlsruhe Institute of Technology Karlsruhe, Germany · Interactive Systems Lab Carnegie Mellon University Pittsburgh PA, USA

Mar 2, 2026J/K move · Enter open · S save

End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation

Minghui Wu, Haitao Tang, Jiahuan Fan, Ruizhi Liao+1

University of Science and Technology of China, China · iFlytek Co., Ltd., China

Clinical Speech Processing TTS Synthesis

Open paper AskPDF arXiv

Dysarthric speech reconstruction (DSR) typically employs a cascaded system that combines automatic speech recognition (ASR) and sentence-level text-to-speech (TTS) to convert dysarthric speech into normally-prosodied speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicit-implicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge distillation. Our system has an average response time of 1.03 seconds on Tesla A100, with an average real-time factor (RTF) of 0.71. On the UASpeech dataset, it attains a mean opinion score (MOS) of 4.67 and demonstrates a 54.25% relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: https://wflrz123.github.io/

Abstract

Dysarthric Speech Recognition

Explore similar work

End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation

DARS: Dysarthria-Aware Rhythm-Style Synthesis for ASR Enhancement

Adapting Foundation ASR Models to Dysarthric Speech: A Case Study