Semi-supervised learning in automatic speech recognition (ASR) typically relies on pseudo-labeling, which often suffers from confirmation bias and error accumulation due to noisy supervision. To address this limitation, we propose ReHear, a framework for iterative pseudo-label refinement that integrates an instruction-tuned, audio-aware large language model (LLM) into the self-training loop. Unlike conventional text-based correctors, our approach conditions the LLM on both the ASR hypothesis and the source audio, allowing it to recover phonetically accurate transcripts even from severe recognition errors. These refined pseudo-labels serve as high-fidelity targets for fine-tuning the ASR model in an iterative cycle. Experimental results across diverse benchmarks demonstrate that ReHear effectively mitigates error propagation, consistently outperforming both supervised and pseudo-labeling baselines.
Figures & tables
Figure 1: Overview of the proposed ReHear framework. The pipeline employs an audio-aware LLM to refine hypotheses via multimodal context, subsequently using these refined pseudo-labels to iteratively fine-tune the ASR model on data.
Split
Earnings-21
Earnings-22
SPGISpeech
AMI
Labeled Train
10.76h
21.81h
19.46h
16.37h
Labeled Val
3.07h
9.87h
9.24h
8.94h
Labeled Test
4.93h
12.41h
11.43h
8.68h
Unlabeled
18.86h
63.67h
58.43h
61.49h
Total
37.63h
107.75h
98.55h
95.48h
Avg. Duration
24.76s
24.13s
9.16s
2.56s
Table 1: Detailed statistics of the datasets utilized in experiments, reporting the total duration for each partition split along with the average segment duration.
Method
Earnings-21
Earnings-22
SPGISpeech
AMI (IHM)
Test
Unlabeled
Test
Unlabeled
Test
Unlabeled
Test
Unlabeled
ISL
6.76 ±0.17
0 8.57 ±0.19
12.54 ±0.11
11.43 ±0.09
2.41 ±0.01
2.42 ±0.02
0 9.66 ±0.24
10.26 ±0.29
IPL
8.02 ±0.18
10.69 ±0.10
14.02 ±0.17
13.30 ±0.08
2.83 ±0.05
2.98 ±0.08
16.52 ±0.26
19.31 ±0.13
IPL + Rule
7.59 ±0.25
10.30 ±0.17
13.64 ±0.12
13.10 ±0.08
2.57 ±0.03
2.59 ±0.04
11.00 ±2.05
12.35 ±1.86
ReHear
6.34 ±0.26
0 7.87 ±0.18
12.01 ±0.08
10.63 ±0.11
2.36 ±0.01
2.23 ±0.03
0 9.45 ±0.23
10.30 ±0.15
ReHear + Rule
6.35 ±0.13
0 7.79 ±0.08
11.82 ±0.16
10.63 ±0.16
2.37 ±0.04
2.28 ±0.04
0 9.35 ±0.24
10.09 ±0.22
Table 2: Word Error Rate (WER, %) comparison across all datasets. We report the mean and standard deviation (subscript) on the labeled test and unlabeled sets. Bold indicates the best result, and underline indicates the second best.
Configuration
Test
Unlabeled
Text Only
7.74 ±0.11
10.46 ±0.12
Text + Audio
6.34 ±0.24
0 8.02 ±0.24
Audio + Text
6.34 ±0.26
0 7.87 ±0.18
Table 3: Ablation study on input modality and positioning using Earnings-21. We compare WER (%) on the labeled test and unlabeled sets for text-only inputs versus different audio integration strategies.
Strategy
Configuration
Test
Unlabeled
Greedy
N=1
6.59 ±0.36
8.67 ±0.34
Beam
N=3
6.21 ±0.14
7.89 ±0.19
Beam
N=5
6.34 ±0.26
7.87 ±0.18
Sampling
T=0.7
6.54 ±0.22
8.72 ±0.23
Table 4: Ablation study on LLM decoding strategies using Earnings-21. We compare WER (%) on the labeled test and unlabeled sets for deterministic methods (greedy, beam search with varying widths) versus stochastic sampling.
Iteration
Without Decay
With Decay
Test
Unlabeled
Test
Unlabeled
0
8.98 ±0.00
10.99 ±0.00
8.98 ±0.00
10.99 ±0.00
1
6.48 ±0.20
0 8.06 ±0.12
6.30 ±0.18
0 8.06 ±0.29
2
6.35 ±0.10
0 7.92 ±0.15
6.43 ±0.22
0 7.94 ±0.13
3
6.51 ±0.21
0 8.07 ±0.17
6.42 ±0.16
0 7.99 ±0.12
4
6.75 ±0.26
0 8.26 ±0.29
6.48 ±0.16
0 8.05 ±0.19
Table 5: Ablation study on iterative dynamics using Earnings-21. We compare WER (%) on the labeled test and unlabeled sets across iterations, with and without the training decay strategy.