Does Financial Speech Need Continual Learning Regularization? A Study of EWC and mini-KFAC on Earnings Call ASR
Authors: Thai Quoc On, Homayoon Beigi
Organizations: Department of Computer Science Columbia University, NYC, NY, USA · Mechanical Engineering Dept. and Electrical Engineering Dept. Columbia University, NYC, NY, USA · and Recognition Technologies, Inc. South Salem, NY, USA
Earnings call transcription presents a realistic continual learning scenario where Automatic Speech Recognition (ASR) systems are updated with new recordings but risk catastrophic forgetting of previously learned financial speech patterns. Adapting Whisper-small from a general-domain speech corpus to the Earnings22 benchmark reveals the opposite: all evaluated methods exhibit improvement on held-out financial speech from the same domain during adaptation rather than forgetting. This task complementarity makes regularization counterproductive. Nevertheless, when regularization is warranted, a layer-wise mini-Kronecker-Factored Approximate Curvature (mini-KFAC) approximation seems to achieve lower word error rates (WERs) and more stable optimization than standard diagonal Elastic Weight Consolidation (EWC). The results suggest that task complementarity should be assessed before applying continual learning constraints, and prefer mini-KFAC EWC over diagonal EWC when needed.
Figures & tables
Figure 1: The architecture of the continual learning experimental pipeline.
Method
Task B WER
Task A WER
Forgetting
(%)
(%)
(%)
Baseline
31.94±1.73
20.68±1.63
−4.87±2.47†
EWC (Diagonal)
35.23±4.91
21.88±2.08
−3.66±1.95†
Mini-KFAC EWC
34.26±3.57
21.52±2.17
−4.02±2.41†
Table 1: Results across five seeds (mean ± std). Task B WER is the best validation WER on the fine-tuning partition. Task A WER is evaluated on the best Task B checkpoint. Forgetting is the change in Task A WER relative to pre-B; negative values indicate improvement. Superscripts indicate statistical significance. † denotes a one-sample, one-tailed t -test showing that forgetting is significantly below zero ( p<0.05 ), see Appendix D .
Figure 2: Task A WER evolution during Stage B fine-tuning.
Figure 3: Task B WER evolution during Stage B fine-tuning.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: EWC penalty evolution during Stage B adaptation.
Figure 5: Total training loss ( Ltotal=LASR+LEWC ) during Stage B adaptation.
Figure 6: ASR loss component ( LASR ) during Stage B adaptation.
Figure 7: EWC regularization loss component ( LEWC ) during Stage B adaptation. The baseline shows zero EWC loss throughout, as no regularization is applied.
Comparison
Metric
Δ (%)
t
df
p
d
mini-KFAC vs EWC
Task A WER
−0.36
−0.27
8.0
0.796
−0.17
Task B WER
−0.97
−0.36
7.3
0.731
−0.23
Forgetting
−0.36
−0.26
7.7
0.802
−0.16
Baseline vs EWC
Task A WER
−1.20
−1.02
7.6
0.341
−0.64
Task B WER
−3.29
−1.41
5.0
0.217
−0.89
Forgetting
−1.21
−0.86
7.6
0.416
−0.54
Appendix
Table 2: Welch’s two-sample t -tests ( n=5 seeds per method, two-tailed). d = Cohen’s d (pooled std). No comparison is significant at α=0.05 .
Method
t
p (one-tailed)
Sig.
Baseline
−4.41
0.006
p<0.01
EWC (Diagonal)
−4.20
0.007
p<0.01
Mini-KFAC EWC
−3.73
0.010
p<0.05
Appendix
Table 3: One-sample, one-tailed t -tests against H0 : Forgetting ≥0 , n=5 seeds per method, df=4 .
We introduce Earnings25, a finance-domain benchmark for evaluating automatic speech recognition (ASR) on English-language earnings calls under realistic conditions. Earnings25 comprises two complementary test sets: (i) testset-full, 498 hours of full English-language S&P 500 earnings calls from Q4 2025, and (ii) testset-segmented, a 46-hour industry-balanced set of 290 segments sampled from English-language U.S. earnings calls in 2025. The benchmark provides aligned transcripts and structured metadata, including speaker roles, industry labels, and call structure, enabling speaker- and industry-aware evaluation beyond aggregate word error rate (WER). We report reproducible baselines for Whisper and Parakeet-TDT using standardized scoring.
We describe the winning system for Task 2 of the KSAA-2026 Shared Task on Arabic Speech Dictation with Automatic Diacritization. The task requires producing fully diacritized Arabic text from speech audio and undiacritized transcripts, with only 2,327 training samples available and no external data permitted. Our system fine-tunes CATT-Whisper, a character-level multimodal model combining a pretrained CATT text encoder with a frozen Whisper speech encoder. The key to our approach is training regularization: R-Drop consistency regularization, Optuna-optimized hyperparameters with high weight decay, and Focal Loss. At inference, we average 200 stochastic forward passes across four model checkpoints using Monte Carlo Dropout at the softmax probability level. The system achieves 23.26% WER on the primary leaderboard metric (with case endings, including no-diacritic positions), placing 1st among all participants.
Meshal Alamr, Hassan Alqaeri, Abdullah Aldahlawi
Thaka, Advanced AI and Information Technology Riyadh, Saudi Arabia
Despite advances in large-scale Automatic Speech Recognition (ASR), disfluent speech remains challenging, as state-of-the-art systems are often optimized to omit disfluencies, leading to information loss and hallucinations. Prior work has focused on verbatim transcription and the integration of disfluency markers, but adapting models on limited datasets can lead to catastrophic forgetting of general-domain knowledge. We address this gap by leveraging continual learning (CL) with explicit disfluency tokens. We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms, and then continue training on additional datasets with varying disfluency distributions. Through a detailed analysis of model dynamics during training, we identify a trade-off between marker learning and ASR performance, and a consistent cross-attention head mechanism shared across CL methods.
Henri-Leon Kordt, Theresa Pekarek Rosin, Jae Hee Lee +1
Knowledge Technology, Department of Informatics, University of Hamburg, Germany