eess.ASOct 9, 2026

Does Financial Speech Need Continual Learning Regularization? A Study of EWC and mini-KFAC on Earnings Call ASR

Authors: Thai Quoc On, Homayoon Beigi

Organizations: Department of Computer Science Columbia University, NYC, NY, USA · Mechanical Engineering Dept. and Electrical Engineering Dept. Columbia University, NYC, NY, USA · and Recognition Technologies, Inc. South Salem, NY, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Earnings call transcription presents a realistic continual learning scenario where Automatic Speech Recognition (ASR) systems are updated with new recordings but risk catastrophic forgetting of previously learned financial speech patterns. Adapting Whisper-small from a general-domain speech corpus to the Earnings22 benchmark reveals the opposite: all evaluated methods exhibit improvement on held-out financial speech from the same domain during adaptation rather than forgetting. This task complementarity makes regularization counterproductive. Nevertheless, when regularization is warranted, a layer-wise mini-Kronecker-Factored Approximate Curvature (mini-KFAC) approximation seems to achieve lower word error rates (WERs) and more stable optimization than standard diagonal Elastic Weight Consolidation (EWC). The results suggest that task complementarity should be assessed before applying continual learning constraints, and prefer mini-KFAC EWC over diagonal EWC when needed.

Figures & tables

Figure 1: The architecture of the continual learning experimental pipeline.
MethodTask B WERTask A WERForgetting
 (%)(%)(%)
Baseline31.94±1.7331.94\pm 1.7320.68±1.6320.68\pm 1.63−4.87±2.47†-4.87\pm 2.47^{\dagger}
EWC (Diagonal)35.23±4.9135.23\pm 4.9121.88±2.0821.88\pm 2.08−3.66±1.95†-3.66\pm 1.95^{\dagger}
Mini-KFAC EWC34.26±3.5734.26\pm 3.5721.52±2.1721.52\pm 2.17−4.02±2.41†-4.02\pm 2.41^{\dagger}
Table 1: Results across five seeds (mean ±\pm std). Task B WER is the best validation WER on the fine-tuning partition. Task A WER is evaluated on the best Task B checkpoint. Forgetting is the change in Task A WER relative to pre-B; negative values indicate improvement. Superscripts indicate statistical significance. † denotes a one-sample, one-tailed tt -test showing that forgetting is significantly below zero ( p<0.05p<0.05 ), see Appendix D .
Figure 2: Task A WER evolution during Stage B fine-tuning.
Figure 3: Task B WER evolution during Stage B fine-tuning.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: EWC penalty evolution during Stage B adaptation.
Figure 5: Total training loss ( Ltotal=LASR+LEWC\mathcal{L}_{\text{total}}=\mathcal{L}_{ASR}+\mathcal{L}_{EWC} ) during Stage B adaptation.
Figure 6: ASR loss component ( LASR\mathcal{L}_{ASR} ) during Stage B adaptation.
Figure 7: EWC regularization loss component ( LEWC\mathcal{L}_{EWC} ) during Stage B adaptation. The baseline shows zero EWC loss throughout, as no regularization is applied.
ComparisonMetricΔ\boldsymbol{\Delta} (%)t\boldsymbol{t}dfp\boldsymbol{p}d\boldsymbol{d}
mini-KFAC vs EWCTask A WER−0.36-0.36−0.27-0.278.08.00.7960.796−0.17-0.17
 Task B WER−0.97-0.97−0.36-0.367.37.30.7310.731−0.23-0.23
 Forgetting−0.36-0.36−0.26-0.267.77.70.8020.802−0.16-0.16
Baseline vs EWCTask A WER−1.20-1.20−1.02-1.027.67.60.3410.341−0.64-0.64
 Task B WER−3.29-3.29−1.41-1.415.05.00.2170.217−0.89-0.89
 Forgetting−1.21-1.21−0.86-0.867.67.60.4160.416−0.54-0.54
Appendix
Table 2: Welch’s two-sample tt -tests ( n=5n=5 seeds per method, two-tailed). dd = Cohen’s dd (pooled std). No comparison is significant at α=0.05\alpha=0.05 .
Methodt\boldsymbol{t}p\boldsymbol{p} (one-tailed)Sig.
Baseline−4.41-4.410.0060.006p<0.01p<0.01
EWC (Diagonal)−4.20-4.200.0070.007p<0.01p<0.01
Mini-KFAC EWC−3.73-3.730.0100.010p<0.05p<0.05
Appendix
Table 3: One-sample, one-tailed tt -tests against H0H_{0} : Forgetting ≥0\geq 0 , n=5n=5 seeds per method, df=4df=4 .
More in this topic

Kronecker-Factored Approximation

See all 14 papers

Explore similar work

CardsList

  1. Earnings25: A Comprehensive 500-Hour Speech Benchmark for Finance

    Jul 26, 2026Denglin Jiang, Haoran Zhou, Anshul Wadhawan +7ASR EvaluationSpeech Recognition Benchmarks

    Bloomberg, United States

  2. Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization

    May 25, 2026Meshal Alamr, Hassan Alqaeri, Abdullah AldahlawiFine-TuningArabic NLP

    Thaka, Advanced AI and Information Technology Riyadh, Saudi Arabia

  3. Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR

    Jun 12, 2026Henri-Leon Kordt, Theresa Pekarek Rosin, Jae Hee Lee +1Speech ProcessingAutomatic Speech Recognition

    Knowledge Technology, Department of Informatics, University of Hamburg, Germany