End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation

Philippe Gonzalez; Vera Margrethe Frederiksen; Torsten Dau; Tobias May

eess.ASMar 20, 2026

End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation

Authors: Philippe Gonzalez, Vera Margrethe Frederiksen, Torsten Dau, Tobias May

Organizations: The authors are with the Department of Health Technology, Technical University of Denmark, 2800 Lyngby, Denmark.

Abstract

A multi-task learning framework is proposed for optimizing a single deep neural network (DNN) for joint noise reduction (NR) and hearing loss compensation (HLC). A distinct training objective is defined for each task, and the DNN predicts two time-frequency masks. During inference, the amounts of NR and HLC can be adjusted independently by exponentiating each mask before combining them. In contrast to recent approaches that rely on training an auditory-model emulator to define a differentiable training objective, we propose an auditory model that is inherently differentiable, thus allowing end-to-end optimization. The audiogram is provided as an input to the DNN, thereby enabling listener-specific personalization without the need for retraining. Results show that the proposed approach not only allows adjusting the amounts of NR and HLC individually, but also improves objective metrics compared to optimizing a single training objective. It also outperforms a cascade of two DNNs that were separately trained for NR and HLC, and shows competitive HLC performance compared to a traditional hearing-aid prescription. To the best of our knowledge, this is the first study that uses an auditory model to train a single DNN for both NR and HLC across a wide range of listener profiles.

Figures & tables

Panel of Figure 1 — Fig. 1: Traditional single-DNN frameworks for (a) NR-only, (b) HLC-only, and (c) joint NR and HLC using auditory models.

Panel of Figure 2 — Fig. 2: Proposed multi-task framework for adjustable joint NR and HLC using a single DNN and two auditory models. During training (a), the DNN predicts both a denoised and a compensated signal, and a distinct training objective is defined for each output. During inference (b), the amount of NR and HLC can be adjusted independently by exponentiating each predicted time-frequency mask.

Fig. 3: BSRNN-based DNN backbone. The band-split module projects

K

frequency bands with increasing width

\{G_{i}\}_{i=1}^{K}

onto a fixed number of channels

N

. Interleaved sequence modelling across time and bands is performed using residual LSTM blocks in

L

layers. Features extracted from the audiogram are included in each layer using FiLM. The band-merging module projects each band back to the original frequency resolution and outputs real- or complex-valued masks

M_{\mathrm{NR}}

and

M_{\mathrm{HLC}}

Panel of Figure 4 — Fig. 4: Overview of the proposed differentiable auditory model.

Fig. 5: Objective metrics for real- or complex-valued masks and MSE or MAE loss. For PESQ, ESTOI, and SNR, a NH audiogram is provided to systems providing HLC. For HASPI and HASQI, results are averaged over all audiograms. Both

\alpha_{\mathrm{NR}}

and

\alpha_{\mathrm{HLC}}

are set to 1 for DNN-CNRHLC.

Fig. 6: Objective metrics as a function of $\alpha_{\mathrm{NR}}$ and $\alpha_{\mathrm{HLC}}$ . For PESQ, ESTOI, and SNR, a NH audiogram is provided to the DNN. For HASPI and HASQI, results are averaged over all audiograms.

Fig. 7: HASPI and HASQI as a function of $\alpha_{\mathrm{NR}}$ and $\alpha_{\mathrm{HLC}}$ for the NH audiogram and HI audiograms N1, N3, N6, and S2.

	Parameters (M)	PESQ $\uparrow$	ESTOI (%) $\uparrow$	SDR $\uparrow$	HASPI (%) $\uparrow$	HASQI (%) $\uparrow$	NRMSE (%) $\downarrow$
Noisy	–	$1.24$	$60.3$	$3.0$	$46.2$	$19.9$	$26.6$
Noisy + NAL-R	–	$1.24$	$60.2$	$3.0$	$72.5$	$56.4$	$26.6$
Clean + NAL-R	–	$4.64$	$100.0$	$\infty$	$85.4$	$99.5$	$24.8$
DNN-SDR	$3.42$	$2.26$	84.6	13.7	$50.6$	$24.4$	$26.3$
DNN-SDR + NAL-R	$3.42$	$2.26$	84.6	13.7	$75.9$	71.9	$26.3$
DNN-NR	$3.42$	$2.28$	$83.4$	$13.3$	$49.8$	$23.7$	$26.6$

TABLE I: Objective metrics for all DNN-based systems, NAL-R, and various combinations. For PESQ, ESTOI, and SNR, a NH audiogram is provided to systems providing HLC. For HASPI and HASQI, results are averaged over all audiograms. DNN-CNRHLC uses

(\alpha_{\mathrm{NR}},\alpha_{\mathrm{HLC}})=(1,1)

for PESQ, ESTOI, and SNR, and

(0.75,1)

for HASPI, HASQI, and NRMSE.

Fig. 8: Example predicted NR mask $M_{\mathrm{NR}}$ , HLC mask $M_{\mathrm{HLC}}$ , combined mask $M$ , and output spectrogram $\hat{Y}$ for the NH audiogram and HI audiograms N1, N2 and N3. Both $\alpha_{\mathrm{NR}}$ and $\alpha_{\mathrm{HLC}}$ are set to 1. The masks are complex-valued and their magnitude is plotted in dB. The magnitude of the combined mask $M$ is thresholded using a minimum gain $G_{\min}$ of $-25\text{\,}\mathrm{dB}$ .

Speech Enhancement

Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.

See all 127 papers

The Differentiable Auditory Loop (DAL): An ML Framework for Hyper-Personalized Hearing Aids
Jun 2, 2026Alejandro Ballesta Rosen, Jason Mikiel-Hunter, Julian Maclaren +3Speech Enhancement
1Google Research Australia · 2Macquarie University
An Interpretable, Controllable Time-Varying IIR Denoiser for On-Device Assistive Hearing
Mar 3, 2026Riccardo Rota, Kiril Ratmanski, Jozef Coldenhoff +1Speech Enhancement On-Device Inference
Logitech Europe S.A., Switzerland · EPFL ( ´Ecole Polytechnique F´ed´erale de Lausanne), Switzerland
Automatic Audio Equalization with Semantic Embeddings
Jul 26, 2026Eloi Moliner, Vesa Välimäki, Konstantinos Drossos +1Audio Editing
Acoustics Lab, Department of Information and Communications Engineering, Aalto University, Espoo, Finland · Nokia Technologies, Tampere/Espoo, Finland

Jun 2, 2026J/K move · Enter open · S save

The Differentiable Auditory Loop (DAL): An ML Framework for Hyper-Personalized Hearing Aids

Alejandro Ballesta Rosen, Jason Mikiel-Hunter, Julian Maclaren, Jack Collins+2

1Google Research Australia · 2Macquarie University

Speech Enhancement

Open paper AskPDF arXiv

Conventional hearing aids rely on fixed, frequency-dependent amplification and compression to manage reduced sensitivity, which often fails to provide sufficient listening support in complex environments, such as situations with multiple speakers (the ``cocktail party'' problem). To more comprehensively address the underlying encoding dysfunctions of hearing loss, we introduce the Differentiable Auditory Loop (DAL), a new open-source framework for personalized hearing aid design and fitting. Our first implementation of DAL incorporates CARFAC, a differentiable model of human cochlear function, which we ported to JAX, to optimize a deep neural network to match impaired auditory neural activity patterns with a normal-hearing reference. To build a hearing aid with the fine-grained spectro-temporal signal processing required, we adopt SEANet, a waveform-to-waveform fully convolutional UNet generator. We fine-tune the network by comparing the outputs of a CARFAC model fitted to normal hearing with that of a CARFAC model fitted to match each subject's individual hearing impairment. The comparison is done using loss functions derived from the respective CARFAC neural activity pattern (NAP) outputs and stabilized auditory images (SAIs), the latter providing a 2D representation that captures phase-insensitive temporal structure in the auditory nerve output. Through gradient descent, the SEANet model learns to both denoise the input and compensate for the hearing loss modelled by the impaired CARFAC model. Across neural-representation and signal-fidelity metrics, the DAL-optimized SEANet model outperformed the tested master hearing aid (MHA) baselines. The DAL framework provides a practical path toward model-based, machine-learning-driven personalization of hearing aid signal processing. Next steps include hardware deployment to enable real-world clinical testing.

Abstract

Speech Enhancement

Explore similar work

The Differentiable Auditory Loop (DAL): An ML Framework for Hyper-Personalized Hearing Aids

An Interpretable, Controllable Time-Varying IIR Denoiser for On-Device Assistive Hearing

Automatic Audio Equalization with Semantic Embeddings