The robustness of deep neural networks (DNNs) can be certified through their Lipschitz continuity, which has made the construction of Lipschitz-continuous DNNs an active research field. However, DNNs for audio processing have not been a major focus due to their poor compatibility with existing results. In this paper, we consider the amplitude modifier (AM), a popular architecture for handling audio signals, and propose its Lipschitz-continuous variants, which we refer to as LipsAM. We prove a sufficient condition for an AM to be Lipschitz continuous and propose two architectures as examples of LipsAM. The proposed architectures were applied to a Plug-and-Play algorithm for speech dereverberation, and their improved stability is demonstrated through numerical experiments.
Figures & tables
Figure 1: Amplitude-Modifying DNNs. Red layers are trainable, blue layers are introduced to enforce Lipschitz continuity. Layers “mul” represent the element-wise multiplication. While the popular architectures, AM-SE and AM-RE, are generally not Lipschitz continuous, Lipschitz constants of our LipsAM-SE and LipsAM-RE can be bounded by (Lip(S))2+1 and Lip(R)+1 , respectively.
Figure 2: Numerical estimate of the value B in Eq. ( 13 ) for each architecture. Upper rows are that of AM-SE in Eq. ( 4 ) and LipsAM-SE in Eq. ( 9 ). Lower rows are that of AM-RE in Eq. ( 5 ) and LipsAM-RE in Eq. ( 10 ). Dots indicate results of 100 trials, and maximum result is marked with large circle. Solid lines show the theoretical bound in Theorem 5 . Areas exceeding termination threshold are darkened.
Figure 3: Parameter λ vs SI-SNR after 500 iterations. Solid colored lines represents the proposed LipsAMs, and dotted colored lines represent the conventional AMs. Use of orthogonal layers is indicated by (Ortho). Gray lines show ℓ1 -norm-based method. Best parameters for each model are shown by the markers. A close-up view of left is on the right. Missing points indicate that algorithms diverged.
Denoiser D in Eq. ( 15 )
SI-SNR ( ↑ )
PESQ ( ↑ )
STOI ( ↑ )
ViSQOL ( ↑ )
AM
-SE
N/A
N/A
N/A
N/A
LipsAM
-SE
16.61
2.91
0.91
3.64
AM
-SE
(Ortho)
9.54
2.30
0.88
3.10
LipsAM
-SE
(Ortho)
14.44
2.68
0.93
3.75
AM
-RE
17.98
3.21
0.97
4.21
LipsAM
-RE
20.57
3.14
0.97
4.21
Table 1: Evaluation results with best parameter λ marked in Fig. 3 . N/A indicates that algorithm diverged. Bold and underlined numbers indicate the best and second-best values, respectively.
Figure 4: Transition of amount of update ∥Δx∥2 . In each box, results of LipsAMs with best λ in Fig. 3 were compared with that of AMs with the same λ as LipsAMs.
The Lipschitz continuity of deep neural networks (DNNs) is essential for establishing theoretical guarantees regarding their behavior. From both theoretical and practical perspectives, various methods have been proposed to construct Lipschitz-continuous architectures and control their Lipschitz constants. However, several DNN architectures common in audio signal processing fall outside the scope of existing theoretical frameworks, hindering the development of Lipschitz-continuous models in acoustic applications. In particular, despite their widespread adoption, DNNs that separately process the magnitude and phase of complex-valued signals cannot be Lipschitz continuous under existing frameworks. In this paper, to address this limitation, we establish a theoretical foundation for constructing amplitude modifiers (AMs), a class of DNN architectures that operate solely on the magnitude of a complex-valued input, with provable Lipschitz continuity. Specifically, we derive a necessary and sufficient condition for an AM to be Lipschitz continuous and propose LipsAMs (Lipschitz-continuous AMs) corresponding to common architectures for audio signals, including time-frequency masking. Furthermore, we develop an efficient framework for evaluating their Lipschitz constants and analytically derive these constants for some of the proposed architectures. As an application, we propose CoReM-LipsAM (Controlled Residual Maps via LipsAM) for plug-and-play (PnP) audio signal recovery, integrating a DNN as a data-driven prior within a model-based signal processing algorithm. The convergence of the obtained PnP algorithm is structurally guaranteed by the CoReM-LipsAM architecture and empirically validated through speech dereverberation experiments.
Kazuki Matsumoto, Ren Uchida, Natsuki Yoshino +1
Tokyo University of Agriculture and Technology, Tokyo 184-8588, Japan
Large audio language models (LALMs) are a class of foundation models for audio understanding. Existing LALMs tend to degrade significantly in real-world noisy acoustic conditions where speech and non-speech sounds interfere. While noise-aware fine-tuning can improve robustness, it requires task-specific noisy data and expensive retraining, limiting scalability. To address this issue, we propose Focus-Then-Listen (FTL), a plug-and-play audio enhancer that improves LALMs' noise robustness. Specifically, FTL first separates the input waveform into speech and non-speech, and a modality router is applied to predict the target audio modality (e.g., speech) based on the user's instruction. Finally, a modality-aware fusion block generates a task-adaptive enhanced signal for improved downstream perception and reasoning. Experiments across multiple LALMs and tasks show that FTL improves performance across different noise levels without fine-tuning on LALMs.
Han Yin, Yang Xiao, Younghoo Kwon +2
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea · University of Melbourne, Australia
Differentiable audio processors are habitually designed and optimised in machine-learning frameworks, but deploying them as real-time audio effects still often requires non-automatic implementation in a dedicated digital signal processing language. The translation is error-prone, demands an onerous verification process, and detaches research prototypes from usable production tools. That being so, we present ADAC, a compiler that lowers a trained model to a framework-agnostic intermediate representation and emits efficient FAUST code whose impulse response matches the source model to within floating-point arithmetic noise, direct paths included. The optimisation loop is made audible by replacing the model in a running plugin after each gradient step. The exported processor carries a small set of macro-controls that leave its stability intact. A stability certificate computed from the shipped parameters is checked before the plugin is built. At the demonstration, a feedback delay network is trained and exported to a working plugin.
Facundo Franchino, Sebastian J. Schlecht
Massachusetts Institute of Technology Cambridge, MA · Artificial Audio, Multimedia Communications and Signal Processing Friedrich-Alexander-Universität Erlangen-Nürnberg Erlangen, Germany