Scaling Audio Models Efficiently: Joint Optimization of Scale, Resolution, Adaptation, Precision, and Sparsity

Vyom Agarwal; Mokshda Gangrade; Siddharth Pal; Jerry Wu

cs.SDJun 22, 2026

Scaling Audio Models Efficiently: Joint Optimization of Scale, Resolution, Adaptation, Precision, and Sparsity

Authors: Vyom Agarwal, Mokshda Gangrade, Siddharth Pal, Jerry Wu

Organizations: Computer, Mathematical, and Natural Sciences, University of Maryland, College Park, MD, USA · RTX BBN Technologies, Columbia, MD, USA · Electrical and Computer Engineering, University of Maryland, College Park, MD, USA

Abstract

Large automatic speech recognition (ASR) models such as Whisper must be deployed across hardware with widely varying memory and inference-speed constraints. We present a compression framework that jointly parametrizes Whisper deployment along \emph{six} dimensions: model size, temporal resolution, encoder token stride, low-rank adaptation capacity, weight precision and sparsity pattern. All axes are jointly optimized using NSGA-III with respect to three deployment objectives: word error rate (WER), inference FLOPs, and memory footprint. Across 50 of the 1,680 candidate configurations evaluated, we characterize the conditional effect of each axis and identify compression combinations that dominate naive single-axis scaling, while finding that 1:4 structured sparsity fails to recover acceptable accuracy under the tested recovery budgets. We report measured WER and resident memory, use analytical EffFLOPs as the search-time compute surrogate, and separately validate representative inference configurations using measured real-time factor (RTF).

Explore similar work

Performance and Complexity Trade-off Optimization of Speech Models During Training
Jan 20, 2026Esteban Gómez, Tom BäckströmLarge Models Stochastic Gradient Descent
Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR
Jul 6, 2026Ho Lam Chung, Yiming Chen, Dau-Cheng Lyu +2Speech Large Language Models Accurate Transcription
GrainSpeech: Less Context, More Detail for Compact Speech Synthesis
Sep 16, 2026Zitao Liang, Chang GaoAcoustic Representation Self-Supervised

Jan 20, 2026 · cs.SDJ/K move · Enter open · S save

Performance and Complexity Trade-off Optimization of Speech Models During Training

Esteban Gómez, Tom Bäckström

Department of Information and Communications Engineering, Aalto University, Espoo, Finland

In speech machine learning, neural network models are typically designed by choosing an architecture with fixed layer sizes and structure. These models are then trained to maximize performance on metrics aligned with the task's objective. While the overall architecture is usually guided by prior knowledge of the task, the sizes of individual layers are often chosen heuristically. However, this approach does not guarantee an optimal trade-off between performance and computational complexity; consequently, post hoc methods such as weight quantization or model pruning are typically employed to reduce computational cost. This occurs because stochastic gradient descent (SGD) methods can only optimize differentiable functions, while factors influencing computational complexity, such as layer sizes and floating-point operations per second (FLOP/s), are non-differentiable and require modifying the model structure during training. We propose a reparameterization technique based on feature noise injection that enables joint optimization of performance and computational complexity during training using SGD-based methods. Unlike traditional pruning methods, our approach allows the model size to be dynamically optimized for a target performance-complexity trade-off, without relying on heuristic criteria to select which weights or structures to remove. We demonstrate the effectiveness of our method through three case studies, including a synthetic example and two practical real-world applications: voice activity detection and audio anti-spoofing. The code related to our work is publicly available to encourage further research.

Large Models Stochastic Gradient Descent Trade-Off

Open paper Ask about it PDF

Abstract

Explore similar work

Performance and Complexity Trade-off Optimization of Speech Models During Training

Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR

GrainSpeech: Less Context, More Detail for Compact Speech Synthesis