Speech Processing

Follow this topic Ask about this topic

Momentum

48 papers in the last four weeks, up 433% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 295

Mar 4, 2026eess.AS

FlowW2N: Whispered-to-Normal Speech Conversion via Flow-Matching

Whispered-to-normal (W2N) speech conversion aims to reconstruct missing phonation from whispered input while preserving content and speaker identity. This task is challenging due to temporal misalignment between whisper and voiced recordings and lack of paired data. We propose FlowW2N, a conditional flow matching approach that trains exclusively on synthetic, time-aligned whisper-normal pairs and conditions on domain-invariant features. We exploit high-level ASR embeddings that exhibits strong invariance between synthetic and real whispered speech, enabling generalization to real whispers despite never observing it during training. We verify this invariance across ASR layers and propose a selection criterion optimizing content informativeness and cross-domain invariance. Our method achieves SOTA intelligibility on the CHAINS and wTIMIT datasets, reducing Word Error Rate by 26-46% relative to prior work while using only 10 steps at inference and requiring no real paired data, validated by a subjective listening study and F0-contour analysis.

Fabian Ritter-Gutierrez, Md Asif Jalal, Pablo Peso Parada +5

Samsung Electronics R&D Institute UK (SRUK), London, United Kingdom · Samsung Electronics, Mobile eXperience Business, Suwon, Republic of Korea

Speech Processing Voice Conversion

Mar 3, 2026cs.SD

Single Microphone Own Voice Detection based on Simulated Transfer Functions for Hearing Aids

This paper presents a simulation-based approach to own voice detection (OVD) in hearing aids using a single microphone. While OVD can significantly improve user comfort and speech intelligibility, enabling reliable OVD with a single microphone is desirable for simplifying hardware and reducing power consumption in compact hearing devices. However, most existing solutions rely on multiple microphones or additional sensors, increasing device complexity and cost. To enable ML-based OVD without requiring costly transfer-function measurements, we propose a data augmentation strategy based on simulated acoustic transfer functions (ATFs) that expose the model to a wide range of spatial propagation conditions. A transformer-based classifier is trained using analytically generated ATFs and further fine-tuned using numerically simulated ATFs with increasing geometric realism. This hierarchical adaptation enables the model to refine its spatial understanding while maintaining generalization. Experimental results show 95.52% accuracy on simulated head-and-torso test data and 90.02% accuracy for one-second speech segments, demonstrating robustness to short durations. When evaluated on real-world hearing-aid recordings, few-shot fine-tuning using a small subset achieves 91% accuracy, demonstrating that limited real-world data can effectively adapt the simulation-trained model. These results highlight the potential of simulation-based training for enabling practical single-microphone OVD systems in hearing aids.

Mathuranathan Mayuravaani, W. Bastiaan Kleijn, Andrew Lensen +1

Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation Speech Processing

Mar 2, 2026cs.SD

When Spoof Detectors Travel: Evaluation Across 66 Languages in the Low-Resource Language Spoofing Corpus

We introduce LRLspoof, a large-scale multilingual synthetic-speech corpus for cross-lingual spoof detection, comprising 2,732 hours of audio generated with 24 open-source TTS systems across 66 languages, including 45 low-resource languages under our operational definition. To evaluate robustness without requiring target-domain bonafide speech, we benchmark 11 publicly available countermeasures using threshold transfer: for each model we calibrate an EER operating point on pooled external benchmarks and apply the resulting threshold, reporting spoof rejection rate (SRR). Results show model-dependent cross-lingual disparity, with spoof rejection varying markedly across languages even under controlled conditions, highlighting language as an independent source of domain shift in spoof detection. The dataset is publicly available at \href{https://huggingface.co/datasets/lab260/LRLspoof}{\textbf{\underline{\textit{HuggingFace}}}} and \href{https://modelscope.cn/datasets/lab260/LRLspoof}{\textbf{\underline{\textit{ModelScope}}}}

Kirill Borodin, Vasiliy Kudryavtsev, Maxim Maslov +2

lab260, Moscow Technical University of Communications and Informatics, Russia · BitmanagerAI, UAE

Audio Deepfake Detection Speech Processing Low-Resource Language Processing Cross-Lingual Transfer

Feb 24, 2026cs.SD

Quantifying Dimensional Independence in Speech: An Information-Theoretic Framework for Disentangled Representation Learning

Speech signals encode emotional, linguistic, and pathological information within a shared acoustic channel; however, disentanglement is typically assessed indirectly through downstream task performance. We introduce an information-theoretic framework to quantify cross-dimension statistical dependence in handcrafted acoustic features by integrating bounded neural mutual information (MI) estimation with non-parametric validation. Across six corpora, cross-dimension MI remains low, with tight estimation bounds (

< 0.15

nats), indicating weak statistical coupling in the data considered, whereas Source--Filter MI is substantially higher (0.47 nats). Attribution analysis, defined as the proportion of total MI attributable to source versus filter components, reveals source dominance for emotional dimensions (80%) and filter dominance for linguistic and pathological dimensions (60% and 58%, respectively). These findings provide a principled framework for quantifying dimensional independence in speech.

Bipasha Kashyap, Björn W. Schuller, Pubudu N. Pathirana

NSBE Research Lab, School of Engineering, Deakin University, Australia · Chair of Health Informatics (CHI), TUM University Hospital, Germany · Group on Language, Audio & Music (GLAM), Imperial College London, UK

Representation Disentanglement Speech Processing

Jan 21, 2026cs.AI

LOGIC: Efficient and Robust Contextual Biasing for Speech LLMs via Logit-Space Integration

Recognizing entity phrases remains a critical challenge for speech large language models. Existing prompting methods lack an explicit decoding-time biasing weight, limiting their controllability. Generative error correction methods can introduce hallucinated over-corrections. To address these limitations, we propose LOGIC (logit-space integration for contextual biasing), a robust framework operating directly in the logit space. By decoupling context injection from input processing, LOGIC enables explicit control over the biasing strength. Extensive experiments with an open-source speech large language model across 11 locales demonstrate that LOGIC achieves an average 9% relative reduction in entity word error rate, with an average false alarm rate increase of 0.3% and a 2.8% relative runtime overhead. When combined with prompting, LOGIC can reduce entity word error rate by 5% relative to the prompt-only method.

Peidong Wang, Jian Xue, Jinyu Li

Microsoft, USA

Language Model Decoding Speech Processing Automatic Speech Recognition Speech Language Models

Jan 20, 2026cs.SD

Performance and Complexity Trade-off Optimization of Speech Models During Training

In speech machine learning, neural network models are typically designed by choosing an architecture with fixed layer sizes and structure. These models are then trained to maximize performance on metrics aligned with the task's objective. While the overall architecture is usually guided by prior knowledge of the task, the sizes of individual layers are often chosen heuristically. However, this approach does not guarantee an optimal trade-off between performance and computational complexity; consequently, post hoc methods such as weight quantization or model pruning are typically employed to reduce computational cost. This occurs because stochastic gradient descent (SGD) methods can only optimize differentiable functions, while factors influencing computational complexity, such as layer sizes and floating-point operations per second (FLOP/s), are non-differentiable and require modifying the model structure during training. We propose a reparameterization technique based on feature noise injection that enables joint optimization of performance and computational complexity during training using SGD-based methods. Unlike traditional pruning methods, our approach allows the model size to be dynamically optimized for a target performance-complexity trade-off, without relying on heuristic criteria to select which weights or structures to remove. We demonstrate the effectiveness of our method through three case studies, including a synthetic example and two practical real-world applications: voice activity detection and audio anti-spoofing. The code related to our work is publicly available to encourage further research.

Esteban Gómez, Tom Bäckström

Department of Information and Communications Engineering, Aalto University, Espoo, Finland

Model Compression Neural Network Optimization Speech Processing

Jan 18, 2026cs.SD

Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs

Audio large language models (LLMs) enable unified speech understanding and generation, but adapting them to linguistically complex and dialect-rich settings such as Arabic-English remains challenging. We present a controlled study of multi-task instruction tuning for an Arabic-centric audio LLM across generative tasks, including automatic speech recognition (ASR) and speech and text summarization, as well as discriminative tasks, including dialect identification (DID) and speech emotion recognition (SER), in a resource-constrained setting. To support end-to-end Arabic speech summarization, we introduce AraMega-SSum, the first Arabic speech summarization dataset designed for training and benchmarking Arabic-centric audio LLMs. We compare four training strategies: (i) Uniform Mixing (UM), (ii) Task-Progressive Curriculum (TPC), (iii) Aligner-Based Diverse Sampling (ADS) for training-time batch construction, and (iv) a two-stage TPC->ADS strategy. Our results reveal a clear efficiency-robustness trade-off. TPC achieves the strongest performance on generative tasks, including ASR and summarization. ADS improves paralinguistic tasks but reduces generative stability when used alone. The two-stage TPC->ADS strategy provides the best overall balance, achieving the strongest DID and SER performance while outperforming large proprietary models such as Gemini-2.5-Pro on discriminative tasks. We will publicly release AraMega-SSum together with all experimental resources to support future research in Arabic speech understanding.

Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar Chowdhury

Qatar Computing Research Institute, Qatar

Spoken Language Understanding Speech Processing Audio Understanding Text Summarization Audio-Language Models Instruction Tuning+1

Jan 8, 2026eess.AS

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.

Junseok Lee, Chang-Jae Chun

OKESTRO · Sejong University

Multimodal Large Language Models Speech Processing Audio Understanding Multimodal Token Compression Speech Language Models

Jan 4, 2026cs.SD

MOSS Transcribe Diarize Technical Report

Speaker-Attributed, Time-Stamped Transcription (SATS) aims to transcribe what is said and to precisely determine the timing of each speaker, which is particularly valuable for meeting transcription. Existing SATS systems rarely adopt an end-to-end formulation and are further constrained by limited context windows, weak long-range speaker memory, and the inability to output timestamps. To address these limitations, we present MOSS Transcribe Diarize, a unified multimodal large language model that jointly performs Speaker-Attributed, Time-Stamped Transcription in an end-to-end paradigm. Trained on extensive real wild data and equipped with a 128k context window for up to 90-minute inputs, MOSS Transcribe Diarize scales well and generalizes robustly. Across comprehensive evaluations, it outperforms state-of-the-art commercial systems on multiple public and in-house benchmarks.

MOSI. AI, :, Donghua Yu +22

MOSI.AI*

Unified Multimodal Models Speech Processing Automatic Speech Recognition Speaker Diarization

Dec 23, 2025cs.SD

Aliasing-Free Neural Audio Synthesis

In neural audio synthesis, neural vocoders and codecs are models that reconstruct waveforms from acoustic and latent representations, which are essential to the resulting audio quality. While current models are capable of generating perceptually natural speech, they still struggle with high-fidelity music and singing voice synthesis, as severe aliasing artifacts are introduced by non-linear activation functions and upsampling layers in existing architectures. Although various anti-aliasing techniques have been proposed in digital signal processing, their integration into neural vocoders and codecs remains under-explored. This paper incorporates differentiable anti-aliasing techniques into the activation and upsampling modules to bridge this gap, and thus presents Pupu-Vocoder and Pupu-Codec. We build a test signal benchmark to evaluate the anti-aliased modules, and validate our proposed models on speech, singing voice, music, and audio. Experimental results show that Pupu-Vocoder and Pupu-Codec outperform existing systems on singing voice, music, and audio, while achieving comparable performance on speech. Demos, codes, and checkpoints are available at: VocodexElysium.github.io/AliasingFreeNeuralAudioSynthesis/.

Yicheng Gu, Junan Zhang, Chaoren Wang +3

Spellbrush, Akihabara, Tokyo 101-0021, Japan · Acoustic Laboratory, Department of Information Communications Engineering, Aalto University, Espoo 02150, Finland · School of Data Science, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China

Singing Voice Synthesis Neural Audio Codecs Speech Processing Audio Generation

Dec 22, 2025cs.CL

MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery

This paper introduces MauBERT, a multilingual extension of HuBERT that leverages articulatory features for robust cross-lingual phonetic representation learning. We continue HuBERT pre-training with supervision based on a phonetic-to-articulatory feature mapping in 55 languages. Our models learn from multilingual data to predict articulatory features or phones, resulting in language-independent representations that capture multilingual phonetic properties. Through comprehensive ABX discriminability testing, we show MauBERT models produce more context-invariant representations than state-of-the-art multilingual self-supervised learning models. Additionally, the models effectively adapt to unseen languages and casual speech with minimal self-supervised fine-tuning (10 hours of speech). This establishes an effective approach for instilling linguistic inductive biases in self-supervised speech models.

Angelo Ortiz Tandazo, Manel Khentout, Youssef Benchekroun +2

ENS, PSL Research University, EHESS, CNRS, Paris, France · Meta AI Research, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France

Speech Processing Self-Supervised Speech Representation Learning Cross-Lingual Transfer

Nov 1, 2025cs.SD

Physics-Informed Neural Networks for Speech Production

The analysis of speech production based on physical models of the vocal folds and vocal tract is essential for studies on vocal-fold behavior and linguistic research. This paper proposes a speech production analysis method using physics-informed neural networks (PINNs). The networks are trained directly on the governing equations of vocal-fold vibration and vocal-tract acoustics. Vocal-fold collisions introduce nondifferentiability and vanishing gradients, challenging phenomena for PINNs. We demonstrate, however, that introducing a differentiable approximation function enables the analysis of vocal-fold vibrations within the PINN framework. The period of self-excited vocal-fold vibration is generally unknown. We show that by treating the period as a learnable network parameter, a periodic solution can be obtained. Furthermore, by implementing the coupling between glottal flow and vocal-tract acoustics as a hard constraint, glottis-tract interaction is achieved without additional loss terms. We confirmed the method's validity through forward and inverse analyses, demonstrating that the glottal flow rate, vocal-fold vibratory state, and subglottal pressure can be simultaneously estimated from speech signals. Notably, the same network architecture can be applied to both forward and inverse analyses, highlighting the versatility of this approach. The proposed method inherits the advantages of PINNs, including mesh-free computation and the natural incorporation of nonlinearities, and thus holds promise for a wide range of applications.

Kazuya Yokota, Ryosuke Harakawa, Masaaki Baba +1

Department of Mechanical Engineering, Nagaoka University of Technology, 1603-1, Kamitomioka, Nagaoka, Niigata, Japan · Department of Electrical, Electronics and Information Engineering, Nagaoka University of Technology

Speech Processing Physics-Informed ML Inverse Problems

Oct 23, 2025cs.SD

UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement

Neural audio codecs have largely promoted the application of language models (LMs) for speech applications. However, the effectiveness of autoregressive LM-based models in unifying speech enhancement (SE) tasks remains underexplored. In this work, we propose UniSE, a unified decoder-only LM-based framework to handle different SE tasks including speech restoration, target speaker extraction, and speech separation. Conditioned on input speech features, it autoregressively generates target discrete tokens, facilitating compatibility between distinct learning patterns of multiple tasks. To further optimize speech quality, we introduce a progressive reinforcement learning strategy with multiple assessment criteria. Experiments on several benchmarks show that UniSE achieves competitive performance compared to discriminative and generative baselines, demonstrating the capacity of LMs in unifying SE tasks. The code and demo are available at: https://github.com/alibaba/unified-audio/tree/main/QuarkAudio-UniSE.

Haoyin Yan, Chengwei Liu, Shaofei Xue +4

Qwen Business Unit of Alibaba, China · Tongyi AI Lab of Alibaba, China

Target Speaker Extraction Decoder-Only Language Models Speech Processing Speech Enhancement Speech Separation

Oct 12, 2025cs.SD

ParsVoice: A Large-Scale Multi-Speaker Persian Speech Corpus for Text-to-Speech Synthesis

Persian remains substantially underrepresented in open speech-text resources, limiting progress in multi-speaker text-to-speech (TTS), speech-language modelling, and low-resource speech processing. We introduce ParsVoice, the largest publicly available Persian speech-text corpus tailored for training multi-speaker TTS systems, along with a scalable pipeline to construct high-quality speech-text data from long-form audiobook recordings. The pipeline combines a fine-tuned ParsBERT sentence-completion classifier, ASR-based boundary optimization, punctuation restoration, speaker identification, and a multi-dimensional quality assessment that covers both audio and Persian-specific text properties. The resulting release contains a 2,200-hour TTS-ready subset with 1.36 million aligned segments from 1,815 automatically inferred speaker IDs, making it more than 25 times larger than the largest previously available open Persian TTS dataset. To validate the corpus, we fine-tune XTTSv2, a zero-shot multilingual TTS model that operates directly on raw Persian text without phoneme representations. The resulting model achieves a naturalness MOS of 3.6/5 and a speaker-similarity MOS of 4.0/5. ParsVoice, its metadata, and the corpus-construction pipeline are publicly available on Hugging Face: https://huggingface.co/datasets/MohammadJRanjbar/ParsVoice and GitHub: https://github.com/MohammadJRanjbar/ParsVoice, supporting reproducible research on Persian speech synthesis and low-resource speech-language technologies.

Mohammad Javad Ranjbar Kalahroodi, Heshaam Faili, Azadeh Shakery

School of Electrical and Computer Engineering, University of Tehran, Iran · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran

TTS Synthesis Speech Processing

Sep 18, 2025eess.AS

BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings

Child-centered daylong recordings are essential for studying early language development, but existing speech models trained on clean adult data perform poorly due to acoustic and linguistic differences. We introduce BabyHuBERT, a self-supervised speech model trained on 13,000 hours of multilingual child-centered recordings from 40+ languages. Evaluated on voice type classification, the task of identifying who produces speech and when in child-centered recordings (key child, other children, male, and female adults), BabyHuBERT-VTC achieves F1-scores from 55.0% to 76.1% across six corpora, consistently outperforming W2V2-LL4300 and HuBERT (pretrained on English daylongs and clean adult speech, respectively). Notable gains include 14.0 and 18.3 absolute F1 points over HuBERT on Vanuatu and Solomon Islands, demonstrating effectiveness on underrepresented languages. We share code and models to support researchers working with child-centered recordings across diverse linguistic contexts.

Théo Charlot, Tarek Kunze, Maxime Poli +3

LSCP, DEC, ENS, EHESS, CNRS, PSL University, France · Laboratoire d’Informatique et Syst`emes, Universit´e Aix-Marseille, CNRS, France

Speech Foundation Models Speech Processing Audio Understanding Self-Supervised Speech Representation Learning Audio Classification Speaker Diarization

Aug 27, 2025eess.AS

An Effective Strategy for Modeling Score Ordinality and Non-uniform Intervals in Automated Speaking Assessment

A recent line of research on automated speaking assessment (ASA) has benefited from self-supervised learning (SSL) representations, which capture rich acoustic and linguistic patterns in non-native speech without underlying assumptions of feature curation. However, speech-based SSL models capture acoustic-related traits but overlook linguistic content, while text-based SSL models rely on ASR output and fail to encode prosodic nuances. Moreover, most prior arts treat proficiency levels as nominal classes, ignoring their ordinal structure and non-uniform intervals between proficiency labels. To address these limitations, we propose an effective ASA approach combining SSL with handcrafted indicator features via a novel modeling paradigm. We further introduce a multi-margin ordinal loss that jointly models both the score ordinality and non-uniform intervals of proficiency labels. Extensive experiments on the TEEMI corpus show that our method consistently outperforms strong baselines and generalizes well to unseen prompts.

Tien-Hong Lo, Szu-Yu Chen, Yao-Ting Sung +1

Department of Computer Science and Information Engineering, National Taiwan Normal University · Department of Educational Psychology and Counseling, National Taiwan Normal University

Speech Processing Ordinal Regression

Jul 17, 2025cs.CL

Balalaika: Data-Centric, Prosody-Aware Annotation Pipeline for Russian Speech

We introduce Balalaika, an open-source, data-centric pipeline for processing audio and producing prosody-aware annotations. It combines semantic VAD for context-preserving segmentation, multi-ASR ensembling with ROVER consensus decoding, while retaining optional word-level timestamps, followed by automatic quality and speaker-purity filtering. The text is further enriched with punctuation restoration, lexical stress and "\textipa{e}/\textipa{He}" normalization, and IPA phonemes. Using Balalaika, we build a 5.1k-hour multi-source Russian corpus with rich annotations, and show consistent gains under equalized training budgets for both speech denoising and TTS; ablations confirm complementary benefits of stress and punctuation and improved synthesis with stricter MOS filtering. The datasets are publicly available at \href{https://huggingface.co/collections/lab260/balalaika-dataset}{\underline{\textbf{HuggingFace}}}

Kirill Borodin, Nikita Vasiliev, Vasiliy Kudryavtsev +3

lab260, Moscow Technical University of Communications and Informatics, Russia · BitmanagerAI, UAE

Speech Processing Speech Prosody

Jul 11, 2025cs.SD

Enforcing Speech Content Privacy in Environmental Sound Recordings using Segment-wise Waveform Reversal

Environmental sound recordings often contain intelligible speech, raising privacy concerns that limit analysis, sharing and reuse of data. In this paper, we introduce a method that renders speech unintelligible while preserving both the integrity of the acoustic scene, and the overall audio quality. Our approach involves reversing waveform segments to distort speech content. This process is enhanced through a voice activity detection and speech separation pipeline, which allows for more precise targeting of speech. In order to demonstrate the effectivness of the proposed approach, we consider a three-part evaluation protocol that assesses: 1) speech intelligibility using Word Error Rate (WER), 2) sound sources detectability using Sound source Classification Accuracy-Drop (SCAD) from a widely used pre-trained model, and 3) audio quality using the Fréchet Audio Distance (FAD), computed with our reference dataset that contains unaltered speech. Experiments on this simulated evaluation dataset, which consists of linear mixtures of speech and environmental sound scenes, show that our method achieves satisfactory speech intelligibility reduction (97.9% WER), minimal degradation of the sound sources detectability (2.7% SCAD), and high perceptual quality (FAD of 1.40). An ablation study further highlights the contribution of each component of the pipeline. We also show that incorporating random splicing to our speech content privacy enforcement method can enhance the algorithm's robustness to attempt to recover the clean speech, at a slight cost of audio quality.

Modan Tailleur, Mathieu Lagrange, Pierre Aumond +1

Nantes Universit´e, ´Ecole Centrale Nantes, CNRS, LS2N, UMR 6004, Nantes, France · Universit´e Gustave Eiffel, CEREMA, UMRAE, Bouguenais, France · Nantes Universit´e, ENSA Nantes, ´Ecole Centrale Nantes, CNRS, AAU-CRENAU, Nantes, France

Speech Processing

May 20, 2025cs.SD

Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker Traits and Speech Properties

We introduce Vox-Profile, a comprehensive benchmark to characterize rich speaker traits and speech properties using speech foundation models. Unlike prior work that targets a single dimension of speaker traits, Vox-Profile provides holistic and multi-dimensional profiles spanning both static speaker traits (e.g., age, sex, accent, voice quality) and dynamic speech traits (e.g., emotion, speech flow). This benchmark is grounded in speech science and linguistics, developed with domain experts to accurately index speaker and speech characteristics. We report benchmark experiments using over 15 publicly available speech datasets and several widely used speech foundation models that target various static and dynamic speaker and speech properties. In addition to benchmark experiments, we showcase several downstream applications supported by Vox-Profile. First, we show that Vox-Profile can augment existing speech recognition datasets to analyze ASR performance variability. Vox-Profile is also used as a tool to evaluate the performance of speech generation systems. Finally, we assess the quality of our automated profiles through comparison with human evaluation and show convergent validity. Vox-Profile is publicly available at: https://github.com/tiantiaf0627/vox-profile-release.

Tiantian Feng, Jihwan Lee, Anfeng Xu +9

University of Southern California · Johns Hopkins University

Speech Processing Speech Foundation Models

May 18, 2025cs.LG

BenSParX: A Robust Explainable Machine Learning Framework for Parkinson's Disease Detection from Bengali Conversational Speech

Early detection of PD remains particularly challenging in resource-constrained settings, where voice-based analysis has emerged as a promising non-invasive and cost-effective alternative. However, existing studies predominantly focus on English or other major languages; notably, no voice dataset for PD exists for Bengali -- a language spoken by over 230 million people worldwide -- posing a significant barrier to culturally inclusive and accessible healthcare solutions. We present BenSparX, the first Bengali conversational speech dataset for PD detection, along with a robust and explainable ML framework tailored for early diagnosis. The proposed framework incorporates diverse acoustic feature categories, systematic feature selection methods, and state-of-the-art ML classifiers with extensive hyperparameter optimization. Furthermore, to enhance interpretability and trust in model predictions, the framework incorporates SHAP (SHapley Additive exPlanations) analysis to quantify the contribution of individual acoustic features toward PD detection. Our framework achieves state-of-the-art performance, yielding an accuracy of 95.67%, F1 score of 95.62%, and AUC of 0.990. We further validated our approach by applying the framework to existing PD datasets in other languages, where it consistently outperforms state-of-the-art approaches. This study lays the foundation for identifying subtle yet clinically meaningful vocal biomarkers, particularly in low-resource settings such as Bengali-speaking populations, and represents a significant step toward equitable, explainable, and robust digital health diagnostics for neurodegenerative disorders. The labelled acoustic-feature dataset derived from the audio recordings in this study is available at https://github.com/riadEDU/BenSParX.

Riad Hossain, Muhammad Ashad Kabir, Arat Ibne Golam Mowla +2

Department of Computer Science and Engineering, Chittagong University of Engineering and Technology, Chittagong, 4349, Bangladesh · School of Computing, Mathematics and Engineering, Charles Sturt University, Bathurst, NSW, 2795, Australia · School of Science, Engineering and Technology, East Delta University, Chittagong, 4209, Bangladesh +1

Speech Processing Low-Resource Language Processing

Mar 21, 2025cs.CR

Measuring the Robustness of Audio Deepfake Detection under Real-World Corruption

Deepfakes have emerged as a widespread and rapidly escalating concern in generative AI, spanning images, audio, and videos. Among these, audio deepfakes are particularly alarming due to the growing accessibility of high-quality voice synthesis tools and the ease with which synthetic speech can be distributed through social media and robocalls. Consequently, detecting audio deepfakes is critical for combating the misuse of AI-generated speech. However, real-world audio is often affected by corruptions such as noise, audio modification, and compression, which can significantly degrade detection performance. In this work, we systematically evaluate the robustness of 10 audio deepfake detection models against 18 common corruption types, grouped into three categories: noise perturbation, audio modification, and compression. Using both traditional deep learning models and state-of-the-art speech foundation models, our study yields four key insights. (1) Most models are robust to noise but remain vulnerable to audio modifications and compression, especially neural codecs. (2) Speech foundation models consistently outperform traditional models across most corruption scenarios, likely due to large-scale pre-training on diverse audio datasets. (3) Increasing model size improves robustness, although the gains diminish as models become larger. (4) Robustness to unseen corruptions can be improved through targeted data augmentation during training or speech enhancement at inference time. These findings highlight the importance of evaluating audio deepfake detectors under diverse real-world corruptions and developing more robust detection frameworks for practical deployment. We further advocate that future research on deepfake detection across all media should account for the diverse and unpredictable distortions encountered in real-world environments.

Xiang Li, Pin-Yu Chen, Wenqi Wei

Fordam University New York City, New York, USA · IBM Research Yorktown Heights, New York, USA · Fordham University New York City, New York, USA

Audio Deepfake Detection Speech Foundation Models Speech Processing

Nov 26, 2023cs.LG

Topology-enhanced machine learning for speech signal processing

In artificial-intelligence-aided signal processing, existing deep learning models often exhibit a black-box structure. Here, conceptually beyond spectral analysis, we demonstrate that topological methods not only effectively capture intrinsic and complex structural information but can also enhance neural networks. We provide a transparent methodology, TopCap, to capture topological features inherent in time series for basic machine learning. Compared to prior approaches, we obtain descriptors that probe finer information such as the vibration of a time series. Notably, in classifying voiced and voiceless consonants, TopCap achieves an accuracy consistently standing in comparison with neural network models. Moreover, by integrating TopCap features into those neural networks, our approach improves upon state-of-the-art methods in terms of robustness against noise, as well as accuracy, stability, convergence of loss function, and interpretability.

Pingyao Feng, Qingrui Qu, Haiyu Zhang +4

Department of Mathematics, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, North Carolina State University, Raleigh, NC, USA. · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen), Shenzhen, China. +1

Interpretable ML Speech Processing Audio Classification Topological Data Analysis

Oct 19, 2021cs.SD

A mathematical model of the vowel space

The articulatory-acoustic relationship is many-to-one and non linear and this is a great limitation for studying speech production. A simplification is proposed to set a bijection between the vowel space (f1, f2) and the parametric space of different vocal tract models. The generic area function model is based on mixtures of cosines allowing the generation of main vowels with two formulas. Then the mixture function is transformed into a coordination function able to deal with articulatory parameters. This is shown that the coordination function acts similarly with the Fant's model and with the 4-Tube DRM derived from the generic model.

Frédéric Berthommier

GIPSA-PCMD

Speech Processing

Aug 12, 2019eess.AS

Personal VAD: Speaker-Conditioned Voice Activity Detection

In this paper, we propose "personal VAD", a system to detect the voice activity of a target speaker at the frame level. This system is useful for gating the inputs to a streaming on-device speech recognition system, such that it only triggers for the target user, which helps reduce the computational cost and battery consumption, especially in scenarios where a keyword detector is unpreferable. We achieve this by training a VAD-alike neural network that is conditioned on the target speaker embedding or the speaker verification score. For each frame, personal VAD outputs the probabilities for three classes: non-speech, target speaker speech, and non-target speaker speech. Under our optimal setup, we are able to train a model with only 130K parameters that outperforms a baseline system where individually trained standard VAD and speaker recognition networks are combined to perform the same task.

Shaojin Ding, Quan Wang, Shuo-yiin Chang +2

Texas A&M University, USA · Google Inc., USA

Speech Processing

Date pendingcs.SD

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Voice agents are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses realistic conversation simulation and comprehensive voice-specific evaluation. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates dynamic bot-to-bot audio conversations with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy) and EVA-X (Experience). EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and multi-trial measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean

\Delta

up to 0.314). We release EVA-Bench under an open-source license.

Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10

ServiceNow AI Research

Voice Agent Evaluation Spoken Dialogue Systems AI Agent Evaluation Agent Evaluation Speech Processing AI Agent Benchmarks