cs.SDJun 23, 2026

Beyond U-Net: A Latent-Representation-Aligned Skip-Free Backbone for Flow-Matching Speech Enhancement

Authors: Wangyi PuMichele Scarpiniti

Organizations: Department of Information Engineering, Electronics and Telecommunications, Sapienza University of Rome, Italy

Abstract

Generative models, particularly diffusion and score-based approaches, have recently achieved strong performance in speech enhancement, but their iterative sampling process limits real-time deployment. Flow Matching offers an efficient alternative by transporting noisy speech toward clean speech through an ordinary differential equation with few function evaluations. In this work, we propose a skip-free encoder-decoder backbone for flow-matching speech enhancement, guided by Latent Representation Alignment (LRA). Instead of relying on U-Net skip connections, which may transfer noise-correlated low-level features to the decoder, the proposed model aligns its bottleneck and decoder representations with clean latent features extracted from a frozen Descript Audio Codec encoder-decoder without quantization. This codec-aligned supervision promotes compact clean-speech representations while preserving efficient few-step inference. Experiments on WSJ0-CHiME3 and VoiceBank-DEMAND show improved PESQ and perceptual quality, especially on VoiceBank-DEMAND, using only five function evaluations.

Explore similar work

CardsList
  1. Speech Enhancement Based on Drifting Models

    Apr 27, 2026Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn +2Speech EnhancementGenerative Framework

  2. DriftSE: Speech Enhancement with Generative Drifting

    Sep 14, 2026Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn +2Speech EnhancementAcoustic Representation