MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates

Zikang Huang; Meng Ge; Tianrui Wang; Xuanchen Li; Xiaobao Wang; Longbiao Wang; Jianwu Dang

cs.SDMar 24, 2026

MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates

Authors: Zikang Huang, Meng Ge, Tianrui Wang, Xuanchen Li, Xiaobao Wang, Longbiao Wang, Jianwu Dang

Organizations: Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, China · Huiyan Technology Company, Ltd., China · Chinese Academy of Sciences, China

Abstract

Self-supervised learning (SSL) has advanced speech processing. However, existing speech SSL methods typically assume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To address this limitation, we propose MSRHuBERT, a multi-sampling-rate adaptive pre-training method. Building on HuBERT, we replace its single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN that maps raw waveforms from different sampling rates to a shared temporal resolution without resampling. This design enables unified mixed-rate pre-training and fine-tuning. In experiments spanning 16 to 48 kHz, MSRHuBERT outperforms HuBERT on speech recognition and full-band speech reconstruction, preserving high-frequency detail while modeling low-frequency semantic structure. Moreover, MSRHuBERT retains HuBERT's mask-prediction objective and Transformer encoder, so existing analyses and improvements that were developed for HuBERT can apply directly.

Figures & tables

Figure 1: The resolution mismatch across diverse sampling rates with a fixed 320× downsampling CNN.

Figure 2: The architecture of MSRHuBERT, which takes raw waveforms at different sampling rates as input. The multi-sampling-rate adaptive downsampling CNN can map waveforms to a common temporal resolution by designing rate-specific downsampling, and supports mixed-rate pre-training.

Sampling Rate	Strides	Kernel width
16 kHz	5, 2, 2, 2, 2, 2, 2	10, 3, 3, 3, 3, 2, 2
22.05 kHz	7, 7, 3, 3	19, 14, 4, 3
24 kHz	5, 3, 2, 2, 2, 2, 2	10, 5, 3, 3, 3, 2, 2
48 kHz	5, 3, 2, 2, 2, 2, 2, 2	10, 5, 3, 3, 3, 3, 2, 2

Table 1: Architecture design for proposed multi-sampling-rate adaptive downsampling CNN. For the CNN branch of each sampling rate, the stride and kernel width is determined based on the calculation formulas for the downsampling rate, stride, and kernel width, as well as the prime factor decomposition.

Model	ASR (WER↓)								Full-band SR (STOI↑)
	16k Hz	22.05k Hz	24k Hz	48k Hz	four sampling rates mixed fine-tuning				16k Hz	22.05k Hz	24k Hz	48k Hz
	16k Hz	22.05k Hz	24k Hz	48k Hz	16 kHz	22.05 kHz	24 kHz	48 kHz	16k Hz	22.05k Hz	24k Hz	48k Hz
HuBERT Base	6.41	3.34	6.84	5.96	7.42	3.35	7.61	6.63	88.46	93.04	88.52	81.42
Re. 16kHz HuBERT Base	6.03	3.15	6.59	5.61	6.89	2.95	6.97	5.53	89.35	93.46	88.49	82.49
- w/o resampling in fine-tuning	6.03	4.47	15.61	38.18	7.69	3.72	10.30	33.72	89.35	91.84	86.71	75.53
Re. 24kHz HuBERT Base	6.15	3.28	6.54	5.70	6.95	3.02	7.24	5.74	89.61	94.32	89.34	83.75

Table 2: The comprehensive results for ASR and SR downstream tasks using different pre-trained models.

Figure 3: Weight analysis on the ASR task of the SUPERB Benchmark. Layer 0 corresponds to the input of the first Transformer layer. The y-axis represents different settings, including pre-trained model and sampling rate of the downstream dataset in ASR.

Model	ASR (WER↓)
Model	16k Hz	22.05k Hz	24k Hz	48k Hz
Re. 16kHz HuBERT Base	6.03	3.15	6.59	5.61
MSRHuBERT	5.89	3.35	6.35	5.83
+ Intermediate Layer Supervision	5.56	3.17	6.16	5.67
+ Progressive Decoupling	5.63	3.30	5.94	5.52

Table 3: The result of MSRHuBERT with adaptation of intermediate layer supervision and progressive decoupling in downstream fine-tuning ASR task.

Self-Supervised Speech Representation Learning

Self-Supervised Speech Representation Learning trains models on unlabeled audio to extract features useful for speech tasks and linguistic analysis.

See all 101 papers

SA-SSL-MOS: Self-supervised Learning MOS Prediction with Spectral Augmentation for Generalized Multi-Rate Speech Assessment
Feb 16, 2026Fengyuan Cao, Xinyu Liang, Fredrik Cumlin +4Speech Quality Assessment Self-Supervised Speech Representation Learning
KTH Royal Institute of Technology, Stockholm, Sweden · Google LLC
SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation
Oct 3, 2026Séverin Baroudi, Hervé Bredin, Ricard MarxerSelf-Supervised Speech Representation Learning Speech Separation
Univ Toulon, Aix Marseille Univ, CNRS, LIS, Toulon, France · CNRS, ILLS, Montr´eal, Canada · pyannoteAI, Toulouse, France
REIMU: Efficient Heterogeneous Hierarchical Reasoning for SSL-Based Speech Deepfake Detection
Aug 1, 2026Kwok-Ho Ng, Tingting Song, Bingwen Feng +1Audio Deepfake Detection ASVspoof
College of Cyber Security, Jinan University, Guangzhou, China

Feb 16, 2026J/K move · Enter open · S save

SA-SSL-MOS: Self-supervised Learning MOS Prediction with Spectral Augmentation for Generalized Multi-Rate Speech Assessment

Fengyuan Cao, Xinyu Liang, Fredrik Cumlin, Victor Ungureanu+3

KTH Royal Institute of Technology, Stockholm, Sweden · Google LLC

Speech Quality Assessment Self-Supervised Speech Representation Learning

Open paper AskPDF arXiv

Designing a speech quality assessment (SQA) system for estimating mean-opinion-score (MOS) of multi-rate speech with varying sampling frequency (16-48 kHz) is a challenging task. The challenge arises due to the limited availability of a MOS-labeled training dataset comprising multi-rate speech samples. While self-supervised learning (SSL) models have been widely adopted in SQA to boost performance, a key limitation is that they are pretrained on 16 kHz speech and therefore discard high-frequency information present in higher sampling rates. To address this issue, we propose a spectrogram-augmented SSL method that incorporates high-frequency features (up to 48 kHz sampling rate) through a parallel-branch architecture. We further introduce a two-step training scheme: the model is first pre-trained on a large 48 kHz dataset and then fine-tuned on a smaller multi-rate dataset. Experimental results show that leveraging high-frequency information overlooked by SSL features is crucial for accurate multi-rate SQA, and that the proposed two-step training substantially improves generalization when multi-rate data is limited.

Abstract

Self-Supervised Speech Representation Learning

Explore similar work

SA-SSL-MOS: Self-supervised Learning MOS Prediction with Spectral Augmentation for Generalized Multi-Rate Speech Assessment

SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation

REIMU: Efficient Heterogeneous Hierarchical Reasoning for SSL-Based Speech Deepfake Detection