From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale

Yaroslav Getman; Tamás Grósz; Tommi Lehtonen; Mikko Kurimo

cs.CLOct 9, 2026

From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale

Authors: Yaroslav Getman, Tamás Grósz, Tommi Lehtonen, Mikko Kurimo

Organizations: Dept. lnformation and Communications Engineering Aalto University Espoo, Finland · Walton Institute, South East Technological University, Ireland Aalto University Espoo, Finland · Finnish National Audiovisual Institute (KAVI) Helsinki, Finland

Abstract

Broadcast archives offer scale and diversity for self-supervised learning (SSL) but contain substantial noisy, non-speech, and multilingual content. Prior work typically relies on voice activity detection or language identification (LID) to extract speech, ensuring presence rather than quality. Whether and how to filter unlabeled speech by quality for SSL, particularly from noisy broadcasts, remains underexplored. We present a large-scale Finnish SSL study starting from 2.1M hours of unconstrained broadcast audio. We propose automatic speech recognition (ASR) confidence as a proxy for pretraining data quality and show 3-7% relative WER improvements over LID-based filtering despite a smaller training set. We further address the lack of controlled architectural comparisons in speech SSL by evaluating Transformer, Macaron-Net, Conformer, and E-Branchformer at matched parameter budgets. Our models improve upon prior Finnish SSL baselines, particularly at Base scale and out-of-domain.

Figures & tables

Transformer [ 41 ]	Macaron-Net [ 28 ]	Conformer [ 17 ]	E-Branchformer [ 23 ]
$\begin{aligned} x^{\prime}_{i}&=x_{i}+\mathrm{MHSA}(x_{i})\\ y_{i}&=x^{\prime}_{i}+\mathrm{FFN}(x^{\prime}_{i})\end{aligned}$	$\begin{aligned} \tilde{x}_{i}&=x_{i}+\tfrac{1}{2}\mathrm{FFN}(x_{i})\\ x^{\prime}_{i}&=\tilde{x}_{i}+\mathrm{MHSA}(\tilde{x}_{i})\\ y_{i}&=x^{\prime}_{i}+\tfrac{1}{2}\mathrm{FFN}(x^{\prime}_{i})\end{aligned}$	$\begin{aligned} \tilde{x}_{i}&=x_{i}+\tfrac{1}{2}\mathrm{FFN}(x_{i})\\ x^{\prime}_{i}&=\tilde{x}_{i}+\mathrm{MHSA}(\tilde{x}_{i})\\ x^{\prime\prime}_{i}&=x^{\prime}_{i}+\mathrm{Conv}(x^{\prime}_{i})\\ y_{i}&=x^{\prime\prime}_{i}+\tfrac{1}{2}\mathrm{FFN}(x^{\prime\prime}_{i})\end{aligned}$	$\begin{aligned} \tilde{x}_{i}&=x_{i}+\tfrac{1}{2}\mathrm{FFN}(x_{i})\\ x^{\prime}_{ig}&=\mathrm{MHSA}(\tilde{x}_{i}),\quad x^{\prime}_{il}=\mathrm{cgMLP}(\tilde{x}_{i})\\ x^{\prime\prime}_{i}&=\tilde{x}_{i}+\mathrm{Merge}(x^{\prime}_{ig},x^{\prime}_{il})\\ y_{i}&=x^{\prime\prime}_{i}+\tfrac{1}{2}\mathrm{FFN}(x^{\prime\prime}_{i})\end{aligned}$

TABLE I : Comparison of encoder architectures. Equations illustrate the sequence of sub-modules and residual connections within a single encoder block.

\mathrm{Merge}(\cdot)

denotes the depth-wise convolutional fusion module. Layer normalizations are omitted for clarity.

Panel of Figure 2 — Fig. 1 : Impact of filtering criteria on downstream validation WER and pretraining data retention rate. All confidence thresholds ( $C_{\text{char}}\geq\tau$ ) are applied to deduplicated VAD segments.

Panel of Figure 3 — Fig. 2 : Pretraining data preparation pipeline. We reduce 2.1M hours of raw broadcast archives to a high-quality 471K-hour subset through VAD, transcript-based deduplication, and the confidence-based filtering threshold $\tau=-0.1$ optimized in Figure 1 .

Backend	L	Enc.	FFN	Params	In-domain			Out-of-domain
Backend	L	Enc.	FFN	Params	LP	FP’16	FP’20	CV	FLEURS	KAVI
Transformer	6	1024	4096	90.6M	28.30	14.08	11.58	15.60	17.13	31.49
	12	768	3072	95.0M	22.28	10.72	8.70	9.74	13.02	24.85
	18	768	1536	95.1M	21.33	10.64	8.66	8.99	12.71	23.91
	24	576	2048	96.4M	21.66	10.69	8.73	8.77	12.85	24.57
Macaron-Net	12	768	1536	95.1M	21.80	10.58	9.10	8.62	12.54	25.22

TABLE II : WER (%) for wav2vec 2.0 Base variants across Finnish benchmarks. The best results within the same encoder backend are underlined , and the best overall results are shown in bold .

Backend	Config	Params	In-domain			Out-of-domain
	(L / Enc. / FFN)		LP	FP’16	FP’20	CV	FLEURS	KAVI
Base models
Transformer [ 14 ] (baseline)	12 / 768 / 3072	95.0M	22.27	10.92	8.42	10.61	13.27	27.45
Transformer (ours)	12 / 768 / 3072	95.0M	22.28	10.72	8.70	9.74	13.02	24.85
	18 / 768 / 1536	95.1M	21.33	10.64	8.66	8.99	12.71	23.91
Conformer	12 / 576 / 2048	96.7M	18.89	10.08	8.07	6.46	11.20	23.71

TABLE III : WER (%) across scales and encoder backends. Best results per scale are underlined and best overall in bold . LP = Lahjoita puhetta (colloquial Finnish), FP = Finnish Parliament, CV = Common Voice, KAVI = real-world broadcast speech.

Conformer

See all 12 papers

Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data
Jun 11, 2026Qixu Chen, Satoshi NakamuraSpeech Translation Audio-Language Models
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence The Chinese University of Hong Kong, Shenzhen, China
SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMs
Aug 27, 2026Mengzhe GengAudio-Language Model Evaluation Hallucination Detection
National Research Council Canada
AudioMosaic: Contrastive Masked Audio Representation Learning
May 14, 2026Hanxun Huang, Qizhou Wang, Xingjun Ma +3Audio Representation Learning Self-Supervised Pre-Training
School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA

Jun 11, 2026J/K move · Enter open · S save

Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data

Qixu Chen, Satoshi Nakamura

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence The Chinese University of Hong Kong, Shenzhen, China

Speech Translation Audio-Language Models

Open paper AskPDF arXiv

Large-scale mined corpora provide abundant training data for end-to-end speech-to-speech translation (S2ST) but may contain noise, misalignment, and semantic errors. Filtering noisy data is crucial to maintain robust speech translation performance. We study how to train an audio-language model to make keep/drop decisions on paired speech directly from audio. To obtain reliable supervision without manual labels, we adopt a scalable two-stage Rank-to-Distill strategy. A lightweight ranker generates keep/drop pseudo-labels from noisy speech pairs, then trains an audio large language model to predict keep/drop directly from raw paired speech. The resulting model jointly captures acoustic fidelity and cross-lingual semantic consistency for the selection of speech-conditioned data. Experiments on CVSS-C and SpeechMatrix show consistent improvements over unfiltered training, yielding up to +1.4 ASR-BLEU for end-to-end S2ST.

Abstract

Conformer

Explore similar work

Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data

SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMs

AudioMosaic: Contrastive Masked Audio Representation Learning