Cluster-based prediction is widely used in self-supervised speech learning. A soft target preserves a distribution over clusters rather than a single label. This distribution specifies both the probability values and which clusters receive them. Comparisons between soft targets and hard labels do not separate the contributions of these two aspects to the learned representation. We study this in S-JEPA, a recent high-performing self-supervised speech model trained with soft Gaussian mixture model (GMM) targets. We compare its original targets with counterfactual targets that preserve the most likely cluster and all probability values but change which remaining clusters receive the other probabilities. Across three training seeds, the original soft distribution is recovered more accurately from Encoders trained with the original than counterfactual targets. Because this could reflect target matching alone, we also test low-level acoustic and phonetic information. Both are more accessible from Encoders trained with the original targets. This suggests that cluster assignments affect acoustic and phonetic properties of the learned representation, not just recovery of the training target.
Figures & tables
Figure 1: Counterfactual target construction and matched training runs. REAL uses the original fixed-GMM posterior qt , whereas RANDOM-REASSIGN preserves the top-1 component and all posterior values while reassigning only the non-maximal values to form qtRR . The two runs share the masked input, architecture, and training setup, and minimize KL divergence to their corresponding targets. Encoder outputs are used for subsequent representation analysis; solid and dashed arrows denote forward and gradient flow, respectively.
Figure 2: Phase-1 differences between REAL SOFT and RANDOM-REASSIGN across three independent seeds. Points show the test-clean gap for each seed; error bars are paired 95% speaker-cluster bootstrap confidence intervals over the 40 test speakers (10,000 draws). Panel (a) reports RANDOM-REASSIGN minus REAL SOFT in DeepTailCE. Panels (b) and (c) report REAL SOFT minus RANDOM-REASSIGN in controlled dynamic R2 and phone accuracy. Positive values therefore favor REAL SOFT in every panel.
Phase 1
Phase 2
Readout
A
B
C
A (100k)
DeepTailCE reduction
0.0679
0.0135
0.0402
0.0370
Controlled dynamic R2 gain
0.0262
0.0247
0.0167
0.0052
Phone accuracy gain (pp)
1.55
0.12
1.07
1.77
Table 1: REAL SOFT gains over RANDOM-REASSIGN across three Phase 1 seeds and after 100k Phase 2 updates for the primary seed.
Figure 3: Graded exposure to the original component correspondence. Each line shows one independent Phase-1 seed across λ∈{0,0.25,0.5,0.75,1} . Panels report (a) DeepTailCE, where lower is better; (b) controlled dynamic R2 ; and (c) phone accuracy, where higher is better. At λ=0 , all frames use RANDOM-REASSIGN; at λ=1 , all frames use REAL SOFT. Intermediate levels retain the original correspondence for nested subsets of frames.
Self-supervised speech encoders are predominantly trained by predicting discrete hard cluster IDs at masked positions, a recipe that collapses acoustic ambiguity at category boundaries and requires interrupting training to re-cluster the entire corpus between iterations. We introduce S-JEPA, a JEPA-style encoder-predictor pair trained to match the soft posteriors of a Gaussian Mixture Model at masked positions via KL divergence. Training runs as one continuous optimization trajectory in two phases: a fixed GMM over MFCC features, then an online GMM over encoder features, with the input layer selected adaptively from a label-free signal, removing both the offline re-cluster step and the hand-tuned choice of which transformer layer to cluster on. Under the SUPERB protocol, S-JEPA achieves the lowest WER among evaluated SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at roughly half its parameter count, establishing a new Pareto frontier without offline re-clustering or teacher distillation. An analysis of the predictor's per-frame entropy on held-out speech reveals a bimodal distribution with a substantial minority of frames near the entropy of a perfect two-cluster tie, providing direct empirical evidence that the soft-target objective preserves the acoustic ambiguity that hard targets would collapse. Code is available at https://github.com/gioannides/s-jepa.
Georgios Ioannides, Adrian Kieback, Judah Goldfeder +5
Carnegie Mellon University · James Silberrad Brown Center for AI · Amazon GenAI +4
Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely on complex, carefully designed prediction targets. We challenge this necessity with GLaS-JEPA, a framework that directly predicts the current encoder's continuous representations at masked positions, without contrastive learning, discrete targets, or separate EMA target encoders. We prevent representation collapse using SIGReg representation-space regularization, eliminating the need for engineered target-generation mechanisms. Pretrained on 960 hours of LibriSpeech, our 57M-parameter model achieves a 6.89% WER on frozen-encoder SUPERB ASR and a 25.87% CER on slot filling, outperforming the best non-distilled sub-90M baselines by 43.1% and 22.0%, respectively. These results demonstrate that highly competitive speech representations can emerge from a radically simplified training recipe.
Gaspard Botté, Séverin Baroudi, Samir Sadok +5
Concordia University · Mila – Qu´ebec AI Institute · Univ Toulon, Aix Marseille Univ, CNRS, LIS +4
Discrete speech tokens obtained from self-supervised learning (SSL) models provide efficient data compression while maintaining strong performance, and have been widely used as intermediate representations in various tasks. However, discretization inevitably causes information loss, leading to degraded performance compared with continuous SSL features. In this work, we propose to apply soft token assignment only during downstream inference. This approach preserves the efficiency of hard discretization during training while enhancing the expressiveness of the tokens at inference. The proposed method outperforms conventional hard assignment on both ASR and speech synthesis tasks, and exhibits particularly strong generalizability to out-of-domain data. For ASR of non-native speech, it even surpasses models using continuous SSL features. Moreover, analysis of the resulting representations shows they align more accurately with phonemes compared with conventional hard assignment.
Kentaro Onda, Satoru Fukayama, Daisuke Saito +1
The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan