Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs' alignment with the brain can be mostly explained by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be explained by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features.
Figures & tables
Fig 1: Summary of the methodology. (a) We constructed six sets of interpretable features—mel-spectrograms, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features—and compared them to representations from three state-of-the-art speech foundation models (SFMs): Whisper, HuBERT, and WavLM. The comparison is based on each representation’s ability to predict neural responses, measured through Pearson’s r and the variance accounted for R2 . (b) An example of each linguistic feature set. The word is represented as phonetic (ARPAbet), syntactic (Part-of-Speech Tag, Dependency Parsing, stop word or not), and semantic (Question Answering) features. (c) Diagram of the variance partitioning approach.
Fig 2: Decomposing SFM brain encoding performance with interpretable features. (a) Diagram of the variance partitioning analysis used to answer three key questions about the encoding model performance of SFMs. (b) The ratio of neural variance shared by the SFM representations and the combined set of all interpretable features, relative to the total variance accounted for by the SFM representations. In early layers, the interpretable model accounts for nearly all of the neural variance, and even in later layers, it continues to account for a substantial majority. (c) The ratio of neural variance shared between different interpretable feature sets and the SFM representations, relative to the total variance accounted for by each interpretable feature set. This ratio peaks in early layers for the mel-spectrogram, but in progressively later layers for higher-level features (phonetic, syntactic, and semantic). (d) Unique neural variance of SFMs with respect to the combined set of all interpretable features, averaged across all electrodes. For all three models, the increase is monotonic over layers. (e) Unique neural variance of SFMs as a function of model size. For each model size, the layer with the highest unique variance was selected. Results show that the SFMs’ unique neural variance increases with model size. (f) Unique neural variance accounted for by the embeddings from Whisper layer 32 with respect to the combined set of all interpretable features, plotted on a template brain. The largest unique contributions appear in the superior temporal gyrus (STG) and inferior frontal gyrus (IFG), regions that are typically associated with the processing of higher-order information. In panels (b–e), shaded regions indicate 95% confidence intervals estimated via bootstrapping.
Fig 3: Interpretable features efficiently enhance the encoding model performance of SFMs. Test correlation value for each SFM and its joint model with the combined set of all interpretable features, averaged across electrodes. Shaded regions denote 95% confidence intervals estimated via bootstrapping. Left : WavLM, Middle : Whisper, Right : HuBERT. Combining interpretable features with SFMs offers two key advantages: (1) it compensates for the fact that no single SFM layer captures all information relevant for neural processing, and (2) it leverages the increasing amount of unaccounted variance in deeper SFM layers, which also scales with model size. Results show that incorporating interpretable features consistently improves encoding performance across all layers (p ¡ 0.001, Wilcoxon signed-rank test).
Model
Baseline 0
Baseline 1
Baseline 2
Our Method
(1 Layer, 1 SFM)
(Early+Late SFM Layers)
(Two SFMs)
(SFM+Interp)
Whisper
0.073
0.075
0.075
0.085
HuBERT
0.069
0.072
0.075
0.083
WavLM
0.073
0.074
0.074
0.085
Table 1: Comparison of encoding model performance (Pearson’s r) across different baselines and the joint model of interpretable features and SFM embeddings. The joint model consistently outperforms all baselines.
Fig 4: Whisper loses fine-acoustic details and gains longer acoustic temporal windows in later layers. (a) We quantify the shared neural variance between Whisper representations and mel-spectrogram features with different window lengths. (b) Average neural variance R2 shared between Whisper representations and mel-spectrogram segments of different temporal windows. Across all anatomical regions, shared variance with the short window [ t−0.05 , t ]s consistently decreases across layers, whereas shared variance with the longer window [ t−0.5 , t ]s exhibits anatomically dependent patterns. (c) Variance R2 of the mel-spectrogram [ t−0.5 , t ]s accounted for by the Whisper representations over different layers. Early layers account for a high proportion of variance for short temporal windows; conversely, later layers account for less variance but reflect information from broader temporal windows.
Fig 5: SFMs encode brain-relevant semantic information. (a) Diagram of the three-way variance partitioning analysis used to assess whether SFM representations encode semantic information. (b) Unique variance accounted for R2 by different feature sets in predicting GPT-2 representations, averaged across GPT-2’s feature dimensions ( p<0.001 , Wilcoxon signed-rank test). The significant unique contribution of our question-answering semantic features confirms that they encode distinct semantic information. (c) Unique neural variance accounted for by semantic QA features relative to non-semantic interpretable features. (d) Box plot of the values in (c), across electrodes in the inferior frontal gyrus triangularis (IFG-Tri). Median values were significantly greater than zero ( p<0.001 , Wilcoxon signed-rank test). (e) The proportion of unique semantic neural variance accounted for by SFM embeddings, averaged across electrodes in IFG-Tri. Shaded regions denote 95% confidence intervals estimated via bootstrapping. Left: Varying context lengths, using the largest variant of each SFM. Right: Varying model sizes, with context length fixed to 5 seconds for WavLM and HuBERT, and 30 seconds for Whisper. It shows SFMs encode brain-relevant semantic information that cannot be reduced to lower-level features, improving with context length and model size.