Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment
Authors: Corentin Caris, Louis Jalouzot, Nona Rajabi, Géraud Ilinca, Ahmed Hassayoune, Asilbek Muminov, Thaddée Delebarre, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual, Antoine Collas
Organizations: Karavela · UNICOG, CNRS, INSERM, CEA, Paris-Saclay University · LSCP, EHESS, ENS, CNRS, PSL University · Inria, CEA, Paris-Saclay University
Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.
Figures & tables
Figure 1: Temporal alignment and stimulus decoding. A. A temporal offset (lag) aligns each stimulus-context endpoint with the central fMRI frame to account for the hemodynamic delay. B. A decoder maps brain recordings to vector representations of segments produced by pretrained stimulus encoders. Timing values are schematic.
Figure 2: NOEMA architecture. A. Information flow from a window of brain recordings to the predicted multimodal embedding. Tensor dimensions are shown beside each stage; only the central-frame representation is projected into the multimodal embedding space. B, N, V, D, H denote the batch size, number of frames, cortical features, target dimension, and hidden dimension respectively. B. Structure of a residual temporal convolutional block (TCN block), with a skip connection that bypasses both convolutions and adds the block input before layer normalization. These convolutions use stride one, include biases, and preserve the hidden dimension; symmetric padding preserves the temporal length.
Figure 3: Top-10@2000 stimulus retrieval across decoder configurations. Baselines are shown in light gray - with horizontal, dotted and diagonal hatchings for Ridge, MindEye MLP, and MLP respectively - Context MLP in dark gray, and NOEMA in indigo. Subject columns show mean accuracy and standard deviation across five folds, with folds indicated by crosses. In the Mean column, circles report subject means, while bars and error bars show the mean and 90% Student- t confidence interval across the three subject means.
Figure 4: Effect of stimulus representations and temporal context ( Cephalonauts One ). All panels show the gain in Top-10@2000 accuracy (percentage points) over a reference configuration, computed within each model, subject, and fold. (a) Adding audio to a text-only target (T → T+A). (b) Adding speech to a text and audio target (T+A → T+A+S). T: Qwen text; A: LCO audio; S: Parakeet speech (ASR). ΔΔ is the mean NOEMA gain minus the mean Context MLP gain. (c) Gain from increasing the input from one frame to 3, 5, or 7 frames, for Context MLP and NOEMA. Here, ΔΔ compares the two models’ gains from one to seven frames. Circles show subject means over five folds. Bars in (a, b) and lines in (c) show the mean across three subjects. Error bars and shaded bands show 90% Student- t confidence intervals. See Appendix F.2 and F.3 for absolute accuracies.
Figure 5: Extended retrieval sets ( Cephalonauts One ). Up to 8,494 segments from unseen episodes are added to each test-session retrieval set. (a) Increase in median rank versus number of added samples. (b) Median rank relative to the original set versus retrieval-set size; the dashed line marks proportional growth, expected if external samples were as confusable as in-set ones. (c) Median rank divided by retrieval-set size. Lower is better. Circles show subject means over five folds; lines and bands show the mean and 90% Student- t confidence interval across three subjects.
Figure 6: Feature importance across input frames. Absolute integrated-gradients attributions with respect to the 7 fMRI frames which NOEMA takes as input. The figure shows the left and right lateral views for Cephalonauts One ’s sub-01 . Other views and more details on the method can be found in Appendix J .
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Participant
Sessions
Runs
Hours
Cephalonauts One
sub-00
24
120
29.94
sub-01
24
121
30.24
sub-02
24
117
29.13
LeBel
UTS01
15
82
16.14
UTS02
15
83
16.37
UTS03
15
83
16.37
Appendix
Table 1: Number of sessions, runs, and hours per participant. Counts cover all training, validation, and test sessions, with each run counted once. LeBel localizer runs and repeated presentations of Where There’s Smoke are excluded. Hours are computed from the number of native fMRI frames multiplied by the TR, before temporal downsampling and window selection.
Operation
Shape
Params
Compute
Input window
N×V
–
–
Conv1,1,H
N×H
VH
NVH
GELU; dropout p=0.7
N×H
–
NH
Conv3,1,H(1)
N×H
H2
NH2
GELU; Conv1,1,H(1)
N×H
H2
NH2
Residual; LayerNorm
N×H
H
NH
Appendix
Table 2: NOEMA decoder architecture. Shapes are time × features, with N=7 frames, V=20,484 cortical vertices, H=4096 , and D=8704 . ConvK,d,C(ℓ) is a temporal convolution with kernel size K , dilation d , and C output channels, in TCN block ℓ (blank for the spatial projection); LinearC is a linear layer with C output channels. Parameter counts and per-example compute are given as orders of magnitude, omitting constants and biases.
Model
Hyperparameter
Search space
Sampling
Neural models
Batch size
{2048,4096,8192}
categorical
Hidden width
{1024,2048,4096,8192}
categorical
Temperature
[10−3,1]
log-uniform
Learning rate
[10−5,10−2]
log-uniform
Weight decay
[10−5,10−2]
log-uniform
Ridge
Penalty α
[10−1,105]
log-uniform
Appendix
Table 3: Hyperparameter search space. Bayesian search with Optuna on Cephalonauts One sub-01, using five session-based training/validation splits of the non-test sessions. Each trial minimizes the validation median relative rank averaged over folds. Neural models are NOEMA, both MLP variants, and MindEye-MLP; hyperparameters not listed are fixed to the values in Section 4.2 .
Model
Batch size
Width
τ
Learning rate
Weight decay
Val. rel. rank ↓
NOEMA
4096
4096
6.23×10−3
1.13×10−4
6.79×10−3
0.0077±0.0024
Context MLP
4096
4096
8.79×10−3
3.68×10−5
5.33×10−3
0.0128±0.0029
MLP
8192
8192
9.12×10−3
1.30×10−4
4.26×10−5
0.0193±0.0047
MindEye-MLP
4096
2048
8.83×10−3
6.65×10−4
2.95×10−5
0.0203±0.0046
Ridge
–
–
–
–
–
0.0919±0.0186
Appendix
Table 4: Selected hyperparameters. Best Optuna trial for each decoder, from the search space in Table 3 . The last column gives the validation median relative rank (mean ± standard deviation over five folds; lower is better). Ridge uses α=4.50×103 . Experiments use temperature, learning rate, weight decay, and Ridge penalty rounded to two significant figures; batch sizes and hidden widths are used exactly as listed.
Figure 8: Sensitivity to the alignment offset ( Cephalonauts One , sub-01). Columns: Top-1@2000 and Top-10@2000 accuracy (higher is better), and median relative rank (lower is better). Top row: absolute values. Bottom row: values as a percentage of each model’s value at the default δ=2.5 seconds (dotted lines), computed within each fold before averaging. Lines show the mean over five folds; shaded bands show one standard deviation.
Figure 9: Effect of the ASR context window. Top-10@2000 accuracy (higher is better) and median relative rank (lower is better) as a function of the trailing Parakeet window, for Cephalonauts One sub-01 (left pair) and LeBel UTS01 (right pair). Dotted lines mark the default window, equal to the marker spacing Δ (2.08 and 2.0 seconds). Lines show the mean over five folds; shaded bands show one standard deviation.
Figure 10: Top-1@2000 accuracy and median relative rank. Top: Top-1@2000 accuracy (higher is better). Bottom: Median relative rank of the correct segment, on a log scale (lower is better). Models and colors follow Figure 3 . In subject columns, bars and error bars show the mean and standard deviation over five folds (crosses); in Mean columns, they show the mean and 90% Student- t confidence interval over three subjects (circles).
Figure 11: Cumulative distribution of retrieval ranks. Percentage of test queries whose correct segment ranks at or above a given position, for the models in Figure 3 ; ranks are on a log scale. Values at ranks 1 and 10 equal Top-1@2000 and Top-10@2000 accuracy. Rows show datasets; columns show individual subjects and the mean over subjects, with folds weighted equally. Higher curves indicate better retrieval.
Figure 12: Distribution of retrieval ranks. Density of the correct segment’s rank for the models in Figure 3 , with ranks binned on a log scale (rank k falls in [k,k+1) ). Density is per decade of rank, so the area under each curve over log10 rank is one. Panels follow Figure 11 . Curves shifted to the left indicate better retrieval.
Figure 13: Effect of stimulus representations and temporal context ( LeBel ). Gain in Top-10@2000 accuracy (percentage points) over a reference configuration, computed within each model, subject, and fold. (a) Adding audio to a text-only target (T → T+A). (b) Adding speech to a text-and-audio target (T+A → T+A+S). (c) Increasing the input window from one frame to three, five, and seven frames (Context MLP and NOEMA only). T: Qwen text; A: LCO audio; S: Parakeet speech. ΔΔ is the mean NOEMA gain minus the mean Context MLP gain, at seven frames in (c). Circles show subject means over five folds; bars in (a, b) and lines in (c) show the mean over three subjects, with 90% Student- t confidence intervals.
Figure 14: Top-10@2000 accuracy across stimulus representations. Absolute accuracies underlying panels (a) and (b) of Figures 4 and 13 , for (a) Cephalonauts One and (b) LeBel . T: text; A: audio; S: speech. MLP uses one input frame; Context MLP and NOEMA use seven. Circles show subject means over five folds; bars and error bars show the mean and 90% Student- t confidence interval over three subjects.
Figure 15: Top-10@2000 accuracy with increasing temporal context. Absolute accuracies underlying panel (c) of Figures 4 and 13 , for (a) Cephalonauts One and (b) LeBel . Circles show subject means over five folds; lines and shaded bands show the mean and 90% Student- t confidence interval over three subjects. The wide bands on LeBel reflect differences in overall accuracy between subjects; the within-subject gains in Figure 13 c are the more direct comparison.
Figure 16: Retrieval accuracy versus decoder parameter count. Context MLP uses one, three, five, or seven input frames; NOEMA is shown with seven frames only. MindEye MLP and Ridge use one frame. Marker area is proportional to the number of input frames. Points show mean Top-10@2000 accuracy across three subjects after averaging five folds per subject. Counts include decoder weights, biases, and normalization parameters, and exclude frozen stimulus encoders. All decoders use 20,484 input vertices and 8,704 output features. Hidden widths are 4,096 for Context MLP and NOEMA, and 2,048 for MindEye MLP.
Figure 17: Sensitivity of Context MLP and NOEMA to fMRI frame order on Cephalonauts One . We randomly permute the temporal dimension of each seven-frame fMRI input. Global shuffling applies a single fixed permutation to all samples across subjects, folds, and data splits. Per-sample shuffling assigns each sample its own permutation, fixed across epochs. Context MLP and NOEMA are trained from scratch for each condition, with the same shuffling rule applied during training, validation, and testing. We report the change in Top-10@2000 accuracy relative to training and evaluating on the original frame order, paired by model, subject, and fold. Each circle represents one subject: paired differences are first averaged over available folds within each permutation seed, then equally across permutation seeds within the subject. Bars show means across three subjects, with 90% Student- t confidence intervals.
Figure 18: Retrieval performance versus training-data duration. Rows show Top-1@2000 accuracy, Top-10@2000 accuracy, and median relative rank (lower is better). Each curve represents one model and subject. Points average five session folds; shaded bands show ±1 standard deviation across folds. Color identifies the model; line style and marker identify the subject. All horizontal axes and the median-relative-rank vertical axes are logarithmic.
Figure 19: NOEMA feature-importance maps. Rows show the left lateral, left medial, right medial, and right lateral views and columns show frames 1–7.
Figure 20: Context MLP feature-importance maps. Rows show the left lateral, left medial, right medial, and right lateral views and columns show frames 1–7.
Figure 21: One-frame MLP feature-importance maps. From left to right: left lateral, left medial, right medial, and right lateral views.
Cephalonauts One is a whole-brain 3 Tesla (3T) functional magnetic resonance imaging (fMRI) dataset recorded while subjects listened to audio podcasts. Three healthy subjects underwent multiple scanning sessions, each consisting of five 15-minute runs, while listening to podcasts in their native language. With 30 hours of fMRI data per subject, the current release is the deepest available fMRI dataset using naturalistic speech stimuli. The dataset pairs brain activity with the corresponding podcast audio, transcript annotations, and derived stimulus embeddings. Furthermore, we introduce a brain decoding benchmark formulated as audio segment retrieval: given fMRI activity from a held-out session, the decoder must identify the corresponding time-aligned podcast audio segment among candidate segments. We provide standardized splits, evaluation metrics, and baseline decoders for this task. Finally, a scaling analysis shows that decoding performance improves continuously with the amount of training data per subject.
Antoine Collas, Louis Jalouzot, Géraud Ilinca +11
Karavela · UNICOG, CNRS, INSERM, CEA, Paris-Saclay University · LSCP, EHESS, ENS, CNRS, PSL University +3
Decoding inner speech from non-invasive brain signals remains a fundamental challenge due to the absence of overt linguistic output, limited training data, and large inter-subject variability. Existing brain-to-text approaches often rely on task-specific decoder fine-tuning, which restricts scalability and complicates adaptation to new participants. We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals without modifying the underlying language model. The first stage learns a subject-specific neural-semantic alignment that maps fMRI activity into a shared multimodal semantic space, extracting a latent semantic sketch of the internally generated sentence. The second stage integrates this sketch with visual context to prompt a frozen multimodal language model for free-form generation. Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines. We further show that the learned semantic-to-language projection can generalize across subjects, enabling effective decoding when paired with subject-specific neural alignment. These results indicate that neural signals modulate semantic content beyond image-driven priors, supporting a scalable and modular direction for brain-to-text decoding.
Muxuan Liu, Ichiro Kobayashi, Satoshi Nishida
Graduate School of Humanities and Sciences Ochanomizu University · Center for Information and Neural Networks Advanced ICT Research Institute National Institute of Information and Communications Technology
Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain-speech alignment ground neural activity to a single anchor modality-audio or text-despite the brain's inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail. We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio-text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training.
Suhyun Kim, Jinmo Han, Danny Dongyeop Han +9
Yonsei University, Seoul, Republic of Korea · Seoul National University, Seoul, Republic of Korea · Dartmouth College, Hanover, NH, USA