Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

Zhuofan Josh Ying; Guanglong Sun; Nolan Ho; Kieran Liu; Rui Yang; Andrew Chen; Nikolaus Kriegeskorte; Pony Ma

cs.LGOct 19, 2026

Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

Authors: Zhuofan Josh Ying, Guanglong Sun, Nolan Ho, Kieran Liu, Rui Yang, Andrew Chen, Nikolaus Kriegeskorte, Pony Ma,

Organizations: Columbia University · Mind Lab · Tsinghua University · HKUST(GZ)

Abstract

Brain encoding models aim to predict neural responses to stimuli. The most widely used approach fits readouts from frozen pretrained backbones. Using neural data to train large models from scratch or fully fine-tune them after task-pretraining has proven difficult because neural data is costly and limited. We show that parameter-efficient fine-tuning (PEFT) enables us to effectively adapt very large pretrained backbones for brain encoding with a small number of trainable parameters. To make it computationally practical, we exploit the backbone's causal structure to predict multiple timepoints in a single pass instead of using sliding windows, significantly reducing feature-extraction cost. This enables efficient fine-tuning of multimodal backbones with up to 104B parameters, the largest model adapted for brain encoding to our knowledge. Trained with a low-rank adapter (LoRA) and a lightweight readout, our approach consistently improves brain encoding performance across nine diverse multimodal backbones. On the same Qwen3-Omni backbone as the prior state-of-the-art (SoTA), our method BrainLoRA achieves the best published performance on the Algonauts 2025 Challenge. We also show the adapted multimodal backbones remain interpretable: modality attribution recovers the canonical sensory topography that prior work often obtains by construction from separate unimodal encoders, and neural adaptation increases the contribution of audio inputs and cross-modal mixing within the backbone. To scale out the number of adapted models, we develop a multi-LoRA kernel that accelerates joint training of multiple LoRAs on a shared backbone. This enables an ensemble of 200+ adapted models that ranks first on the public Algonauts post-challenge leaderboard at the time of writing. Together, our results highlight PEFT as an efficient and powerful paradigm for brain encoding.

Figures & tables

Figure 1: From frozen backbones to PEFT, and three directions to scale PEFT for brain encoding. (a) The dominant paradigm: a frozen pretrained backbone with readouts trained for brain encoding. The backbone prior is strong, but it is not specialized for brain encoding. (b) We advocate using PEFT to adapt strong pretrained backbones, retaining the powerful prior while allowing flexible neural adaptation. This suggests three complementary scaling directions: Scale up to stronger priors leveraging more capable backbones; Scale down the size of trainable adapters for brain encoding with limited data; and Scale out the number of adapters with different specializations.

Figure 2: One causally masked pass replaces a sliding window. The baseline re-encodes an overlapping window per fMRI timestep; a single pass over the window gives features at every timestep.

Figure 3: Adaptation improves performance on nine tested backbones. All backbones share the same training configuration except for the feature extraction layer (App. 15 ). LoRA improves the performance consistently.

Single model
	Trainable	Pearson $r$
Model	params	val	ID	OOD
TRIBE † ( d’Ascoli et al., 2026 )	–	–	0.303	0.196
MIRAGE † ( Gokce et al., 2026a )	1.1B	0.319	0.310	0.217
Linear + frozen backbone	52.3M	0.291	0.287	0.187
Linear + full finetuning	20.3B	0.323	0.320	0.217

Table 1: Main results on CNeuroMod/Algonauts 2025. Validation is on Friends season 6; ID and OOD are the challenge server’s scores on held-out Friends season 7 and OOD movies. Two rows within

0.00077

are not separated by seed variability (App. 10 ). Rows marked † are cited authors’ own numbers quoted from the papers. Our method BrainLoRA achieves a new state of the art.

Configuration	s06
Frozen backbone	0.2907
+ LoRA r16 (ours)	0.3263
without closed-form re-fit	0.3202
Rank 32	0.3268
Gated-MLP readout	0.3269
Cross-attention readout	0.3310

Table 2: Ablation. LoRA r16+linear readout, 60s context lead, closed-form re-fit. The first block builds our main configuration; the rest vary one choice from the main configuration. Rows within the 0.00077 seed spread are not separable.

Figure 4: Modality attribution recovers the canonical topography, and adaptation reweights and mixes the modalities. (a) Contribution to encoding accuracy of the video, audio, and text content, on our LoRA+linear model, left hemisphere (lateral above, medial below). (b) Modality contribution by region. The Shapley terms from the three modalities and a small layout residual sum to the region’s $r$ . (c) Change in modality contribution from the frozen to the adapted model. Audio contribution increases the most, while text falls. (d) Share of the prediction carried from each stimulus modality (left) to the backbone’s output at each modality’s tokens (right), which is the readout’s input. Crossing ribbons are cross-modal mixing within the backbone, which nearly doubles after adaptation.

Figure 5: Scaling out: training a population of adapted models. (a) Training-step time for $N$ rank-16 adapters on a shared backbone: one fused batched gather-matmul vs. $N$ sequential runs. The fused step matches the cost of a single adapter on the same batch (App. 9 ). (b) Ensemble performance vs. ensemble size, averaged over random subsets of the 261 adapted models, which is nearly a straight line in the log-log space (App. 13 ). (c) Performance of 2,000 random 20-model ensembles vs. the performance predicted linearly on two properties of their members: mean single-model accuracy (quality) and mean pairwise residual correlation (diversity). Neither factor predicts ensemble accuracy alone ( $R^{2}$ 0.23 and 0.05), but together they predict the performance well ( $R^{2}{=}0.97$ ).

	Extract	MIRAGE head (1,123M)			Linear (52.3M)			Train
Stimulus design	(GPU-h)	frozen	LoRA	+ re-fit	LoRA	+ re-fit	s/step	(GPU-h)
Sliding window, full-res. media	$\sim$ 700	0.3187	–	–	–	–	–	–
Sliding window, $336{\times}224$ media	$\sim$ 700	0.3178	–	–	–	–	–	–
Single pass, whole clip	1.99	0.3149	–	–	–	–	–	–
Single pass, whole clip, $336{\times}224$	1.99	0.3137	–	–	–	–	–	–
+ window-local (149 s), no lead	1.99	0.3091	–	–	0.3148	0.3206	9.7	4.0

Table 3: From MIRAGE’s sliding window to a single causal pass. Friends s06 subject-mean Pearson over four subjects. The sliding-window protocol re-encodes the preceding stimulus at every timestep (

\sim

2,800 forward passes per clip); a single causal pass encodes each clip or window once, and a context lead supplies the preceding stimulus that the first timesteps of a window would otherwise lack. Step time and training cost are for the linear readout. The sliding-window cost is the

\sim

700 GPU-hours reported by the baseline’s authors, which we reproduced on our hardware (App. 9 ). The frozen MIRAGE head was trained at a 30 s lead and has no longer-lead entries; with LoRA the head was trained at 30 and 60 s, and its re-fit re-solves the per-subject output layer in closed form. Both LoRA column pairs show seed 0. The linear readout has no whole-clip entries because whole-clip encoding is the limit of the context length sweep.

Figure 6: Accuracy within the window. Accuracy against TR position inside the 149 s window for three input designs, scored with the same frozen MIRAGE head on Friends s06 (606 complete interior windows; the band is the s.e.m. over windows for the no-lead configuration). The dashed box marks the first 10 s, where the designs differ. Accuracy here is the per-TR spatial correlation between predicted and measured 1000-parcel patterns, which is defined at a single time point; it is not comparable to the per-parcel headline metric. Without a lead, a window starts 0.110 below its plateau and needs about five TRs to reach it. With a 30 s lead it starts 0.048 below plateau, and with the whole clip 0.051. The first window of each clip is excluded because of a clip-onset transient.

Figure 7: Accuracy against cost. Held-out Friends S7 accuracy against trainable parameters (a) and against end-to-end compute including feature extraction (b) , both on log scales. Published baselines are squares and our models circles; an

N

-model ensemble is plotted at

N

times a member’s cost.

Hardware and precision
	Frozen + 1,123M readout	LoRA + 52.3M readout
Device	1 $\times$ A800-80GB
Precision	bf16 backbone, fp32 patch-embed conv
Stimulus encoding
Protocol	sliding window	single causal pass
Forward passes / clip	2,758	5.1

Table 4: Compute for the two single-model configurations. Both columns use one A800-80GB, the same 331-clip corpus and the same 336

\times

224 media cache (built once on CPU and shared). Sliding window re-encodes a capped prefix (60 s audio, 4 s video, 1,024 words) at every cutoff; single causal pass encodes each 149 s window once with a 60 s lead. Sliding-window extraction was measured over the 49 validation clips and scaled to 331 clips. Baseline training is our reproduction of the published head recipe. The closed-form re-fit is included in our column because all reported accuracies are post-re-fit. Extraction is 352

\times

cheaper and the end-to-end cost 83

\times

lower.

Figure 8: Multi-adapter kernel benchmark. Qwen3-Omni truncated at the feature extraction layer, rank-16 adapters on

q/k/v/o

, sequence length 128, one sample per adapter, one A800, median of 5 timed steps after 2 warm-up steps. (a) One fused batched gather-matmul over

N

different adapters (blue) matches the same batch through one adapter (grey); running the

N

jobs one at a time (orange) grows linearly in

N

. (b) The ratio of the fused pass to the single-adapter batch is

1.00\times

throughout, up to run-to-run noise of

\pm 0.03

Rank	Readout lr	Runs	s06 (mean)	s.d.
16	3 $\times 10^{-4}$	4	0.3262	0.00007
32	3 $\times 10^{-4}$	1	0.3268	–
32	5 $\times 10^{-4}$	9	0.3257	0.00109
64	5 $\times 10^{-4}$	4 (+1 diverged)	0.3237	0.00128
128	5 $\times 10^{-4}$	0 (+2 diverged)	–	–

Table 5: LoRA rank. Shipped recipe otherwise unchanged ( Qwen3-Omni , linear readout, extraction layer 32, 60 s lead, LoRA lr

6\times 10^{-4}

, 1,429 steps), Friends s06 subject-mean after the closed-form re-fit. Rank and readout learning rate were not varied independently; the matched comparison is the two rows at

3\times 10^{-4}

. Diverged runs are counted in parentheses.

Feature-extraction layer	Frozen	Trained
16	0.2682	–
24	0.2839	–
30	0.2889	0.3252
31	0.2885	0.3260
32	0.2881	0.3263
33	0.2872	0.3258

Table 6: Feature-extraction layer. Frozen is the closed-form ridge readout on unadapted features (no adapter, no training); trained is the full recipe, one seed per layer, after the re-fit. The frozen screen peaks at layer 30; the trained runs at layers 31–34 lie within 0.0006 of one another, inside the 0.00077 seed spread of this configuration.

LoRA lr	Readout lr	s06 (pre-re-fit)
0	2e-5	0.2818
1e-3	1e-4	0.3071
1e-3	3e-4	0.3131
6e-4	1e-3	0.3076
6e-4	3e-4	0.3134

Table 7: Learning rates. Measured in an earlier configuration without context lead or closed-form re-fit, so these are pre-re-fit validation scores about 0.013 below the final recipe, comparable to each other but not to Tab. 1 . The per-modality projection uses

2\times

the readout rate. The first row freezes the adapter (readout-only control).

Step	200	400	600	800	1000	1200	1400	1600	1800	2000	2200	2400	2600	2800	2858
s06	0.2789	0.2856	0.2877	0.2921	0.2977	0.3007	0.3052	0.3009	0.2999	0.2996	0.3017	0.3019	0.3031	0.3030	0.3028

Table 8: A second epoch overfits. Validation score through a 2858-step (two-epoch) run, same configuration as Tab. 7 . The peak is at step 1400, the end of the first pass over the 1,429 training windows. Matched one- versus two-epoch pairs give

-0.0147

at readout lr

3\times 10^{-4}

-0.0047

at readout lr

1\times 10^{-4}

Prefix ends at	audio	vision	text
140 s	0.9579	0.9895	0.9960
160 s	0.9569	0.9908	0.9970
180 s	0.9553	0.9901	0.9969
200 s	0.9584	0.9906	0.9973
same span twice	1.0000	1.0000	1.0000

Table 9: Cosine between representations at shared timesteps, prefix pass against single pass, on friends_s01e01a with a 209 s span, feature extraction layer 32.

Configuration	Members	Seeds
shipped recipe	34	15 seeds
rank 32, readout lr 0.0005	27	13 seeds
lead 30	16	8 seeds
backbone Qwen3.8-27B , chunk 2, no audio	10	6 seeds
readout cross-attn (tokens), tokens stream	8	3 seeds
bag 0.6	6	2 seeds

Table 10: The 261-model pool. Each row is a distinct configuration; members differing only in random seed are grouped, with the seed count given. Entries list only where a member departs from the shipped recipe ( Qwen3-Omni backbone, linear readout, feature extraction layer 32, rank 16, 60 s lead, 1,429 steps). Counts sum to 261.

Method	Params	s06	$\Delta$ frozen	% of LoRA gain	$\Delta$ / M params
Frozen backbone (readout only)	0	0.2907	—	—	—
IA 3 , attention + FFN	0.057M	0.3127	$+0.0220$	62%	0.3833
IA 3 , + MoE experts	3.18M	0.3177	$+0.0270$	76%	0.0085
Bottleneck adapter, FFN only ( $r{=}32$ )	4.26M	0.3164	$+0.0257$	72%	0.0060
Bottleneck adapter, attention + FFN ( $r{=}32$ )	8.52M	0.3190	$+0.0283$	79%	0.0033
LoRA $r{=}16$ (ours)	8.91M	0.3263	$+0.0356$	100%	0.0040

Table 11: Three PEFT families at one protocol. Friends s06 subject-mean after the closed-form re-fit, at the main configuration ( Qwen3-Omni , linear readout, feature extraction layer 32, 60 s lead, 1,429 steps, one seed), with only the adaptation method changed. The column Params is backbone-side trainable parameters, read from each checkpoint; the 52.3M readout is common to every row and is excluded. The column of LoRA is the gain over the frozen backbone as a fraction of LoRA’s, and the last column is that gain per million adapter parameters. IA 3 learns three element-wise rescaling vectors per block (keys, values, FFN intermediate) and adds no weights; a bottleneck adapter inserts

x+W_{\mathrm{up}}\,\mathrm{gelu}(W_{\mathrm{down}}x)

after the attention sub-layer and the FFN (Houlsby) or after the FFN alone (Pfeiffer). Each family runs at its own screened rate (

1\times 10^{-2}

3\times 10^{-4}

6\times 10^{-4}

), since a PEFT rate does not transfer between families.

Backbone	Params	Extraction layer	Frozen	+LoRA	$\Delta$	+cross-attn
Phi-4-MM	4.7B	20	0.2521	0.2841	+0.0320	0.2888
Qwen2.5-Omni 3B	6.0B	30	0.2807	0.3139	+0.0331	0.3175
Gemma 3n E4B	7.8B	25	0.2631	0.3026	+0.0395	0.3049
LCO-Omni 7B	8.9B	26	0.2683	0.3199	+0.0517	0.3200
MiniCPM-o	9.4B	28	0.2594	0.3249	+0.0655	0.3265
Qwen2.5-Omni 7B	11.2B	24	0.2748	0.3178	+0.0431	0.3214

Table 12: The nine omni-modal backbones. Total parameters from each checkpoint’s own index ( Qwen3-Omni and Ming are mixture-of-experts, with 3B and 6B active per token); feature extraction layer is chosen per backbone. All arms share the recipe of § 5.2 : rank-16 LoRA, 60 s context lead, closed-form re-fit, Friends s06 subject-mean.

Figure 9: Cross-seed agreement. Mean agreement between adapters over the 15 seeds of the main configuration (105 pairs). (a) At successive levels of the model, from near zero for the weights to near one for the prediction. Levels below the first are cosines between the two seeds’ changes relative to the frozen model, averaged over the audio, video, and text streams. (b) Agreement at every decoder layer, per stream. It rises through the adapted stack, peaks at the feature extraction layer, and declines slowly through the 16 frozen layers above it.

Configuration	Training targets	s06
LoRA + linear (ours)	aligned	0.3263
Frozen backbone	aligned	0.2907
Frozen backbone	shifted	0.2617
LoRA + linear	shifted	0.1874

Table 13: Shifted-target negative control. Friends s06 subject-mean after the closed-form re-fit on aligned training data, one run per configuration. Differences are paired per parcel with a functional-block bootstrap interval; the last column counts the parcels on which the first configuration scores higher.

Held-out	frozen	3-subj., held out	3-subj., in-sample	4-subject	vs. frozen	vs. in-sample	vs. 4-subj.
sub-01	0.2875	0.3210	0.3215	0.3229	+0.0335	-0.0005	-0.0019
sub-02	0.2968	0.3286	0.3286	0.3301	+0.0317	-0.0000	-0.0015
sub-03	0.3130	0.3485	0.3490	0.3497	+0.0355	-0.0005	-0.0012
sub-05	0.2734	0.3034	0.3014	0.3025	+0.0301	+0.0021	+0.0010

Table 14: Transfer to an unseen subject. Validation score per subject, with the readout solved in closed form, under: the frozen backbone; an adapter trained on the other three subjects only (bold, the transfer); the mean of the three 3-subject adapters that did train on the subject (in-sample); and the shared 4-subject adapter. The last three columns compare the held-out score with each of the others.

	subject-mean $r$			% of noise ceiling
	frozen	LoRA	gain	frozen	LoRA	gain, points [95% CI]
Algonauts, validation	0.2907	0.3263	+0.0356	81.3	92.0	+10.7 [+10.0, +11.4]
NNDb, readout transferred	0.0991	0.1223	+0.0232	43.5	54.6	+11.1 [+10.5, +11.6]
NNDb, readout re-fitted	0.0914	0.1252	+0.0338	40.3	55.7	+15.4 [+14.6, +16.2]

Table 15: Transfer to a second dataset. The same LoRA and frozen-backbone models on Algonauts validation and on NNDb (20 participants, one film; different site, scanner, preprocessing and TR). Transferred applies the Algonauts readout unchanged; re-fitted solves a fresh readout per NNDb participant under cross-validation over contiguous film time.

r

columns are subject means over all 1,000 parcels. Ceiling columns give

r

as a percentage of each dataset’s noise ceiling, over the 922 (Algonauts) and 816 (NNDb) parcels with an estimable ceiling, with a 95% parcel-block bootstrap on the gain. Paired over the 800 parcels both datasets admit, transferred minus Algonauts is

+0.55

[

+0.01

+1.06

] points.

Figure 10: Performance against the noise ceiling on NNDb. 20 participants, readout re-fitted per participant on the fixed backbone, over the 816 parcels with an estimable ceiling. (a) Subject-mean $r$ per parcel against the corrected noise ceiling (dashed line). (b) Mean percentage of the ceiling per Yeo network.

	calibration data from the new subject (min)
Algonauts, held-out subject	0	5	10	20	40	79	159	all
Ridge from scratch	–	0.1495	0.1967	0.2273	0.2493	0.2687	0.2869	0.3254
Low-rank deviation ( $r{=}16$ )	0.2574	0.2572	0.2578	0.2591	0.2603	0.2621	0.2642	–
Anchored ridge	0.2574	0.2597	0.2641	0.2718	0.2789	0.2889	0.2997	–
LoRA $-$ frozen (anchored)	+0.0236	+0.0254	+0.0260	+0.0272	+0.0273	+0.0298	+0.0326	+0.0326

Table 16: Calibrating a new subject with little data. Validation

r

of a subject whose fMRI was never used for training the adapter, with its readout fitted on

K

windows (149 s each) of its own data, shown in minutes. Bold marks the best estimator at each budget. The last row is the gain of the adapted model over the frozen backbone under anchored ridge, which beats it for every subject and draw (20/20) and every NNDb participant (20/20) at every budget. Top: the four 3-subject adapters of Tab. 14 , mean over the four held-out subjects and five random draws of the calibration windows; all is the held-out column of Tab. 14 , solved on all of that subject’s training data. Bottom: the LoRA + linear model of Tab. 1 on NNDb, 20 participants, a fixed 7-window test segment and a contiguous calibration run, averaged over placements at the start and end of the film. Its 0 min scores therefore differ from Tab. 15 , which scores the whole film.

region	$n$	frozen	LoRA	gain [95% CI]	rel. %
Yeo 7 networks
DorsAttn	122	0.330	0.380	+0.0502 [+0.0468, +0.0533]	15
Vis	162	0.398	0.438	+0.0401 [+0.0364, +0.0440]	10
Cont	129	0.279	0.317	+0.0379 [+0.0347, +0.0412]	14
SalVentAttn	121	0.241	0.277	+0.0352 [+0.0319, +0.0397]	15
Default	212	0.318	0.351	+0.0330 [+0.0306, +0.0355]	10

Table 17: Gain from the adapter by region. Validation

r

with and without LoRA in the same configuration (the frozen arm trains the readout with the same rates and steps), over the

n

parcels of each region: the seven Yeo networks, then ten ROIs defined from parcel centroids. Arms are paired per parcel; gain intervals are functional-block bootstraps over parcels, and the last column is the gain relative to the frozen score. One run per arm; averaging the LoRA map over 15 seeds leaves both orderings unchanged.

Figure 11: Validation

r

by region, with and without the adapter. Frozen backbone and the same configuration with LoRA, over the

n

parcels of each region (Tab. 17 ); bars show the gain, printed at the right. (a) The seven Yeo networks. (b) Ten coordinate-defined ROIs. One run per configuration, paired per parcel; every gain’s 95% functional-block bootstrap interval excludes zero.

Figure 12: All subsets of the three input streams. Subject-mean Pearson

r

on the 250 validation windows used for attribution, with absent streams replaced by their training-mean embeddings. The top bar of (a) is the full model (dashed line). (a) Any single stream recovers most of the full score, and removing one costs far less than that stream achieves alone: the streams are largely redundant. (b) Cost of removing each stream, before and after adaptation. The video cost barely changes, the audio cost grows, and the transcript cost falls by more than a third. This removes a stream from a trained model and differs from the no-audio configuration of Tab. 2 , which is trained without audio throughout.

Figure 13: Audio contribution by region, before and after adaptation. Subject-mean contribution of the audio stream to encoding accuracy for the frozen and LoRA + linear models, with episode block-bootstrap intervals; labels give the relative change. Auditory cortex gains least; the gain is largest in regions that relied least on audio before adaptation.

pool	val. $r$	explained %	missed %	cross %	missed
main single model (LoRA + linear)	0.3263	60.5	36.2	3.3	0.0480
3-member mean	0.3262	60.8	36.5	2.6	0.0485
261-member uniform mean	0.3401	66.2	32.5 [30.0, 35.3]	1.3	0.0431

Table 18: What the pool still misses. The cross-subject partition of the stimulus-locked (inter-subject) covariance on the validation episodes, as a percentage of it: the part shared by the predictions (explained), the part shared by the residuals (missed) and the cross terms, for three pools; the interval is a functional-block bootstrap over parcels, and the last column is the missed covariance in units of a unit-variance parcel.

Block	$\delta_{b}$ (s)	$w_{b}$ (s)	covers (s)	full	clipped	zero
1	0.5	2.0	$[0.5,2.5)$	98	1	1
2	2.5	2.0	$[2.5,4.5)$	99	1	0
3	4.5	4.0	$[4.5,8.5)$	97	2	1
4	8.5	8.0	$[8.5,16.5)$	92	5	3
5	16.5	16.0	$[16.5,32.5)$	81	10	9
6	32.5	32.0	$[32.5,64.5)$	60	21	19

Table 19: Lag blocks. Each block is a (delay, width) pair in seconds. Block

b

of TR

j

uniformly averages the feature frames in

[c_{j}-(\delta_{b}+w_{b})f,c_{j}-\delta_{b}f)

, where

c_{j}

is the TR’s block center and

f=2

Hz. A block that reaches before the window start is clipped, or is a zero row if it lies entirely before it. The last three columns count, over the 100 TRs of a window, the rows at full width, clipped, and zero (158 of 800 rows are zero).

Backbone	Hugging Face repository	Revision
Phi-4-MM	microsoft/Phi-4-multimodal-instruct	467a5d77dcfa
Qwen2.5-Omni 3B	Qwen/Qwen2.5-Omni-3B	f75b40e3da20
Gemma 3n E4B	unsloth/gemma-3n-E4B-it	45e9fb1dd0e3
LCO-Omni 7B	LCO-Embedding/LCO-Embedding-Omni-7B	3d38f58aae12
MiniCPM-o	openbmb/MiniCPM-o-4_5	503e754207c9
Qwen2.5-Omni 7B	Qwen/Qwen2.5-Omni-7B	ae9e1690543f

Table 20: Backbone revisions. Hugging Face repository and commit for every backbone in Tab. 12 , as loaded by our runs.

Neural Encoding

Neural encoding studies how information is represented in biological neural activity and how models map stimuli or signals into neural codes.

See all 91 papers

MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding
May 28, 2026Abdulkadir Gokce, Badr AlKhamissi, Martin SchrimpfNeuroimaging Neural Encoding
NeuroAI Lab, EPFL
Channel Adaptation for EEG Foundation Models: A Systematic Benchmark Across Architectures, Tasks, and Training Regimes
Apr 25, 2026Kuntal Kokate, Bruno Aristimunha, Dung Truong +1Fine-Tuning EEG Decoding
Swartz Center for Computational Neuroscience, Institute for Neural Computation, University of California San Diego, La Jolla, CA 92093, USA · Yneuro, Paris, France · Centre de Recherche Cerveau et Cognition, CNRS, Université Paul Sabatier, Toulouse, France
BrainCodes: A Lightweight EEG Foundation Model over Discrete Neural Codes
Oct 21, 2026Kleanthis Avramidis, Ivan Tashev, Dimitra EmmanouilidouEEG Foundation Models Representation Learning
Signal Analysis and Interpretation Lab, University of Southern California, Los Angeles, CA 90007, USA · Microsoft Research, Redmond, WA, USA

May 28, 2026J/K move · Enter open · S save

MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

Abdulkadir Gokce, Badr AlKhamissi, Martin Schrimpf

NeuroAI Lab, EPFL

Neuroimaging Neural Encoding Multimodal Fusion Multimodal Learning Multimodal Understanding Gated Multimodal Fusion

Open paper AskPDF arXiv

Recent progress in task-optimized neural networks has established encoding models as a powerful tool for predicting brain responses to naturalistic stimuli, yet most existing approaches rely on unimodal representations. The emergence of omni-modal foundation models and rich multimodal neural datasets enables encoding models that jointly integrate visual, auditory, and linguistic information across subjects. We introduce MIRAGE, a brain encoding framework for predicting whole-brain fMRI responses to naturalistic audiovisual stimuli. MIRAGE achieves state-of-the-art performance via a native multimodal backbone and adaptive feature gating across layers. These representations are then combined with a transformer-based brain encoder and a subject-specific linear head over the cortical parcels. Controlled comparisons show that natively multimodal features consistently outperform post-hoc aggregation of independent unimodal features, across architectural levels and backbones. Beyond predictive accuracy, the learned attention weights are directly inspectable to interpret the modality-specific gating profile over the backbone, and each modality traces a distinct anatomical pattern across cortex. Together, these results propose adaptive layer-wise aggregation of natively multimodal features as a generalizable, interpretable, and accurate approach for whole-brain encoding.

Abstract

Neural Encoding

Explore similar work

MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

Channel Adaptation for EEG Foundation Models: A Systematic Benchmark Across Architectures, Tasks, and Training Regimes

BrainCodes: A Lightweight EEG Foundation Model over Discrete Neural Codes