Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG

Hanning Guo; Hanwen Bi; Farah Abdellatif; Andrei Galbenus; Jon. N. Shah; Abigail Morrison; Jürgen Dammers

cs.LGFeb 26, 2026

Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG

Authors: Hanning Guo, Hanwen Bi, Farah Abdellatif, Andrei Galbenus, Jon. N. Shah, Abigail Morrison, Jürgen Dammers

Organizations: INM-4, Forschungszentrum Jülich, Germany · Department of Computer Science 3 - Software Engineering, RWTH Aachen University, Germany · INM-7, Forschungszentrum Jülich, Germany · Institute of Systems Neuroscience, Heinrich Heine University, Germany ·

Abstract

Brain foundation models have achieved remarkable advances across a wide range of neuroscience tasks. However, most existing models are limited to a single functional modality, restricting their ability to exploit complementary spatiotemporal dynamics and the collective data scale across different neuroimaging techniques. This limitation largely arises from severe semantic heterogeneity and resolution discrepancies among modalities. To address these challenges, we propose Brain-OF, an omnifunctional brain foundation model jointly pretrained on fMRI, EEG and MEG, capable of handling both unimodal and multimodal inputs within a unified framework. To reconcile heterogeneous spatiotemporal resolutions, we introduce the Any-Resolution Neural Signal Sampler, which projects diverse brain signals into a shared semantic space. To further manage semantic shifts, the Brain-OF backbone integrates DINT attention with a Sparse Mixture of Experts, where shared experts capture modality-invariant representations and routed experts specialize in modality-specific semantics. Furthermore, to explicitly internalize the characteristics of neural activity through self-supervised learning, we propose Masked Temporal-Frequency Modeling, a dual-domain pretraining objective that jointly reconstructs brain signals in both the time and frequency domains. Brain-OF is pretrained on a large-scale corpus comprising around 40 datasets and demonstrates superior performance across diverse downstream tasks, highlighting the benefits of joint multimodal integration and dual-domain pretraining.

Figures & tables

Figure 1: Overview of the Brain-OF Architecture. Brain-OF is an omnifunctional foundation model jointly pretrained on fMRI, EEG and MEG. (a) Pretraining pipeline: The original signals are masked in both the temporal and frequency domains to encourage the model to jointly learn coupled time–frequency representations. (b) ARNESS projects arbitrary resolution signals into a unified semantic space and also serves as a decoder to inverse-sample latent representations for reconstruction during pretraining. (c) Multimodal Fusion serially resamples multiple unimodal sequences via ARNESS without introducing modality-specific fusion branches. (d) Sparse MoE addresses semantic heterogeneity by dynamically routing tokens to specialized experts while extracting modality-invariant knowledge through shared experts. (e) DINT Attention mitigates attention allocation to irrelevant contextual information. Here, sFFT and tFFT denote Fast Fourier Transform along spatial and temporal dimensions; iFFT denotes the inverse Fast Fourier Transform; and M indicates the masking.

Downstream Tasks	Datasets	Modalities	#Chs/ROIs	Durations	#Subjects	#Samples	Labels
Emotion Recognition	SEED-V	EEG	62	1s	20	117,744	5-class
Seizure Detection	CHB-MIT	EEG	22	10s	22	326,993	2-class
Abnormal Detection	TUAB	EEG	19	10s	2,383	409,455	2-class
AD Diagnosis	ADNI	fMRI	246	$\sim$ 300s	222	618	2-class
ADHD Identification	ADHD-200	fMRI	246	$\sim$ 360s	926	1,369	2-class
Brain Age Prediction	CamCAN	fMRI	246	520s	652	652	regression

Table 1: Overview of downstream tasks and evaluation datasets.

Methods	SEED-V	CHB-MIT	TUAB	ADNI	ADHD-200	CamCAN	CamCAN	LEMON	LEMON
	EEG	EEG	EEG	fMRI	fMRI	fMRI	MEG	fMRI	EEG
	Kappa(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	MAE $\downarrow$	MAE $\downarrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$
EEGNet	10.06 $\pm$ 1.43	56.58 $\pm$ 1.06	76.42 $\pm$ 0.36	60.22 $\pm$ 2.01	57.66 $\pm$ 2.03	15.57 $\pm$ 1.79	11.60 $\pm$ 0.47	59.12 $\pm$ 2.10	78.84 $\pm$ 2.89
EEGConformer	17.72 $\pm$ 1.74	59.76 $\pm$ 1.41	77.58 $\pm$ 0.49	65.16 $\pm$ 2.45	62.00 $\pm$ 2.64	8.56 $\pm$ 0.72	10.03 $\pm$ 0.21	67.27 $\pm$ 4.15	78.94 $\pm$ 2.35
ContraWR	19.05 $\pm$ 1.88	63.44 $\pm$ 0.02	77.46 $\pm$ 0.41	59.69 $\pm$ 4.97	62.67 $\pm$ 1.69	9.15 $\pm$ 0.39	9.01 $\pm$ 0.40	63.97 $\pm$ 1.28	77.42 $\pm$ 0.45
CNN-Transformer	20.72 $\pm$ 1.83	63.89 $\pm$ 0.67	77.77 $\pm$ 0.22	62.27 $\pm$ 4.87	61.94 $\pm$ 0.43	16.00 $\pm$ 0.10	15.69 $\pm$ 0.09	64.83 $\pm$ 1.54	78.68 $\pm$ 0.77

Table 2: Comparison results of different methods on downstream tasks.

Methods	SEED-V	CHB-MIT	TUAB	ADNI	ADHD-200	CamCAN	CamCAN	LEMON	LEMON
	EEG	EEG	EEG	fMRI	fMRI	fMRI	MEG	fMRI	EEG
	Kappa(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	MAE $\downarrow$	MAE $\downarrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$
Brain-OF (Base)	23.98 $\pm$ 0.29	73.99 $\pm$ 1.48	81.88 $\pm$ 0.17	68.23 $\pm$ 2.07	62.57 $\pm$ 1.19	12.09 $\pm$ 0.78	8.99 $\pm$ 0.14	58.95 $\pm$ 2.13	76.05 $\pm$ 0.44
Objective Ablation
- w/o MTFM	21.52 $\pm$ 0.57	71.43 $\pm$ 2.30	81.49 $\pm$ 0.18	65.26 $\pm$ 1.83	60.35 $\pm$ 1.50	12.80 $\pm$ 1.15	8.96 $\pm$ 0.33	57.18 $\pm$ 1.91	76.90 $\pm$ 0.51
Modality Ablation

Table 3: Ablation studies on downstream tasks.

Datasets	Metrics	Modality	Models
Datasets	Metrics	Modality	Base	Large	Huge
CamCAN	MAE $\downarrow$	fMRI-only	12.09 $\pm$ 0.78	11.71 $\pm$ 0.67	9.51 $\pm$ 0.51
		MEG-only	8.99 $\pm$ 0.14	8.86 $\pm$ 0.21	7.87 $\pm$ 0.22
		Multimodal Fusion	8.87 $\pm$ 0.09	8.78 $\pm$ 0.07	8.21 $\pm$ 0.11
LEMON	BAC(%) $\uparrow$	fMRI-only	58.95 $\pm$ 2.13	62.85 $\pm$ 4.29	61.79 $\pm$ 1.29
		EEG-only	76.05 $\pm$ 0.44	78.04 $\pm$ 1.59	80.39 $\pm$ 0.59

Table 4: Performance benefits of multimodal fusion.

Figure 3: Visualization of Brain-OF Interpretability.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Modalities	Datasets	Paradigms	SR/TR	#Chs/ROIs	Durations	#Participants	#Samples
EEG	FACED	task	250Hz	32	10s	123	10,332
	PhysioNet-MI	task	160Hz	64	4s	109	18,697
	PREDICT	rest	5,000Hz	63	10s	159	17,511
	REEG-BACA	rest	1,000Hz	64	10s	608	53,089
	SEED	task	1,000Hz	62	10s	15	14,895
	SEED-DV	task	1,000Hz	62	10s	20	7,644

Appendix

Table 5: Summary of all pretraining datasets, covering 3 heterogeneous functional neuroimaging modalities (fMRI, EEG, MEG) across 37 public datasets, encompassing over 32K unique participants, approximately 5.9M samples, and 2.8 TB of preprocessed data.

Dataset	Pretraining	Relation	Train Subjects in PT	Val./Test Overlap	OOD Level
SEED-V	$\times$	Independent	$\times$	$\times$	Dataset + Subject
CHB-MIT	$\times$	Independent	$\times$	$\times$	Dataset + Site + Subject
TUAB	$\triangle$	Partial	$\checkmark$	$\times$	Subject
ADNI	$\times$	Independent	$\times$	$\times$	Dataset + Site + Subject
ADHD-200	$\times$	Independent	$\times$	$\times$	Dataset + Site + Subject
CamCAN	$\triangle$	Partial	$\checkmark$	$\times$	Subject

Appendix

Table 6: Relationship between the pretraining corpus and downstream evaluation datasets.

\checkmark

denotes inclusion in pretraining,

\times

denotes no inclusion or overlap, and

\triangle

denotes partial overlap through the same dataset or a related parent corpus. PT denotes pretraining. All downstream validation and test subjects are strictly excluded from pretraining.

Task-Specific Models (Trained from scratch)
Models	Params (M)	GFLOPs	Architecture	Pretraining Modality
EEGNet	0.004	0.13	CNN	✗
EEGConformer	0.81	0.58	Convolutional Transformer	✗
ContraWR	1.57	0.28	CNN	✗
CNN-Transformer	3.16	0.33	CNN + Transformer	✗
FFCL	2.47	0.96	CNN + LSTM	✗

Appendix

Table 7: Overview of baseline models. We categorize baselines into task-specific models (trained from scratch) and foundation models (pretrained). Params denotes parameter count in millions (M); GFLOPs measures computational cost per inference sample.

Mask Ratios	TUAB (EEG)			ADNI (fMRI)			CamCAN (MEG)
Mask Ratios	BAC(%) $\uparrow$	AUC-PR(%) $\uparrow$	AUROC(%) $\uparrow$	BAC(%) $\uparrow$	AUC-PR(%) $\uparrow$	AUROC(%) $\uparrow$	MAE $\downarrow$	R2(%) $\uparrow$	MAPE(%) $\downarrow$
$\mathcal{N}(0.4,0.05^{2})$	81.02 $\pm$ 0.12	87.17 $\pm$ 0.09	88.67 $\pm$ 0.06	61.10 $\pm$ 2.57	69.84 $\pm$ 2.36	67.54 $\pm$ 2.47	11.93 $\pm$ 0.38	32.70 $\pm$ 3.92	23.25 $\pm$ 0.77
$\mathcal{N}(0.5,0.05^{2})$	81.74 $\pm$ 0.21	87.59 $\pm$ 0.13	89.29 $\pm$ 0.19	65.01 $\pm$ 2.91	75.24 $\pm$ 2.28	71.42 $\pm$ 2.84	8.92 $\pm$ 0.18	58.19 $\pm$ 1.71	17.00 $\pm$ 0.31
$\mathcal{N}(0.6,0.05^{2})$	81.82 $\pm$ 0.31	87.64 $\pm$ 0.20	89.45 $\pm$ 0.27	63.74 $\pm$ 2.76	72.87 $\pm$ 3.12	70.85 $\pm$ 3.31	8.83 $\pm$ 0.19	57.02 $\pm$ 1.94	16.50 $\pm$ 0.48
$\mathcal{N}(0.7,0.05^{2})$	81.88 $\pm$ 0.17	87.69 $\pm$ 0.11	89.26 $\pm$ 0.09	68.23 $\pm$ 2.07	76.23 $\pm$ 1.87	73.27 $\pm$ 0.71	8.99 $\pm$ 0.14	56.19 $\pm$ 1.84	16.92 $\pm$ 0.30
$\mathcal{N}(0.8,0.05^{2})$	81.51 $\pm$ 0.09	87.46 $\pm$ 0.06	89.14 $\pm$ 0.09	67.86 $\pm$ 2.62	76.77 $\pm$ 3.17	74.53 $\pm$ 4.71	9.45 $\pm$ 0.40	53.27 $\pm$ 3.62	18.06 $\pm$ 0.51

Appendix

Table 8: Performance comparison across different mask ratios in the MTFM.

Figure 4: The impact of the router bias update rate $\gamma$ during pretraining. The performances are evaluated on downstream tasks across three modalities.

Methods	BrainHarmonix-F	Brain-JEPA	Brain-OF Base
HAD (Top-10 Acc %) $\uparrow$	6.38 $\pm$ 0.16	12.31 $\pm$ 1.69	16.17 $\pm$ 0.44
BOLD5000 (BAC %) $\uparrow$	38.94 $\pm$ 1.66	39.79 $\pm$ 0.41	40.49 $\pm$ 0.67

Appendix

Table 9: Additional downstream evaluation on task-state fMRI.

Figure 5: Qualitative visualization of signal reconstruction across fMRI, EEG and MEG.

Methods	CHB-MIT	TUAB	CamCAN
	EEG	EEG	MEG
	BAC(%) $\uparrow$	BAC(%) $\uparrow$	MAE $\downarrow$
BrainOmni Base	69.47 $\pm$ 2.19	81.90 $\pm$ 0.50	11.99 $\pm$ 0.84
Brain-OF Base	73.99 $\pm$ 1.48	81.88 $\pm$ 0.17	8.99 $\pm$ 0.14

Appendix

Table 10: Additional comparison between Brain-OF and BrainOmni.

Methods	CamCAN			LEMON
	fMRI	MEG	Multimodal	fMRI	EEG	Multimodal
	MAE $\downarrow$	MAE $\downarrow$	MAE $\downarrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$
BrainHarmonix	14.81 $\pm$ 0.49	8.73 $\pm$ 0.09	11.61 $\pm$ 0.75	63.11 $\pm$ 1.32	77.95 $\pm$ 2.36	74.43 $\pm$ 0.98
Brain-OF Base	12.09 $\pm$ 0.78	8.99 $\pm$ 0.10	8.87 $\pm$ 0.09	58.95 $\pm$ 2.13	76.05 $\pm$ 0.44	76.77 $\pm$ 0.60
Brain-OF Large	11.71 $\pm$ 0.67	8.86 $\pm$ 0.21	8.78 $\pm$ 0.07	62.85 $\pm$ 4.29	78.04 $\pm$ 1.59	78.32 $\pm$ 1.64
Brain-OF Huge	9.51 $\pm$ 0.51	7.87 $\pm$ 0.22	8.21 $\pm$ 0.11	61.79 $\pm$ 1.29	80.39 $\pm$ 0.59	81.10 $\pm$ 0.54

Appendix

Table 11: Comparison of multimodal fusion performance on CamCAN and LEMON. BrainHarmonix denotes an adapted architecture in which the original sMRI branch is replaced with a duplicated fMRI branch.

Hyperparameters	Brain-OF Base	Brain-OF Large	Brain-OF Huge
Model Size	47.5M	331M	1.7B
Activated Parameters	21.5M	150M	500M
Patch size	64
Max sequence length	2048
Semantic space size	128
Padding token	Full zero

Appendix

Table 12: Hyperparameters for pretraining Brain-OF.

Datasets	Global bsz	Epochs	Warmup epochs	Lr	Warmup lr	Wd	Drop out	Drop path
SEED-V	1,024	50	5	1e-4	1e-6	5.0	0.0	0.0
CHB-MIT	1,024	50	5	3e-5	3e-7	0.1	0.0	0.1
TUAB	1,024	50	5	7e-6	7e-8	0.1	0.0	0.0
ADNI	32	50	5	1.5e-4	1.5e-6	0.1	0.1	0.0
ADHD-200	32	50	5	3e-5	3e-7	0.01	0.1	0.0
CamCAN-fMRI	32	100	5	7.5e-4	7.5e-6	0.1	0.1	0.0

Appendix

Table 13: Hyperparameters for downstream finetuning (Brain-OF Base).

Datasets	Global bsz	Epochs	Warmup epochs	Lr	Warmup lr	Wd	Drop out	Drop path
SEED-V	1,024	50	5	4e-4	4e-6	5.0	0.0	0.1
CHB-MIT	1,024	50	5	3e-5	3e-7	0.5	0.0	0.1
TUAB	1,024	50	5	3e-5	3e-7	0.1	0.0	0.3
ADNI	32	50	5	1e-4	1e-6	0.1	0.0	0.3
ADHD-200	32	50	5	4e-5	4e-7	0.01	0.1	0.2
CamCAN-fMRI	32	100	5	3e-3	3e-5	0.1	0.1	0.1

Appendix

Table 14: Hyperparameters for downstream finetuning (Brain-OF Large).

Datasets	Global bsz	Epochs	Warmup epochs	Lr	Warmup lr	Wd	Drop path
SEED-V	1,024	50	5	1e-4	1e-6	5.0	0.1
CHB-MIT	1,024	50	5	1e-4	1e-6	0.1	0.1
TUAB	1,024	50	5	7e-6	7e-8	0.1	0.1
ADNI	32	50	5	3e-4	3e-6	0.1	0.1
ADHD-200	32	50	5	1e-4	1e-6	0.1	0.3
CamCAN-fMRI	32	100	5	5e-3	5e-5	0.1	0.1

Appendix

Table 15: Hyperparameters for downstream finetuning (Brain-OF Huge).

Model Variants	Total	Active	Comp.	Pretraining Cost		Inference (Batch=16)
Model Variants	Params	Params	GFLOPs	Time	# GPUs	Latency	Throughput
Brain-OF Base	47.5M	21.5M	11.29	71h 50m	32	88 ms	181.8 samples/s
Brain-OF Large	331M	150M	61.01	82h 06m	88	178 ms	89.9 samples/s
Brain-OF Huge	1.7B	500M	178.19	153h 03m	224	297 ms	53.8 samples/s

Appendix

Table 16: Efficiency analysis across model variants. Active Params denotes the number of parameters used per forward pass (due to sparse MoE). Inference Latency is reported for a batch size of 16 with sequence length 2,048 on a single A100 GPU.

Figure 6: PCA visualization of learned representations across model scales. Comparison of raw input features (Left) with representations from Brain-OF Base, Large and Huge (Right). Rows correspond to different modalities and tasks: Top: TUAB (EEG, binary classification); Middle: ADNI (fMRI, binary classification); Bottom: CamCAN (MEG, age regression). As model size increases, the representations exhibit clearer class separation (Top/Middle) and a more coherent continuous age-related trajectory for regression (Bottom), demonstrating the benefits of scaling.

Models	Training Setting	TUAB	ADNI	CamCAN
		EEG	fMRI	MEG
		BAC (%) $\uparrow$	BAC (%) $\uparrow$	MAE $\downarrow$
Brain-OF Base	Pretrained + Full FineTuning	81.88 $\pm$ 0.17	68.23 $\pm$ 2.07	8.99 $\pm$ 0.14
	Pretrained + Linear Probing	79.14 $\pm$ 0.24	65.22 $\pm$ 1.47	10.92 $\pm$ 0.22
	No Pretraining + Full FineTuning	77.59 $\pm$ 0.27	63.39 $\pm$ 2.36	9.69 $\pm$ 1.07
Brain-OF Huge	Pretrained + Full FineTuning	82.87 $\pm$ 0.18	71.79 $\pm$ 2.19	7.87 $\pm$ 0.22

Appendix

Table 17: Comparison of full finetuning, frozen-backbone linear probing, and training from scratch for Brain-OF Base and Huge.

Methods	SEED-V	TUAB	ADNI	ADHD-200	CamCAN
	EEG	EEG	fMRI	fMRI	MEG
	Kappa(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	MAE $\downarrow$
Brain-OF (Base)	23.98 $\pm$ 0.29	81.88 $\pm$ 0.17	68.23 $\pm$ 2.07	62.57 $\pm$ 1.19	8.99 $\pm$ 0.14
w/ Dense FFN	20.86 $\pm$ 0.28	81.86 $\pm$ 0.08	64.60 $\pm$ 3.32	59.10 $\pm$ 1.80	9.78 $\pm$ 0.27
w/ Vanilla Attn.	22.82 $\pm$ 0.38	81.85 $\pm$ 0.09	64.32 $\pm$ 2.82	62.82 $\pm$ 1.76	9.50 $\pm$ 0.53

Appendix

Table 18: Ablation analysis of key components.

Methods	20 dB	10 dB	5 dB	0 dB	-5 dB
DINT Attention	80.18 $\pm$ 0.33	75.27 $\pm$ 0.88	75.02 $\pm$ 0.28	73.47 $\pm$ 1.11	69.29 $\pm$ 2.34
Vanilla Attention	78.93 $\pm$ 0.40	74.20 $\pm$ 1.06	72.71 $\pm$ 1.21	70.53 $\pm$ 1.86	63.96 $\pm$ 3.47

Appendix

Table 19: SNR stress test on the TUAB dataset comparing DINT and vanilla attention.

Methods	SEED-V	TUAB	ADNI	ADHD-200	CamCAN	LEMON
	EEG	EEG	fMRI	fMRI	MEG	fMRI
	Kappa(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	BAC(%) $\uparrow$	MAE $\downarrow$	BAC(%) $\uparrow$
Full Finetuning	23.98 $\pm$ 0.29	81.88 $\pm$ 0.17	68.23 $\pm$ 2.07	62.57 $\pm$ 1.19	8.99 $\pm$ 0.14	58.95±2.13
LoRA ( $\sim$ 2% Params)	21.81 $\pm$ 0.13	81.30 $\pm$ 0.04	68.44 $\pm$ 2.13	62.10 $\pm$ 1.04	9.17 $\pm$ 0.44	59.36 $\pm$ 3.02

Appendix

Table 20: Comparison between full fine-tuning and LoRA-based parameter-efficient fine-tuning on Brain-OF (Base).

Figure 7: Label efficiency of Brain-OF under limited supervision. Brain-OF Base is evaluated across varying training data fractions (10%, 30%, 50%, 70%, 100%). Shaded regions denote standard deviation across five random seeds. Brain-OF achieves strong performance even with limited labeled data, highlighting the effectiveness of heterogeneous multimodal pretraining for low-resource settings.

Functional Magnetic Resonance Imaging

See all 19 papers

One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models
Feb 25, 2026Changli Tang, Shurui Li, Junliang Wang +8Neural Decoding Multimodal Fusion
Tsinghua University · Shanghai AI Laboratory · ShanghaiTech University
iBrain: A Unified Foundation Model Reading the Brain from Surface to Spikes
Sep 7, 2026Ying Chen, Tiou Wang, Zhifeng YueEEG Foundation Models Multimodal Pretraining
Chinese Institute for Brain Research
A Language-Guided Multimodal Foundation Model for Zero-Shot and Multi-Task Brain Signal Analysis
Sep 14, 2026Mingzhi Chen, Yiyu Gui, Guibo Luo +1Cross-Modal Alignment Multimodal Foundation Models
New Cornerstone Science Laboratory, Guangdong Provincial Key Laboratory of In-Memory Computing Chips, School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Center for Brain Inspired Intelligence, Chinese Institute for Brain Research (CIBR), Beijing, China · New Cornerstone Science Laboratory, Beijing Advanced Innovation Center for Integrated Circuits, School of Integrated Circuits, Peking University, Beijing, China

Feb 25, 2026J/K move · Enter open · S save

One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models

Changli Tang, Shurui Li, Junliang Wang, Qinfan Xiao+7

Tsinghua University · Shanghai AI Laboratory · ShanghaiTech University

Neural Decoding Multimodal Fusion EEG Decoding Multimodal Large Language Models

Open paper AskPDF arXiv

Deciphering brain function through non-invasive recordings requires synthesizing complementary high-frequency electromagnetic (EEG/MEG) and low-frequency metabolic (fMRI) signals. However, despite their shared neural origins, extreme discrepancies have traditionally confined these modalities to isolated analysis pipelines, hindering a holistic interpretation of brain activity. To bridge this fragmentation, we introduce NOBEL, a neuro-omni-modal brain-encoding large language model (LLM) that unifies these heterogeneous signals within the LLM's semantic embedding space. Our architecture integrates a unified encoder for EEG and MEG with a novel dual-path strategy for fMRI, aligning non-invasive brain signals and external sensory stimuli into a shared token space, then leverages an LLM as a universal backbone. Extensive evaluations demonstrate that NOBEL serves as a robust generalist across standard single-modal tasks. We also show that the synergistic fusion of electromagnetic and metabolic signals yields higher decoding accuracy than unimodal baselines, validating the complementary nature of multiple neural modalities. Furthermore, NOBEL exhibits strong capabilities in stimulus-aware decoding, effectively interpreting visual semantics from multi-subject fMRI data on the NSD and HAD datasets while uniquely leveraging direct stimulus inputs to verify causal links between sensory signals and neural responses. NOBEL thus takes a step towards unifying non-invasive brain decoding, demonstrating the promising potential of omni-modal brain understanding.

Abstract

Functional Magnetic Resonance Imaging

Explore similar work

One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models

iBrain: A Unified Foundation Model Reading the Brain from Surface to Spikes

A Language-Guided Multimodal Foundation Model for Zero-Shot and Multi-Task Brain Signal Analysis