fMRI-TAMCL: Text-Anchored Supervised Multimodal Contrastive Learning for fMRI-Based Brain Disorder Classification

Juliana Mantebea Danso; Enoch Opanin Gyamfi; Mylene C. Q. Farias

cs.CVOct 5, 2026

fMRI-TAMCL: Text-Anchored Supervised Multimodal Contrastive Learning for fMRI-Based Brain Disorder Classification

Authors: Juliana Mantebea Danso, Enoch Opanin Gyamfi, Mylene C. Q. Farias

Organizations: Department of Computer Science, Texas State University, San Marcos, TX 78666 USA · University of Technology and Applied Sciences, Navrongo, Ghana

Abstract

Resting-state fMRI is important in the classification of brain disorders, but highly multimodal and exhibits strong multisite heterogeneity. Existing methods fuse images, BOLD-based functional connectivity, and phenotypic data modalities. Unlike other medical imaging datasets, rs-fMRI datasets rarely include a text modality, so they are generated from phenotypic data or BOLD activations. These text generation methods rely on fixed assumptions for subjects, sites, devices, and protocols, leading to poor generalization across datasets. We propose fMRI-TAMCL, a text-anchored multimodal contrastive learning framework that integrates fMRI images, sparse FC, and generated subject-specific text. Its Subject-Adaptive Threshold Derivation module generates BOLD activation text, while Feature-Value Serialization module generates phenotypic text. All three modalities are encoded as clustered graphs, projected onto a shared unit hypersphere space, aligned using pairwise, text-anchored supervised contrastive learning, and fused with attention. fMRI-TAMCL proves its generalization capability across five datasets outperforming 29 baselines with 78.6%-86.4% accuracy in downstream classification.

Figures & tables

Fig. 1 : Overview of the proposed fMRI-TAMCL framework. Multimodal fMRI image, FC, and text features are encoded, projected into a shared contrastive latent space, fused through attention, and classified for diagnosis

Fig. 2 : Subject-Adaptive Ternary Discretization (SATD) method for thresholding ROI-level fMRI timeseries data

Unimodal: (i) Functional Connectivity (FC-Only) Baselines
Method	Modalities	ABIDE		ADHD-200		ADNI		HCP-EP		REST-meta-MDD
Method	Modalities	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1
SVM	$\times~\|~\checkmark~\|~\times$	$63.2\pm 2.1$	$62.8\pm 2.3$	$58.6\pm 2.4$	$57.9\pm 2.6$	$65.4\pm 1.9$	$64.7\pm 2.0$	$70.1\pm 1.8$	$69.5\pm 1.9$	$56.3\pm 2.5$	$55.7\pm 2.7$
GCN [ 10 ]	$\times~\|~\checkmark~\|~\times$	$68.4\pm 1.7$	$67.9\pm 1.8$	$63.1\pm 1.9$	$62.4\pm 2.1$	$70.2\pm 1.6$	$69.6\pm 1.7$	$72.8\pm 1.5$	$74.3\pm 1.6$	$61.4\pm 2.0$	$60.8\pm 2.2$
GAT [ 11 ]	$\times~\|~\checkmark~\|~\times$	$69.7\pm 1.5$	$69.2\pm 1.7$	$64.3\pm 1.8$	$63.7\pm 1.9$	$71.8\pm 1.5$	$71.2\pm 1.6$	$75.0\pm 1.4$	$75.5\pm 1.5$	$62.7\pm 1.9$	$62.1\pm 2.0$
GraphSAGE [ 12 ]	$\times~\|~\checkmark~\|~\times$	$68.9\pm 1.6$	$68.4\pm 1.7$	$63.8\pm 1.8$	$63.2\pm 2.0$	$71.1\pm 1.6$	$70.5\pm 1.7$	$73.4\pm 1.4$	$74.9\pm 1.5$	$62.0\pm 2.0$	$61.4\pm 2.1$

TABLE I : Diagnostic performance on five datasets, reported as mean

\pm

STD percentages over five runs. Best results are bolded. Modalities: I = f MRI image, FC = functional connectivity, Txt = text,

\checkmark

= used,

\times

= not used.

Panel of Figure 4 — Fig. 3 : Ablation study on five datasets, where each variant removes one architectural component from the full fMRI-TAMCL model.

Unimodal
Method	ACC (%)	SEN (%)	SPE (%)	AUC-ROC	F1 (%)
BrainNetCNN [ 22 ]	62.3 $\pm$ 4.2	60.8 $\pm$ 5.1	63.8 $\pm$ 4.7	0.668 $\pm$ 0.045	62.0 $\pm$ 4.3
BrainGNN [ 1 ]	64.7 $\pm$ 3.9	63.2 $\pm$ 4.6	66.3 $\pm$ 4.3	0.693 $\pm$ 0.042	64.5 $\pm$ 3.9
BERT-Text	66.4 $\pm$ 2.9	64.9 $\pm$ 3.8	67.1 $\pm$ 3.7	0.718 $\pm$ 0.031	65.7 $\pm$ 3.1
BrainNetTF	67.9 $\pm$ 2.9	66.8 $\pm$ 2.2	69.1 $\pm$ 2.8	0.731 $\pm$ 0.021	67.9 $\pm$ 2.3
EC-GCL	68.2 $\pm$ 3.6	67.0 $\pm$ 4.2	69.5 $\pm$ 3.9	0.736 $\pm$ 0.038	68.0 $\pm$ 3.6

TABLE II : LOSOCV results on ABIDE. Mean

\pm

STD across 17 held-out sites. Best results are in bold.

Method	Miss Image ACC (%)	Miss FC ACC (%)	Miss Complete Text ACC (%)	Only Phenotypic/ Demographic Text ACC (%)	All Modalities ACC (%)	Avg Drop
TUMSyn	71.56	N/A	N/A	74.6	78.9	5.8
BrainPrompt+	N/A	N/A	N/A	73.8	78.1	4.3
PMIL	N/A	74.4	74.7	N/A	81.3	6.8
fTSPL [ 26 ]	75.6	73.1	73.9	N/A	80.8	6.6
RTGMFF [ 27 ]	77.2	N/A	75.7	76.3	81.5	5.1
fMRI-TAMCL	79.5	77.8	78.2	81.8	83.9	4.6

TABLE III : Missing-modality sensitivity on ABIDE. Avg Drop from

ACC_{\mathrm{stable}}=\bar{x}_{\mathrm{ACC}}-\sigma_{\mathrm{ACC}}

. Best results are in bold.

Contrastive loss weight sensitivity $(\lambda_{\mathrm{pair}},\lambda_{\mathrm{text}},\lambda_{\mathrm{sup}})$
Hyperparameter	ACC (%)	F1
$(0.5,0.5,0.5)$	$83.1\pm 0.9$	$83.0\pm 1.0$
$(1.0,0.5,0.5)$	$84.7\pm 0.8$	$83.2\pm 0.7$
$(0.5,0.5,1.0)$	$83.9\pm 0.9$	$83.8\pm 0.9$
Number of Graph Clusters $(N^{(I)},N^{(L)},N^{(T)})$
$(32,64,128)$	$84.7\pm 0.8$	$83.2\pm 0.7$

TABLE IV : Hyperparameter sensitivity on ABIDE. Bolded rows are optimal settings used in all experiments.

SATD Outcome	Count ( $N$ )	Proportion (%)
Full convergence (2 crossing points)	1,061	95.4
Partial (1 crossing point; $\tau_{2}$ from mean scaling)	34	3.1
Fallback (percentile-based thresholds)	17	1.5

TABLE V : GGMM convergence across all ABIDE subjects (

N=1{,}112

Multimodal Method	Parameters (M)	Train Time/ Epoch (s)	Inference Time/ Subject (ms)	$ACC_{\mathrm{stable}}$ (%)
TUMSyn	15.1	78	15.1	78.9
BrainPrompt+	17.8	62	13.2	78.1
PMIL	18.3	67	21.3	81.3
fTSPL [ 26 ]	18.7	74	19.9	80.8
RTGMFF [ 27 ]	20.7	70	17.9	81.5
fMRI-TAMCL	21.2	81	22.6	83.9

TABLE VI : Comparison of computational efficiency (training and inference times) and conservative stability-adjusted accuracy

ACC_{\mathrm{stable}}

Panel of Figure 10 — Fig. 4 : Two-dimensional latent space visualization on ABIDE for different model variants. The projections illustrate the separation between ASD and typically developing control (TC) subjects.

Panel of Figure 11 — Fig. 5 : Condition-specific brain saliency maps visualized using BrainNet Viewer on the BrainMesh_ICBM152_smoothed surface. The maps show the most salient ROIs identified for ASD, ADHD, AD, MDD, and EP diagnostic groups.

Panel of Figure 12 — Fig. 6 : Cross-dataset biomarker interpretability visualization. Condition–key region mapping (left). ROI saliency in MNI-152 stereotactic space (right).

Explore similar work

MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data
Jun 15, 2026Muxuan Liu, Ichiro Kobayashi, Satoshi NishidaBrain Alignment Limited Data
Graduate School of Humanities and Sciences Ochanomizu University · Center for Information and Neural Networks Advanced ICT Research Institute National Institute of Information and Communications Technology
FM-fMRI: Event Conditioned Flow Matching for Rest-to-Task fMRI Time-Series Synthesis
May 26, 2026Peiyu Duan, Jiyao Wang, Nicha C. Dvornek +4Functional Magnetic Resonance Imaging Time-Series Generation
Department of Biomedical Engineering, Yale University, New Haven, CT, USA · Department of Radiology & Biomedical Imaging, Yale University, New Haven, CT, USA · Department of Electrical Engineering, Yale University, New Haven, CT, USA
Learning Robust and Task-Invariant Functional Representation from fMRI through Siamese Self-Supervised Learning
May 27, 2026Jiyao Wang, Peiyu Duan, Nicha C. Dvornek +4Multimodal Neuroimaging Functional Magnetic Resonance Imaging
Department of Biomedical Engineering, Yale University, New Haven, CT, USA · Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA · Electrical Engineering, Yale University, New Haven, CT, USA +1

Jun 15, 2026, cs.CLJ/K move · Enter open · S save

MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data

Muxuan Liu, Ichiro Kobayashi, Satoshi Nishida

Graduate School of Humanities and Sciences Ochanomizu University · Center for Information and Neural Networks Advanced ICT Research Institute National Institute of Information and Communications Technology

Brain Alignment Limited Data

Open paper AskPDF arXiv

Decoding inner speech from non-invasive brain signals remains a fundamental challenge due to the absence of overt linguistic output, limited training data, and large inter-subject variability. Existing brain-to-text approaches often rely on task-specific decoder fine-tuning, which restricts scalability and complicates adaptation to new participants. We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals without modifying the underlying language model. The first stage learns a subject-specific neural-semantic alignment that maps fMRI activity into a shared multimodal semantic space, extracting a latent semantic sketch of the internally generated sentence. The second stage integrates this sketch with visual context to prompt a frozen multimodal language model for free-form generation. Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines. We further show that the learned semantic-to-language projection can generalize across subjects, enabling effective decoding when paired with subject-specific neural alignment. These results indicate that neural signals modulate semantic content beyond image-driven priors, supporting a scalable and modular direction for brain-to-text decoding.

Abstract

Explore similar work

MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data

FM-fMRI: Event Conditioned Flow Matching for Rest-to-Task fMRI Time-Series Synthesis

Learning Robust and Task-Invariant Functional Representation from fMRI through Siamese Self-Supervised Learning