Large Language Models have achieved remarkable success in language understanding and reasoning, and their multimodal extensions enable comprehension of images, video, and audio. Inspired by this, foundation models for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks. However, existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs. To address this, we propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning. Our approach employs a multi-scale FCN encoder capturing brain-region, functional subnetwork, and whole-brain features, projecting them into the semantic space of LLM. We design multi-paradigm instruction tasks covering 19 subject-specific attributes across demographics, phenotypes, and psychiatric conditions. A multi-stage learning strategy first aligns FCN embeddings with the LLM and then jointly fine-tunes the entire model to capture high-level semantic information. Experiments on a large-scale, multi-site FCN database show that FCN-LLM achieves strong zero-shot generalization on unseen datasets, outperforming conventional supervised and foundation models. This work introduces a new paradigm for integrating brain functional networks with LLMs, offering a flexible and interpretable framework for neuroscience.
Figures & tables
Figure 1: The model architecture of our proposed FCN-LLM that supports queries related to various FCN-involved tasks. It employs multi-scale encoder to extract FCN features, maps these features into the text embedding space.
Figure 2: Illustration of the proposed multi-paradigm task design for instruction tuning (left), and the multi-stage learning of FCN knowledge learning for our proposed FCN-LLM (right).
Usage
Name
Size
Available Attributes
Tuning & Internal Test
HBN ( Alexander et al., 2017 )
2254
Gender, Age, and Handness
HCP ( Van Essen et al., 2013 )
1080
Gender, Age, Handness, and Phenotype
QTIM ( Strike et al., 2023 )
1024
Gender, Age, and Handness
GSP ( Holmes et al., 2015 )
1569
Gender, Age, and Handness
ABIDE ( Di Martino et al., 2014 )
855
Gender, Age, Handness, Cognition, and Diagnosis
ADHD ( consortium, 2012 )
872
Gender, Age, Handness, Cognition, and Diagnosis
Table 1: Basic information and usage of the 10 datasets in this study.
Type
Method
Gender Classification
Disease Classification
Age Prediction
Cog Prediction
Pheno Prediction
ACC
MCC
F1
ACC
MCC
F1
MAE
PCC
MAE
PCC
MAE
PCC
Supervised
GCN
65.59
30.14
65.53
66.86
50.26
63.37
6.197
73.13
0.097
79.11
0.146
4.75
HGCN
63.50
27.97
59.62
66.97
50.73
63.89
5.892
74.91
0.095
78.73
0.144
5.56
BrainNetCNN
64.28
29.16
62.61
65.83
49.73
64.87
6.185
76.10
0.12
72.72
0.167
-2.16
BrainGNN
63.86
28.40
63.89
68.11
52.99
66.43
6.062
73.67
0.091
81.42
0.144
3.57
Transformer
69.60
38.06
69.44
68.23
52.30
65.33
5.041
80.42
0.091
79.50
0.144
0.98
Table 2: Performance of three types of methods on the internal test set.
Type
Method
ABIDEII
CNP
Disease Classification
Cog Prediction
Disease Classification
ACC
MCC
F1
MAE
PCC
ACC
MCC
F1
Supervised
Transformer
22.77
3.59
26.73
0.154
-3.00
36.74
0.16
29.78
BNT
31.82
6.19
40.16
0.158
10.66
44.44
2.88
30.26
Foundation
BrainNPT
31.56
2.21
27.59
0.147
4.32
40.71
5.23
31.43
PTGB
32.14
3.28
29.53
0.145
3.79
38.91
1.24
28.87
Table 3: Performance of three types of methods on the zero-shot test set.
Figure 3: (a) The impact of continual training on the generalization performance of FCN-LLM in Stage2; (b) The ablation study on the proposed multi-scale FCN encoder, which explores the effects of different types of FCN tokens—specifically, combinations of 116 ROI-level tokens, 7 subnetwork-level tokens, and 1 brain-level token.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Table 4: Demographic Characteristics, Diagnosis Distribution and Phenotype Sample Sizes (Original vs. Time-Window Enhanced Dataset)
Variable Name
Description
What It Measures
Higher Value Indicates
VSPLOT_TC
Visual-Spatial Processing Test, Total Correct
Visuospatial ability
Better spatial reasoning and more correct responses
ReadEng_Unadj
NIH Toolbox Oral Reading Recognition Test
Reading and word recognition ability
Stronger reading and language decoding skills
PercStress_Unadj
Perceived Stress
Subjective stress level
Higher perceived stress
AngAggr_Unadj
Anger - Aggression (from PROMIS)
Aggressive behavior when angry
Greater tendency toward aggressive responses
Strength_Unadj
NIH Toolbox Grip Strength Test
Physical strength (hand grip)
Greater muscular strength
Endurance_Unadj
2-Minute Walk Test
Cardiovascular/physical endurance
Better endurance and longer walking distance
Appendix
Table 5: Detailed Information of Phenotypic Indicators
Abbreviation
Full Name
Meaning
Description
FIQ
Full-scale IQ
Overall intellectual ability
Reflects a person’s overall intelligence level, typically derived from a weighted combination of multiple subtest scores
VIQ
Verbal IQ
Verbal intellectual ability
Measures language-related cognitive abilities, such as vocabulary, comprehension, and verbal reasoning
PIQ
Performance IQ
Performance (non-verbal) intellectual ability
Measures non-verbal cognitive abilities, such as spatial reasoning, pattern recognition, and hand-eye coordination
Appendix
Table 6: Details of Cognitive Function Indicators
Size
Stage 1
Stage 2
Batch size
32
32
Learning rate
1e-3
1e-5
Epochs
1
1
Learning schedule
Cosine decay
Cosine decay
Warm-up ratio
0.03
0.03
Weight decay
0
0
Appendix
Table 7: The hyperparameters for model training.
Stage
Dataset
Samples
Stage1
Enhanced FCN data
Predictive task
806K
Judgment task
806K
Comparative task
700K
Stage2
High-level SFT data
Predictive task
38K
Judgment task
38K
Comparative task
350K
Appendix
Table 8: The dataset details used for model training.
Figure 4: Visualization of connections among FCN tokens of different groups of diseases from the subnetwork perspective.
Comparative task
Gender Classification
Disease Classification
Age Prediction
ACC
MCC
F1
ACC
MCC
F1
MAE
PCC
×
64.23
29.03
64.89
68.72
54.69
68.85
6.631
73.28
✓
65.48
29.82
65.39
70.97
57.41
70.78
6.293
75.81
Appendix
Table 9: Ablation study on the comparative paradigm task for FCN-LLM (3B) on the internal test set, evaluated in terms of gender classification, disease classification, and age prediction.
Figure 5: Visualization of statistical evidence comprehensive.