TractoGraphVLM: A Unified Vision-Language Framework for White Matter Tractography

Gurucharan Marthi Krishna Kumar; Janine Dale Mendola; Amir Shmuel

eess.IVAug 16, 2026

TractoGraphVLM: A Unified Vision-Language Framework for White Matter Tractography

Authors: Gurucharan Marthi Krishna Kumar, Janine Dale Mendola, Amir Shmuel

Organizations: Montreal Neurological Institute, McGill University, Montreal, Canada · Department of Ophthalmology, McGill University, Montreal, Canada · McConnell Brain Imaging Centre, Montreal Neurological Institute, Departments of Neurology, Neurosurgery, Physiology and Biomedical Eng., McGill University, Montreal, QC, Canada

Abstract

Vision language models have transformed 2D medical imaging, yet extending them to 3D white matter tractography remains challenging due to the complex topology of fiber bundles. We introduce TractoGraphVLM, a unified framework for four tasks, bundle classification, text-to-tract retrieval, anatomical captioning, and visual question answering, built on a shared GPS architecture, training procedure, and read-out design. Fiber bundles are represented as streamline graphs whose nodes encode 3D position and tangent orientation. A General, Powerful, Scalable (GPS) graph transformer produces bundle embeddings aligned with a frozen BiomedBERT text encoder via contrastive learning, while a BioGPT decoder with visual prefix tokens generates captions and answers. A single shared encoder and decoder is trained jointly across all four tasks and evaluated from one checkpoint. Trained on HCP Young Adult subjects, TractoGraphVLM achieves 91.8% bundle classification accuracy, 84.7% retrieval R@1, BLEU-4=20.1, ROUGE-L=66.8, and 66.4% VQA accuracy on a held-out test set. The same checkpoints transfer zero-shot to HCP Aging subjects, with a modest drop on discriminative tasks and a larger drop on generative tasks, showing robustness to age and acquisition shift. Language supervision yields richer representations than label-only training, recovering structure like hemisphere and fiber family, carried by captions but never given as a label. Swapping only the visual encoder, graphs preserving fiber orientation outperform volumetric baselines, with GPS giving the best balance. Generative metrics measure consistency with a structured knowledge base rather than independent clinical text; even so, TractoGraphVLM shows that classifying, retrieving, describing, and answering questions about a white matter bundle can be served by one jointly trained model that learns transferable neuroanatomy from language alone.

Figures & tables

Figure 1: Streamline-to-graph construction. A fiber bundle (exemplified by the arcuate fasciculus) is transformed by sampling $P{=}32$ equidistant nodes per streamline. Each node carries a 6-D feature vector of 3D position and unit tangent orientation. See Section 2.2 for details.

Figure 2: TractoGraphVLM framework. Whole-brain tractograms are segmented into bundles (RecoBundles) and encoded as streamline graphs, which a GPS encoder maps to a visual embedding $\mathbf{v}$ (left). A symmetric InfoNCE loss $\mathcal{L}_{\mathrm{NCE}}(\mathbf{v},\mathbf{t})$ aligns $\mathbf{v}$ with a frozen BiomedBERT text embedding $\mathbf{t}$ ; the captions, queries, and questions come from templates grounded in a neuroanatomical knowledge base (bottom). Four read-outs share this embedding (right): classification by nearest prototype, retrieval by ranking bundles against a query, and captioning and VQA by feeding $\mathbf{v}$ to a BioGPT decoder as visual prefix tokens.

Classification		Retrieval		Captioning		VQA
Acc (%)	F1 (%)	R@1 (%)	R@5 (%)	BLEU-4	ROUGE-L	Acc (%)
$91.8{\scriptstyle\pm 1.4}$	$90.9{\scriptstyle\pm 1.3}$	$84.7{\scriptstyle\pm 2.8}$	$93.2{\scriptstyle\pm 2.4}$	$20.1{\scriptstyle\pm 2.3}$	$66.8{\scriptstyle\pm 2.9}$	$66.4{\scriptstyle\pm 3.0}$

Table 1: Performance of the proposed framework (GPS encoder) on the HCP-YA test set. All columns are produced by one jointly trained model evaluated from a single checkpoint. Mean

\pm

standard deviation over five random seeds.

Figure 3: Qualitative outputs of TractoGraphVLM across all four tasks (GPS encoder, HCP-YA test set). Top left : Bundle classification identifies the left inferior longitudinal fasciculus at 97.8% confidence. Top right : Text-to-tract retrieval returns the correct left ILF (cosine similarity 0.91) for a compositional query, with semantically related bundles following. Bottom left : Anatomical captioning produces a description consistent with the knowledge base, correctly noting commissural type and connectivity. Bottom right : VQA correctly answers a connectivity question at 98.5% confidence. Instance-level BLEU-4 / ROUGE-L values differ from corpus-level averages in Table 1 .

Non-graph baselines
	Classification		Retrieval		Captioning		VQA
Encoder	Acc (%)	F1 (%)	R@1 (%)	R@5 (%)	BLEU-4	ROUGE-L	Acc (%)
CNN3D [ 20 ]	$79.4{\scriptstyle\pm 2.7}$	$79.1{\scriptstyle\pm 2.8}$	$58.7{\scriptstyle\pm 3.5}$	$80.3{\scriptstyle\pm 2.7}$	$\phantom{0}7.9{\scriptstyle\pm 2.8}$	$16.1{\scriptstyle\pm 3.3}$	$21.8{\scriptstyle\pm 3.3}$
ViT [ 21 ]	$80.6{\scriptstyle\pm 2.5}$	$80.3{\scriptstyle\pm 2.6}$	$49.5{\scriptstyle\pm 3.6}$	$79.8{\scriptstyle\pm 2.8}$	$\phantom{0}6.9{\scriptstyle\pm 2.6}$	$15.2{\scriptstyle\pm 3.2}$	$20.7{\scriptstyle\pm 3.4}$
MLP	$77.8{\scriptstyle\pm 2.9}$	$77.2{\scriptstyle\pm 3.0}$	$51.4{\scriptstyle\pm 3.7}$	$74.1{\scriptstyle\pm 2.9}$	$\phantom{0}6.5{\scriptstyle\pm 2.8}$	$13.8{\scriptstyle\pm 3.1}$	$18.9{\scriptstyle\pm 3.2}$
Graph

Table 2: Systematic encoder evaluation on the HCP-YA test set. Only the visual backbone differs; all other components are identical. † marks metrics on which GPS obtains the highest score at

p<0.05

(one-sided Wilcoxon signed-rank test, paired by random seed, uncorrected). Mean

\pm

standard deviation over five seeds.

	Classification		Retrieval		Captioning		VQA
Encoder	Acc (%)	F1 (%)	R@1 (%)	R@5 (%)	BLEU-4	ROUGE-L	Acc (%)
TractCloud [ 53 ]	$91.3{\scriptstyle\pm 1.4}$	$91.0{\scriptstyle\pm 1.5}$	$84.2{\scriptstyle\pm 2.7}$	$87.4{\scriptstyle\pm 2.3}$	$\phantom{0}8.7{\scriptstyle\pm 2.6}$	$20.8{\scriptstyle\pm 3.2}$	$44.8{\scriptstyle\pm 3.2}$
FINTA [ 26 ]	$87.5{\scriptstyle\pm 2.1}$	$87.1{\scriptstyle\pm 2.2}$	$71.4{\scriptstyle\pm 3.0}$	$79.8{\scriptstyle\pm 2.6}$	$17.1{\scriptstyle\pm 2.7}$	$55.9{\scriptstyle\pm 3.1}$	$57.2{\scriptstyle\pm 3.2}$
FIESTA-AE [ 12 ]	$89.7{\scriptstyle\pm 1.7}$	$89.4{\scriptstyle\pm 1.8}$	$62.1{\scriptstyle\pm 3.2}$	$80.4{\scriptstyle\pm 2.5}$	$15.9{\scriptstyle\pm 2.8}$	$54.7{\scriptstyle\pm 3.4}$	$56.5{\scriptstyle\pm 3.1}$
PointNet [ 38 ]	$91.0{\scriptstyle\pm 1.3}$	$90.7{\scriptstyle\pm 1.4}$	$82.8{\scriptstyle\pm 2.5}$	$86.2{\scriptstyle\pm 2.1}$	$17.6{\scriptstyle\pm 2.4}$	$59.4{\scriptstyle\pm 2.8}$	$55.8{\scriptstyle\pm 2.9}$
PointNet++ [ 39 ]	$91.5{\scriptstyle\pm 1.2}$	$91.2{\scriptstyle\pm 1.3}$	$83.9{\scriptstyle\pm 2.4}$	$87.1{\scriptstyle\pm 2.2}$	$18.2{\scriptstyle\pm 2.3}$	$61.3{\scriptstyle\pm 2.7}$	$58.7{\scriptstyle\pm 2.8}$

Table 3: Comparison with existing tractography methods on the HCP-YA test set. All methods use the identical VLM framework; only the encoder differs. † marks metrics on which TractoGraphVLM obtains the highest score at

p<0.05

(one-sided Wilcoxon signed-rank test, paired by random seed, uncorrected). Mean

\pm

standard deviation over five seeds.

Evaluation set	Cls. Acc (%)	Cls. F1 (%)	R@1 (%)	VQA Acc (%)	Cap. BLEU-4	Cap. ROUGE-L
HCP-YA (in-distribution)	$91.8{\scriptstyle\pm 1.4}$	$90.9{\scriptstyle\pm 1.3}$	$84.7{\scriptstyle\pm 2.8}$	$66.4{\scriptstyle\pm 3.0}$	$20.1{\scriptstyle\pm 2.3}$	$66.8{\scriptstyle\pm 2.9}$
HCP Aging (zero-shot)	$84.3{\scriptstyle\pm 2.3}$	$84.0{\scriptstyle\pm 2.4}$	$71.8{\scriptstyle\pm 2.7}$	$50.2{\scriptstyle\pm 3.5}$	$12.9{\scriptstyle\pm 2.9}$	$57.1{\scriptstyle\pm 3.2}$
$\Delta$	$-7.5$	$-6.9$	$-12.9$	$-16.2$	$-7.2$	$-9.7$

Table 4: Cross-dataset zero-shot transfer (GPS encoder, HCP-YA

\to

HCP Aging, no adaptation). Mean

\pm

standard deviation over five seeds.

Configuration	Cls. Acc (%)	R@1 (%)	BLEU-4	VQA Acc (%)	Training (h)
$P=8$	$87.2{\scriptstyle\pm 1.6}^{{\dagger}}$	$78.9{\scriptstyle\pm 2.8}^{{\dagger}}$	$17.0{\scriptstyle\pm 2.5}^{{\dagger}}$	$64.8{\scriptstyle\pm 2.8}^{{\dagger}}$	$\phantom{0}2.8{\scriptstyle\pm 0.1}$
$P=16$	$90.1{\scriptstyle\pm 1.4}^{{\dagger}}$	$82.2{\scriptstyle\pm 2.4}^{{\dagger}}$	$19.4{\scriptstyle\pm 2.3}^{{\dagger}}$	$67.1{\scriptstyle\pm 2.7}^{{\dagger}}$	$\phantom{0}3.5{\scriptstyle\pm 0.2}$
$\mathbf{P=32}$	$\mathbf{93.1}{\scriptstyle\pm 1.2}$	$\mathbf{86.2}{\scriptstyle\pm 2.5}$	$\mathbf{21.2}{\scriptstyle\pm 2.1}$	$\mathbf{71.3}{\scriptstyle\pm 2.4}$	$\phantom{0}\mathbf{4.2}{\scriptstyle\pm 0.2}$
$P=64$	$93.3{\scriptstyle\pm 1.3}$	$86.5{\scriptstyle\pm 2.4}$	$21.4{\scriptstyle\pm 2.0}$	$71.5{\scriptstyle\pm 2.3}$	$\phantom{0}7.8{\scriptstyle\pm 0.4}$

Table 5: Node sampling density ablation ( TractoGraphVLM , HCP-YA validation set). Bold marks the selected configuration. † indicates statistically significant difference from

P=32

(

p<0.05

, Wilcoxon signed-rank test).

Features	Cls. Acc (%)	R@1 (%)	BLEU-4	ROUGE-L
Position only	$81.6{\scriptstyle\pm 1.9}$	$71.6{\scriptstyle\pm 2.7}$	$16.7{\scriptstyle\pm 2.4}$	$64.1{\scriptstyle\pm 2.7}$
Position+orientation	$\mathbf{93.1}^{\dagger}{\scriptstyle\pm 1.2}$	$\mathbf{86.2}^{\dagger}{\scriptstyle\pm 2.5}$	$\mathbf{21.2}^{\dagger}{\scriptstyle\pm 2.1}$	$\mathbf{68.5}^{\dagger}{\scriptstyle\pm 2.7}$

Table 6: Node feature ablation ( TractoGraphVLM , HCP-YA validation set). Node position alone versus position augmented with the local tangent orientation.

\dagger

: significant difference from position-only (

p<0.05

, Wilcoxon signed-rank). Mean

\pm

s.d. over five seeds.

	Discriminative		Generative
$\boldsymbol{\tau}$	Cls. Acc (%)	R@1 (%)	BLEU-4	VQA Acc (%)
$0.04$	$93.1{\scriptstyle\pm 1.7}$	$82.1{\scriptstyle\pm 2.8}^{{\dagger}}$	$20.6{\scriptstyle\pm 2.2}$	$68.1{\scriptstyle\pm 2.5}^{{\dagger}}$
$\mathbf{0.07}$	$\mathbf{93.1}{\scriptstyle\pm 1.2}$	$\mathbf{86.2}{\scriptstyle\pm 2.5}$	$\mathbf{21.2}{\scriptstyle\pm 2.1}$	$\mathbf{71.3}{\scriptstyle\pm 2.4}$
$0.12$	$90.8{\scriptstyle\pm 1.6}^{{\dagger}}$	$83.7{\scriptstyle\pm 2.5}$	$21.2{\scriptstyle\pm 2.6}$	$68.5{\scriptstyle\pm 2.7}^{{\dagger}}$

Table 7: Contrastive temperature ablation ( TractoGraphVLM , HCP-YA validation set).

\tau=0.07

attains the best or statistically equivalent score on all four tasks. † indicates statistically significant difference from

\tau=0.07

(

p<0.05

, Wilcoxon signed-rank test). Bold marks the selected configuration. Mean

\pm

S.D. over five seeds.

Bundle Identity (%)			Hemisphere (%)			Family (%)
VLM	Clf.	$\Delta$	VLM	Clf.	$\Delta$	VLM	Clf.	$\Delta$
$\mathbf{94.2}{\scriptstyle\pm 0.6}^{\dagger}$	$91.8{\scriptstyle\pm 0.9}$	$+2.4$	$\mathbf{99.4}{\scriptstyle\pm 0.2}^{\dagger}$	$96.3{\scriptstyle\pm 1.8}$	$+3.1$	$\mathbf{94.1}{\scriptstyle\pm 0.5}^{\dagger}$	$90.8{\scriptstyle\pm 2.6}$	$+3.3$

Table 8: Multi-task VLM versus supervised classifier, measured by training a small probe on the frozen TractoGraphVLM features. VLM is our full framework; Classifier is standard 78-way classification training. Hemisphere and family are never used as explicit classification targets, though they appear in the caption text used to train the VLM. † denotes a significant improvement (

p<0.05

); mean

\pm

S.D. over five seeds. The

94.2\%

here is a probe result on frozen features; the

91.8\%

in Table 1 is the full framework’s end-to-end accuracy.

Feature	Symbol	Mean $\pm$ Std	Units
Streamline count	$N$	$4{,}218\pm 3{,}107$	streamlines
Spatial extent	$E$	$148.4\pm 31.2$	mm
Normalised density	$\rho$	$0.31\pm 0.18$	(scaled)
Complexity: simple	—	24% of samples	—
Complexity: moderate	—	51% of samples	—
Complexity: complex	—	25% of samples	—

Table S1 : Geometric feature statistics over the HCP-842 atlas (78 bundles

\times

1,113 subjects).

Style	Template
Anatomical	“White matter bundle showing <ANATOMY> in the <HEMISPHERE> .”
Functional	“ <TYPE> pathway supporting <FUNCTION> , connecting <REGION_A> and <REGION_B> .”
Trajectory	“The <BUNDLE_NAME> follows <TRAJECTORY> .”
Clinical	“Damage to the <BUNDLE_NAME> is associated with <DEFICIT> .”
Composite	“ <TYPE> tract connecting <REGION_A> with <REGION_B> , involved in <FUNCTION> .”
Whole-brain	“Whole-brain tractography with <COMPLEXITY> coverage and bounding extent of <EXTENT> mm.”

Table S2 : Representative caption templates (one per style; placeholders in angle brackets, no raw counts).

Tier	Skill	Example question	Answer
1	Identification	“Which white matter tract is shown?”	bundle name
2	Localisation	“Which hemisphere does this bundle belong to?”	left / right
3	Connectivity	“What cortical regions does this tract connect?”	region pair
4	Morphology	“Is this a dorsal or ventral pathway?”	dorsal / ventral
5	Clinical	“What deficit follows disruption of this tract?”	deficit

Table S3 : VQA reasoning tiers with representative questions and answer types.

Type	Family	Representative query
1	Identity	“Locate the <BUNDLE_NAME> .”
2	Geometric	“ <COMPLEXITY> <TYPE> bundle in the <HEMISPHERE> .”
3	Connectivity	“Tract connecting <REGION_A> to <REGION_B> .”
4	Functional	“Pathway supporting <FUNCTION> .”
5	Morphological	“ <TYPE> pathway in the <HEMISPHERE> .”
6	Combined	“ <TYPE> tract linking <REGION_A> and <REGION_B> , supporting <FUNCTION> .”

Table S4 : Retrieval query types (representative, count-free forms).

Placeholder	Source
<BUNDLE_NAME> , <ANATOMY>	knowledge base (bundle identity)
<REGION_A> , <REGION_B>	knowledge base (origin / termination)
<FUNCTION> , <DEFICIT>	knowledge base (function / clinical)
<TYPE>	bundle class (projection / association / commissural)
<HEMISPHERE>	laterality (left / right / midline)
<TRAJECTORY>	knowledge base (spatial path)

Table S5 : Template placeholders and their sources.

Bundle	Connectivity	Function	Type
Commissural Fibers
CC	L $\leftrightarrow$ R hemispheres, mid-sagittal	Interhemispheric communication and integration	Comm.
MCP	Pons $\leftrightarrow$ cerebellar cortex (bilateral)	Motor coordination and error correction	Comm.
Projection Fibers
CST L	Motor cortex $\to$ spinal cord (L)	Voluntary movement control, right body	Proj.
CST R	Motor cortex $\to$ spinal cord (R)	Voluntary movement control, left body	Proj.

Table S6 : Anatomical knowledge base (representative entries). Comm. = commissural; Proj. = projection; Assoc. = association.

Field	Value
full_name	Arcuate Fasciculus (left)
bundle_type	association
hemisphere	left
origin_regions	inferior frontal gyrus, precentral gyrus
termination_regions	posterior superior temporal gyrus
functions	phonological processing, language production

Table S7 : Example knowledge-base record (left arcuate fasciculus).

System	Cap. BLEU-4	Cap. ROUGE-L	VQA Acc. (%)
Classifier + KB lookup (predicted label)	$14.8\pm 2.1$	$52.3\pm 2.8$	$48.7\pm 3.1$
Classifier + KB lookup (oracle label)	$18.2\pm 1.9$	$61.7\pm 2.4$	$62.1\pm 2.7$
TractoGraphVLM (unified)	$\mathbf{20.1\pm 2.3}$	$\mathbf{66.8\pm 2.9}$	$\mathbf{66.4\pm 3.0}$

Table S8 : Classifier-plus-lookup control on the HCP-YA test set. The predicted-label row uses the classifier’s output to look up the knowledge-base entry; the oracle-label row uses the ground-truth label. Mean

\pm

standard deviation over five random seeds.

Component	Params	Role
Visual encoder	0.073–10.842M *	Architecture-specific (Table S10 )
Text encoder (BiomedBERT)	110M	Contrastive text embedding; frozen
Language decoder (BioGPT)	347M	Autoregressive caption and VQA generation

Table S9 : System component parameter counts.

Family	Encoder	Layers	Hidden	Params (M)	Key Choices
Graph	GCN [ 24 ]	8 GCNConv	384	1.585	Isotropic aggregation, residual, BatchNorm
	GAT [ 48 ]	8 GATConv	4 heads $\times$ 96	1.592	Anisotropic, learned attention weights
	GraphSAGE [ 19 ]	8 SAGEConv	384	2.765	Neighbourhood sampling, mean aggregator
	GIN [ 52 ]	8 GINConv	384	6.028	Sum aggregation, injective MLP update
	GPS [ 41 ]	4 GPSConv	192	1.908	Laplacian PE, 8-head attn, virtual node
Point Cloud	PointNet [ 38 ]	3 MLP	64/128/256	1.360	Max-pool global aggregation, T-Net

Table S10 : Visual encoder specifications (16 architectures). All project to a shared 256-D latent space.

Query Type	R@1 (%)	$\Delta$ from overall
1. Identity	$92.1{\scriptstyle\pm 2.0}$	$+7.4$
2. Geometric	$86.3{\scriptstyle\pm 2.4}$	$+1.6$
3. Anatomical	$87.9{\scriptstyle\pm 2.5}$	$+3.2$
4. Functional	$80.2{\scriptstyle\pm 3.3}$	$-4.5$
5. Morphological	$83.5{\scriptstyle\pm 2.8}$	$-1.2$
6. Combined	$77.4{\scriptstyle\pm 3.6}$	$-7.3$

Table S11 : Retrieval R@1 by query type ( TractoGraphVLM , HCP-YA test). Types 1–3 are structurally similar to caption templates; Types 4–6 are more compositional. Mean

\pm

standard deviation over five seeds.

Tier	Skill	HCP-YA (%)	HCP Aging (%)	$\Delta$
1	Identification	$89.2{\scriptstyle\pm 2.1}$	$82.4{\scriptstyle\pm 2.8}$	$-6.8$
2	Localisation	$87.1{\scriptstyle\pm 2.4}$	$78.6{\scriptstyle\pm 3.1}$	$-8.5$
3	Connectivity	$65.3{\scriptstyle\pm 3.2}$	$39.8{\scriptstyle\pm 4.0}$	$-25.5$
4	Morphology	$52.8{\scriptstyle\pm 3.5}$	$29.4{\scriptstyle\pm 3.9}$	$-23.4$
5	Clinical	$37.6{\scriptstyle\pm 3.8}$	$20.8{\scriptstyle\pm 4.2}$	$-16.8$
Overall		$\mathbf{66.4}$	$\mathbf{50.2}$	$-16.2$

Table S12 : Per-tier VQA accuracy , in-distribution (HCP-YA) versus zero-shot transfer (HCP Aging).

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 458 papers

Tractogram foundation model
Jun 3, 2026Guikun Chen, Yuqian Chen, Yijie Li +5Medical Imaging Foundation Models Representation Learning
The State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou 310058, Zhejiang, China · Department of Radiology, Brigham and Women’s Hospital, Mass General Brigham, Boston 02115, Massachusetts, USA · Harvard Medical School, Boston 02115, Massachusetts, USA +4
FiberGeoText: A Vision-Language Model for Population- Level Organization of Superficial White Matter
Oct 2, 2026Yuqian Chen, R. Jarrett Rushmore, Guikun Chen +5Neuroimaging Medical Image Representation Learning
Department of Radiology, Mass General Brigham, Boston MA, USA · Harvard Medical School, Boston MA, USA · Department of Anatomy and Neurobiology, Boston University Chobanian & Avedisian School of Medicine, Boston MA, USA +2
Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging
Aug 13, 2026Zhi Qiao, Xintong Wu, Yichu He +1Medical Imaging Foundation Models 3D Medical Imaging
United Imaging Intelligence, Shanghai, China

Jun 3, 2026J/K move · Enter open · S save

Tractogram foundation model

Guikun Chen, Yuqian Chen, Yijie Li, Yogesh Rathi+4

The State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou 310058, Zhejiang, China · Department of Radiology, Brigham and Women’s Hospital, Mass General Brigham, Boston 02115, Massachusetts, USA · Harvard Medical School, Boston 02115, Massachusetts, USA+4

Medical Imaging Foundation Models Representation Learning Diffusion MRI Neuroimaging

Open paper AskPDF arXiv

Diffusion MRI (dMRI) tractography is the only noninvasive approach for mapping white-matter pathways in the living human brain. It represents each brain as a tractogram: a large, unordered set of three-dimensional streamlines that includes information about both local streamline geometry and whole-brain anatomical organization. This structure makes tractograms a natural but challenging target for representation learning. Existing methods treat streamline classification and subject-level prediction as separate problems: streamline classifiers focus on geometric patterns, whereas subject-level prediction often depends on hand-crafted features. As a result, current methods do not learn reusable representations that connect streamline anatomy with whole-brain inter-subject variation. Here we introduce TractFM, a tractogram foundation model that learns reusable representations directly from whole-brain streamline sets. TractFM combines a local streamline encoder with a permutation-equivariant tractogram encoder, allowing all streamlines from a subject to be contextualized jointly in a single forward pass. Pretraining on dense anatomical tract parcellation, i.e., assigning anatomical labels to individual streamlines, yields two complementary representations: contextualized streamline-level embeddings for tract parcellation and compact subject-level descriptors for downstream prediction of subject phenotypes. Across three tractography algorithms and five dMRI datasets, TractFM transfers to both streamline-level and subject-level tasks. Its frozen representations achieve accurate tract parcellation and predict age and sex across independent datasets. These results show that whole-brain geometric context, learned once, can generalize across tractography pipelines, datasets, and prediction tasks.

Abstract

Medical VLMs

Explore similar work

Tractogram foundation model

FiberGeoText: A Vision-Language Model for Population- Level Organization of Superficial White Matter

Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging