AG-JEPA: Anatomically Guided Joint Embedding Predictive Architecture for Neurodegenerative Representation Learning
Organizations: Microsoft Research India
Abstract
Brain MRI can reveal early signs of Alzheimers disease (AD) before measurable cognitive decline becomes apparent. However, during the transition from cognitively normal (CN) to mild cognitive impairment (MCI), the disease-related signals are often subtle and confined to only a few brain regions. Existing Joint Embedding Predictive Architecture (JEPA) models learn general structural representations but do not explicitly capture the anatomical regions affected in AD. We propose Anatomically Guided JEPA (AG-JEPA), which introduces anatomical tokens that interact with patch tokens during pretraining to learn region-specific representations from brain regions relevant to neurodegeneration. Building on this representation space, we further propose Prog-AG-JEPA, which models disease progression trajectory as a conditioned latent displacement within this space. We evaluate AG-JEPA across five downstream tasks spanning Alzheimer's disease diagnosis and progression prediction. AG-JEPA improves diagnosis and MCI prognosis, achieving an 8.4 percentage point AUROC gain over the strongest baseline on CN/MCI classification and a 7.1 percentage point gain on out-of-distribution CN/AD classification on the MR-RATE dataset, highlighting its effectiveness across both early-stage detection and cross-cohort generalization. For disease progression modeling, Prog-AG-JEPA consistently outperforms all baselines on future diagnosis and brain volume change prediction, demonstrating the value of anatomically grounded neurodegeneration representations for diagnosis and prognosis.
Figures & tables
| Diagnosis Classification (AUROC) | C-Index / MAE | Cognitive Scores (MAE) | ||||||
|---|---|---|---|---|---|---|---|---|
| Method | CN/MCI | CN/AD | sMCI/pMCI | TTC | Age | MMSE | ADAS-13 | CDR-SB |
| NeuroVFM | ||||||||
| Neuro-JEPA | ||||||||
| AG-JEPA | ||||||||
| Diagnosis (AUROC) | Cognitive Score Change (MAE) | Volume Change (MAE) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | CN/AD | CN/MCI | MMSE | ADAS-13 | CDR-SB | Hipp. | Amyg. | Temp. Horn | Lat. Vent. |
| Latent Drift | |||||||||
| BrLP | |||||||||
| AG-LDM | |||||||||
| Prog-AG-JEPA | |||||||||
| Objective terms | AUROC | ||||||
|---|---|---|---|---|---|---|---|
| CN/ MCI | CN/ AD | sMCI/ pMCI | |||||
| Dataset | Task | 5 Regions | 18 Regions |
|---|---|---|---|
| ADNI | CN/AD | 0.887 | 0.889 |
| ADNI | CN/MCI | 0.804 | 0.788 |
| AIBL | CN/AD | 0.905 | 0.900 |
| AIBL | CN/MCI | 0.705 | 0.690 |
| ADNI | AIBL | |||
| Tokens | CN/AD | CN/MCI | CN/AD | CN/MCI |
| All anatomical + patch tokens | 0.887 | 0.804 | 0.905 | 0.705 |
| All five anatomical tokens | 0.885 | 0.801 | 0.908 | 0.703 |
| Single token | ||||
| Hippocampus | 0.882 | 0.802 | 0.903 | 0.701 |
| Temporal horn | 0.880 | 0.768 | 0.875 | 0.698 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Patch | Region voxels | Fraction of patch | |
|---|---|---|---|
| Component | Predicts | From | Enforces | Architectural role |
|---|---|---|---|---|
| image fidelity | Retains full-resolution patch detail and prevents collapse to regional summaries. | |||
| token content | Uses five tokens to predict masked patches, enforcing subject-specific content. | |||
| stream consistency | Predicts full-volume tokens from visible patches, closing the bidirectional stream loop. | |||
| region identity | Shares one projector across regions, placing identity in the tokens rather than per-region heads. |
| Use / endpoint | Cohort / split | MRI scans | Pairs | Subjects | MRI scan count diagnosis-wise |
| AG-JEPA pretraining; CN/MCI–CN/AD classification; age/MMSE/ADAS-13/CDR-SB prediction | |||||
| Pretraining + probe fitting Within-cohort evaluation | ADNI train ADNI Test | 2,523 629 | – – | 540 135 | CN 737; MCI 1,027; AD 759 CN 192; MCI 271; AD 166 |
| Evaluation | MIRIAD | 708 | – | 69 | CN 243; MCI 0; AD 465 |
| Evaluation | MR-RATE | 220 | – | 220 | CN 165; MCI 0; AD 55 |
| Evaluation | AIBL | 1,023 | – | 527 | CN 757; MCI 128; AD 138 |
| AG-JEPA -based sMCI/pMCI classification and time-to-conversion prediction | |||||
| Grade | Cues |
|---|---|
| Exclusion | Clause-local negation: no , not , without , negative for , excludes , ruled out , unlikely , and does not suggest/support/favor ; likelihood … reduced/low ; largely excluded ; ERICA score 0–1. |
| Assertion | Significant for , in terms of , regarding , consistent with , compatible with , supports , favors , suggests , indicative of , typical of , diagnostic of , primarily considered , and likelihood … high ; ERICA score 2–3. |
| Hedged assertion | A qualifier directly modifying an assertion cue, such as may be significant , potentially compatible , or possibly consistent ; alternatively, the diagnosis is expressed as a question, such as Alzheimer’s? . |
| Hedge | Suspicious , suspected , possible , probable , may represent , might , could be , cannot be excluded , differential , and versus . |
| Parameter | Variable | Value | Parameter | Variable | Value |
|---|---|---|---|---|---|
| Image and masking setup | |||||
| Input ROI | Patch size | ||||
| Patch tokens | Context patch count | ||||
| Target patch count | Mask blocks | ||||
| Target-mask ratio | Mask aspect ratio | ||||
| Mask spatial scale | Mask depth scale | ||||
| Parameter | Variable | Value | Parameter | Variable | Value |
|---|---|---|---|---|---|
| Input representation and conditioning | |||||
| Encoder | AG-JEPA ( anatomy5 ) | Encoder status | – | frozen | |
| Patch tokens | Anatomy tokens | ||||
| Conditioning variables | , age, regional volumes | Conditioning dimension | |||
| Conditioning injection | – | learned prefix | Prefix tokens | ||
| Architecture | |||||
| Model | Native MAE | Ridge MAE |
|---|---|---|
| Prog-AG-JEPA | N/A | 0.06503 |
| BrLP | 0.07184 | 0.07606 |
| Latent Drift | 0.08539 | 0.09297 |
| AG-LDM | 0.08182 | 0.08162 |
| Dataset | Task | Spatial-control JEPA | AG-JEPA |
|---|---|---|---|
| ADNI | CN/AD | 0.875 | 0.887 |
| ADNI | CN/MCI | 0.742 | 0.804 |
| AIBL | CN/AD | 0.856 | 0.905 |
| AIBL | CN/MCI | 0.631 | 0.705 |
| Anatomical tokens | Predictor conditioning | AUROC | Cognitive Score Change (MAE) | Volume (MAE) | |||
|---|---|---|---|---|---|---|---|
| CN/AD | CN/MCI | MMSE | ADAS-13 | CDR-SB | %TBV | ||
| Yes | |||||||
| + age | |||||||
| + age + vol. | |||||||
| No | + age + vol. | ||||||
| Readout | Hipp. | Amyg. | Ent. | Temp. horn | Lat. vent. | Mean, five regions |
|---|---|---|---|---|---|---|
| Full | 0.018 | 0.010 | 0.015 | 0.024 | 0.185 | 0.0505 |
| Hippocampus | 0.018 | 0.010 | 0.015 | 0.025 | 0.194 | 0.0524 |
| Amygdala | 0.017 | 0.010 | 0.015 | 0.025 | 0.200 | 0.0535 |
| Entorhinal cortex | 0.018 | 0.010 | 0.015 | 0.025 | 0.192 | 0.0520 |
| Temporal horn | 0.018 | 0.010 | 0.015 | 0.025 | 0.200 | 0.0536 |
| Lateral ventricle | 0.018 | 0.010 | 0.014 | 0.025 | 0.200 | 0.0534 |
| Diagnosis (AUROC) | Cognitive change (MAE) | Volume change (MAE) | ||
|---|---|---|---|---|
| Model | CN/AD | CN/MCI | Mean | Mean |
| Covariate-only | 0.5864 | 0.5952 | 2.4982 | 0.07255 |
| Current-state | 0.8291 | 0.7521 | 2.4160 | 0.07181 |
| Prog-AG-JEPA | 0.8435 | 0.7648 | 2.3500 | 0.05939 |
| Method | AUROC | 95% bootstrap CI |
|---|---|---|
| NeuroVFM | [0.734, 0.839] | |
| Neuro-JEPA | [0.851, 0.927] | |
| AG-JEPA | [0.876, 0.947] |
| Dataset | Task | Neuro-JEPA | AG-JEPA | [95% CI] | |
|---|---|---|---|---|---|
| ADNI | CN/MCI | 0.720 | 0.804 | [ , ] | 0.0002 |
| ADNI | CN/AD | 0.876 | 0.887 | [ , ] | 0.0001 |
| ADNI | sMCI/pMCI | 0.628 | 0.689 | [ , ] | 0.0257 |
| MR-RATE | CN/AD | 0.875 | 0.946 | [ , ] | |
| MIRIAD | CN/AD | 0.966 | 0.981 | [ , ] | |
| AIBL | CN/MCI | 0.661 | 0.705 | [ , ] | 0.0006 |
| Task | BrLP | Prog-AG-JEPA | [95% CI] | |
|---|---|---|---|---|
| CN/AD (AUROC) | 0.870 | 0.886 | [ , ] | 0.036 |
| CN/MCI (AUROC) | 0.745 | 0.763 | [ , ] | 0.047 |
| Hippocampus (MAE) | 0.023 | 0.019 | [ , ] | 0.0005 |
| Amygdala (MAE) | 0.015 | 0.012 | [ , ] | 0.026 |
| Temporal horn (MAE) | 0.029 | 0.023 | [ , ] | 0.0005 |
| Lateral ventricle (MAE) | 0.220 | 0.205 | [ , ] | 0.178 |
Medical Image Representation Learning
Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.
See all 281 papers