BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction
Organizations: The University of Sydney, Australia · University of New South Wales, Australia · CSIRO’s Data61
Abstract
Naturalistic fMRI offers a time-resolved window into cortical dynamics during continuous multimodal experience, where future brain states are jointly governed by endogenous neural context and exogenous sensory drive. Forecasting such dynamics, however, remains challenging because fast-changing stimulus streams must be reconciled with the sluggish hemodynamic BOLD response, while cortical activity is organized across functionally heterogeneous brain networks. To address these challenges, we introduce BrainVista, a multimodal autoregressive framework for observed-stimulus conditional brain-state forecasting. BrainVista predicts future fMRI states from the history of brain activity and stimulus tokens temporally aligned to each prediction query, enabling autoregressive prediction without access to future ground-truth fMRI states. Recursive rollout provides a within-run predictive test of brain-state evolution under continuing sensory input. Specifically, we combine Network-wise Tokenizers for functionally structured brain representations, a Spatial Mixer Head for cross-network interaction refinement, and Stimulus-to-Brain (S2B) masking to preserve temporally causal stimulus-brain conditioning while preventing future brain-state leakage and stimulus look-ahead. We evaluate BrainVista on Algonauts 2025, CineBrain, and HAD, three large-scale naturalistic fMRI benchmarks, under long-horizon conditional autoregressive forecasting. BrainVista consistently outperforms baselines, improving pattern correlation by 10.8% and 9.4% relative to the strongest baseline on Algonauts 2025 and CineBrain, respectively. Code is available at https://github.com/xuanhuayin/BrainVista.
Figures & tables
| Algonauts 2025 | CineBrain | HAD | |||||||
| Model | Cls@Top10 | Cls@Top10 | Cls@Top10 | ||||||
| Non-ARX references | |||||||||
| TRIBE [ 35 ] | 0.1379 | 0.1323 | 0.6823 | 0.4742 | 0.3012 | 0.8560 | 0.2685 | 0.1845 | 0.7012 |
| MindSimulator [ 33 ] | 0.1432 | 0.1478 | 0.6919 | 0.4856 | 0.3235 | 0.8615 | 0.2610 | 0.1960 | 0.7105 |
| BrainLM [ 6 ] | 0.1191 | 0.0829 | 0.6493 | 0.4633 | 0.2762 | 0.8478 | 0.2465 | 0.1450 | 0.6912 |
| BrainJEPA [ 7 ] | 0.1493 | 0.0916 | 0.6812 | 0.4941 | 0.2445 | 0.8455 | 0.2515 | 0.2353 | 0.7321 |
| Model | ||||||
|---|---|---|---|---|---|---|
| Ridge-ARX | 0.5740 | 0.5530 | 0.2860 | 0.2675 | 0.2630 | 0.2327 |
| LSTM-ARX | 0.5815 | 0.5740 | 0.3002 | 0.2940 | 0.2642 | 0.2605 |
| TemporalCNN-ARX | 0.5869 | 0.5831 | 0.3065 | 0.3072 | 0.2654 | 0.2741 |
| Transformer-ARX | 0.5890 | 0.5876 | 0.3112 | 0.3160 | 0.2678 | 0.2724 |
| BrainHarmony-AR | 0.5912 | 0.5925 | 0.3136 | 0.3215 | 0.2685 | 0.2769 |
| Variant | Algonauts 2025 | CineBrain | HAD |
|---|---|---|---|
| Stimulus only | 0.1391 | 0.3136 | 0.1908 |
| Past fMRI only | 0.1764 | 0.4872 | 0.2761 |
| Past fMRI + stimulus | 0.2090 | 0.5643 | 0.2950 |
| Temporal mask | No mixer | Attention mixer |
|---|---|---|
| Vanilla causal | 0.1573 | 0.1420 |
| S2B | 0.1812 | 0.2090 |
Appendix figures & tables58 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Visual | Audio | Text | Active input dim. | Projection output |
|---|---|---|---|---|---|
| Algonauts 2025 | Yes | Yes | Yes | ||
| CineBrain | Yes | Yes | Yes | ||
| HAD | Yes | No | Yes |
| Dataset | #Subjects | Native TR | Raw brain representation | Model-space representation | Split and rollout protocol |
|---|---|---|---|---|---|
| Algonauts 2025 | 4 | 1.49s | 1,000 Schaefer parcels in MNI space | 1,000 parcels mapped to Yeo-7 systems; | Official train/validation/test split; rollout within run; context reset at run boundaries. |
| CineBrain | 6 | 0.8s | HCP-MMP ROI voxels: 8,405 visual voxels and 10,541 auditory voxels | Training-set PCA to 512 visual + 512 auditory components; | Official train/validation/test split; audiovisual stimulus and fMRI aligned within run or episode. |
| HAD | 30 | 2.0s | Surface-based fMRI responses from participants viewing human-action clips | fsLR-to-Schaefer-1000 aggregation mapped to Yeo-7 systems; | Official train/validation/test split; rollout within run; context reset at run boundaries. |
| Dataset | TR | fMRI boundary | Visual feature window | Audio feature window | Text feature source and boundary |
|---|---|---|---|---|---|
| Algonauts 2025 | 1.49s | Bin end | V-JEPA 2 clips sampled over 2.0s windows; window end | Wav2Vec2-BERT windows of 1.0s with 0.5s stride; window end | Dataset-provided subtitles or aligned transcripts; only text tokens ending before are used. |
| CineBrain | 0.8s | Bin end | V-JEPA 2 clips sampled over 2.0s windows; window end | Wav2Vec2-BERT windows of 1.0s with 0.4s stride; window end | Aligned movie transcript/subtitles; sentence embeddings use causal prefixes only. |
| HAD | 2.0s | Bin end | V-JEPA 2 clips sampled over 2.0s windows; window end | Not used | Clip-level action descriptions and available annotations are aligned causally to clip boundaries. |
| Dataset | s | s | s | s | s | s |
|---|---|---|---|---|---|---|
| Algonauts 2025 | 0.185 | 0.193 | 0.199 | 0.205 | 0.209 | 0.213 |
| CineBrain | 0.523 | 0.536 | 0.548 | 0.558 | 0.564 | 0.568 |
| HAD | 0.275 | 0.281 | 0.288 | 0.292 | 0.295 | 0.296 |
| Stimulus construction | Algonauts 2025 | CineBrain | HAD |
|---|---|---|---|
| Bin-wise average pooling | 0.2090 | 0.5643 | 0.2950 |
| Canonical HRF-convolved features | 0.2075 | 0.5618 | 0.2938 |
| Causal FIR lag bank | 0.2112 | 0.5671 | 0.2965 |
| Stimulus access | Algonauts 2025 | CineBrain | HAD |
|---|---|---|---|
| Past fMRI only | 0.1764 | 0.4872 | 0.2761 |
| Past fMRI + stimulus | 0.2021 | 0.5520 | 0.2904 |
| Past fMRI + stimulus | 0.2090 | 0.5643 | 0.2950 |
| Diagnostic stimulus lookahead | 0.2131 | 0.5682 | 0.2961 |
| Setting | Algonauts 2025 | CineBrain | HAD |
|---|---|---|---|
| Past fMRI only | 0.1764 | 0.4872 | 0.2761 |
| Aligned stimulus | 0.2090 | 0.5643 | 0.2950 |
| Circular-shifted stimulus | 0.1788 | 0.4915 | 0.2774 |
| Cross-run replaced stimulus | 0.1775 | 0.4894 | 0.2769 |
| Dataset | #Subj. | Units/subj. | Test frames/subj. | 1-step anchors/subj. | anchors/subj. |
|---|---|---|---|---|---|
| Algonauts 2025 | 4 | 28 / 4 / 4 | |||
| CineBrain | 6 | 18 / 3 / 3 | |||
| HAD | 30 | 12 / 2 / 2 |
| Dataset | Subject | Train units | Val units | Test units | One-step anchors | anchors |
| Algonauts 2025 | sub-01 | 28 | 4 | 4 | 2,397 | 2,321 |
| sub-02 | 28 | 4 | 4 | 2,459 | 2,383 | |
| sub-03 | 28 | 4 | 4 | 2,435 | 2,359 | |
| sub-05 | 28 | 4 | 4 | 2,389 | 2,313 | |
| CineBrain | sub-01 | 18 | 3 | 3 | 7,951 | 7,894 |
| sub-02 | 18 | 3 | 3 | 8,004 | 7,947 |
| ROI group | Raw dimension | Retained dimension | Explained variance | Latent dimension |
|---|---|---|---|---|
| Visual ROI | 8,405 | 512 | 410 | |
| Auditory ROI | 10,541 | 512 | 410 | |
| Total | 18,946 | 1,024 | 820 |
| Components per ROI | Model-space dim. | Explained variance | ||
|---|---|---|---|---|
| 256 | 512 | 410 | 87.1% | 0.5521 |
| 512 | 1,024 | 820 | 92.9% | 0.5643 |
| 768 | 1,536 | 1,230 | 95.2% | 0.5650 |
| Dataset | Video rate | Audio rate | Visual encoder | Audio encoder | Text source and context rule |
|---|---|---|---|---|---|
| Algonauts 2025 | 30 fps | 16 kHz | V-JEPA 2; 2.0s clips; 0.5s stride; end-aligned pooling | Wav2Vec2-BERT; 1.0s windows; 0.5s stride | Dataset-provided subtitles/transcripts; causal prefix ending at or before . |
| CineBrain | 25 fps | 16 kHz | V-JEPA 2; 2.0s clips; 0.4s stride; end-aligned pooling | Wav2Vec2-BERT; 1.0s windows; 0.4s stride | Aligned transcript/subtitles; no sentence tokens after are used. |
| HAD | 30 fps | Not used | V-JEPA 2; 2.0s clips; 0.5s stride; end-aligned pooling | Not used | Clip-level action descriptions and annotations aligned to clip boundaries. |
| Dataset | Pool size | Chance Top10 | Distractor source | Subject constraint |
|---|---|---|---|---|
| Algonauts 2025 | 200 | 5.0% | Same split, different run when available | Same subject |
| CineBrain | 400 | 2.5% | Same split, different episode when available | Same subject |
| HAD | 300 | 3.3% | Same split, different run or clip block | Same subject |
| Dataset | Strongest baseline | Baseline | BrainVista |
|---|---|---|---|
| Algonauts 2025 | BrainHarmony-AR | ||
| CineBrain | BrainHarmony-AR | ||
| HAD | BrainHarmony-AR |
| Component | Setting |
|---|---|
| Brain grouping | Yeo-7 or dataset-defined ROI groups |
| Brain tokenizer | Independent two-layer MLP per network |
| Brain detokenizer | Independent two-layer MLP per network |
| Stimulus encoders | Frozen pretrained encoders |
| Stimulus projection output | |
| Temporal backbone | Pre-LN causal Transformer |
| Component | Setting |
|---|---|
| Tokenizer input dimension | |
| Tokenizer output dimension | |
| Detokenizer input dimension | |
| Detokenizer output dimension | |
| Tokenizer architecture | Two-layer MLP |
| Detokenizer architecture | Two-layer MLP |
| Dataset | Group | Preprocessing | Projection | |||
| Algonauts 2025 | Visual | 162 | Schaefer parcel input | 162 | 130 | |
| Somatomotor | 194 | Schaefer parcel input | 194 | 156 | ||
| Dorsal Attention | 122 | Schaefer parcel input | 122 | 98 | ||
| Ventral Attention | 121 | Schaefer parcel input | 121 | 97 | ||
| Limbic | 60 | Schaefer parcel input | 60 | 48 | ||
| Frontoparietal Ctrl | 129 | Schaefer parcel input | 129 | 104 |
| Component | Default setting |
|---|---|
| Input sequence length | network tokens |
| Shared mixer dimension | |
| Attention heads | 4 |
| Mixer layers | 1 |
| Input projection | Network-specific |
| Output projection | Network-specific |
| Dataset | GPU | Training setup | Total training time |
|---|---|---|---|
| Algonauts 2025 | RTX 5090 | Single GPU, mixed precision | 12 h 32 min |
| CineBrain | RTX 5090 | Single GPU, mixed precision | 17 h 18 min |
| HAD | RTX 5090 | Single GPU, mixed precision | 22 h 46 min |
| Asset | Type | Usage in this work | Source / access condition | License / terms |
|---|---|---|---|---|
| Algonauts 2025 / CNeuroMod challenge data [ 22 ] | Dataset | Naturalistic fMRI forecasting benchmark; used for training, validation, and held-out test evaluation. | Official Algonauts 2025 challenge release. | CC0 |
| CineBrain [ 23 ] | Dataset | Multimodal naturalistic audiovisual fMRI benchmark; visual and auditory ROI responses are reduced with training-set PCA before tokenization. | Official public dataset release. | Apache-2.0 |
| HAD [ 24 ] | Dataset | Human-action fMRI forecasting benchmark; released surface responses are aggregated to Schaefer-1000 parcels for model-space evaluation. | OpenNeuro dataset release. | CC0 |
| V-JEPA 2 [ 50 ] | Pretrained video encoder | Frozen visual feature extractor for video stimulus streams. | Official pretrained checkpoint / repository. | MIT; selected utility files under Apache-2.0 |
| W2v-BERT 2.0 [ 51 ] | Pretrained audio encoder | Frozen audio feature extractor for auditory stimulus streams. | Official model card / checkpoint release. | MIT |
| Llama 3.2 3B [ 52 , 53 ] | Pretrained language model | Frozen text feature extractor for subtitles, transcripts, or aligned textual descriptions. | Meta model release. | Llama 3.2 Community License |
| Model | Original role | Input under our protocol | Adaptation |
|---|---|---|---|
| TRIBE | Stimulus-to-brain encoding | Aligned video, audio, and text features | Used as a stimulus-only reference on the same target frames. No future fMRI state is provided. |
| MindSimulator | Synthetic fMRI / concept-localization modeling | Stimulus features and dataset-specific ROI/parcellation space | Used as a non-ARX reference with its prediction branch adapted to each dataset’s output space. No target fMRI value is provided as input. |
| BrainLM | Self-supervised fMRI representation learning | Past fMRI sequence | The encoder is frozen, and a lightweight causal forecasting head is trained on top of the learned representation. |
| BrainJEPA | Predictive brain representation learning | Past fMRI sequence | The encoder is frozen, and representation tokens are passed to the same lightweight forecasting interface. |
| Brain Harmony | Brain foundation representation model | Past fMRI sequence | Used as a non-ARX representation reference with a dataset-specific prediction head. |
| Ridge-ARX | Linear autoregressive baseline | Flattened past fMRI tokens and aligned stimulus tokens | Ridge regression predicts the next fMRI state from the concatenated context window and is recursively rolled out with predicted fMRI states. |
| Model | Past fMRI | Observed stimulus | Target/future fMRI |
|---|---|---|---|
| TRIBE, MindSimulator | No | Yes | No |
| BrainLM, BrainJEPA, Brain Harmony | Yes | No | No |
| Ridge-, LSTM-, TemporalCNN-ARX | Yes | Yes | No |
| Transformer-ARX, BrainHarmony-AR | Yes | Yes | No |
| BrainVista | Yes | Yes | No |
| Model | ||||||
|---|---|---|---|---|---|---|
| Non-ARX stimulus-to-brain and representation references | ||||||
| TRIBE | 0.5711 | 0.5419 | 0.3150 | 0.2845 | 0.2685 | 0.1845 |
| MindSimulator | 0.5782 | 0.5698 | 0.3210 | 0.2960 | 0.2610 | 0.1960 |
| BrainLM | 0.5695 | 0.5489 | 0.2785 | 0.2250 | 0.2465 | 0.1450 |
| BrainJEPA | 0.5971 | 0.5723 | 0.2950 | 0.2810 | 0.2515 | 0.2353 |
| Model | fMRI | Stim. | Layers | Hidden | Params A/C/H | Adaptation / protocol |
|---|---|---|---|---|---|---|
| Ridge-ARX | Yes | Yes | 1 | – | M / M / M | Regularized linear ARX over flattened context windows |
| LSTM-ARX | Yes | Yes | 2 | 256 | M / M / M | Recurrent model over projected fMRI–stimulus tokens |
| TemporalCNN-ARX | Yes | Yes | 4 | 256 | M / M / M | Causal temporal convolution over matched input sequence |
| Transformer-ARX | Yes | Yes | 4 | 256 | M / M / M | Transformer forecasting baseline using matched fMRI–stimulus inputs |
| BrainHarmony-AR | Yes | Yes | 2 | 256 | M / M / M | Frozen Brain Harmony representation with causal forecasting head |
| Algonauts 2025 | CineBrain | |||||||||||
| Model | ||||||||||||
| TRIBE | 0.3695 | 0.3545 | 0.2045 | 0.1962 | 0.1379 | 0.1323 | 0.7647 | 0.4857 | 0.5714 | 0.3630 | 0.4742 | 0.3012 |
| MindSimulator | 0.3837 | 0.3960 | 0.2124 | 0.2192 | 0.1432 | 0.1478 | 0.7831 | 0.5217 | 0.5852 | 0.3898 | 0.4856 | 0.3235 |
| BrainLM | 0.3191 | 0.2221 | 0.1767 | 0.1230 | 0.1191 | 0.0829 | 0.7471 | 0.4454 | 0.5583 | 0.3328 | 0.4633 | 0.2762 |
| BrainJEPA | 0.4000 | 0.2454 | 0.2214 | 0.1359 | 0.1493 | 0.0916 | 0.7968 | 0.3943 | 0.5954 | 0.2946 | 0.4941 | 0.2445 |
| Dataset | BrainVista | Split-half reliability | Reliability ceiling | NC-Norm |
|---|---|---|---|---|
| Algonauts 2025 | 0.2090 | 0.243 | 0.391 | 0.535 |
| CineBrain | 0.5643 | 0.586 | 0.739 | 0.764 |
| HAD | 0.2950 | 0.338 | 0.505 | 0.584 |
| Dataset | BrainHarmony-AR | BrainVista | 95% CI of | ||
|---|---|---|---|---|---|
| Algonauts 2025 | 4 | ||||
| CineBrain | 6 | ||||
| HAD | 30 |
| Dataset | Subject | BH-AR | BrainVista | |
|---|---|---|---|---|
| Algonauts 2025 | sub-01 | 0.1814 | 0.1970 | 0.0156 |
| sub-02 | 0.1920 | 0.2109 | 0.0189 | |
| sub-03 | 0.1856 | 0.2074 | 0.0218 | |
| sub-05 | 0.1958 | 0.2207 | 0.0249 | |
| CineBrain | sub-01 | 0.5054 | 0.5466 | 0.0412 |
| sub-02 | 0.5180 | 0.5631 | 0.0451 |
| Subj. | BH-AR | BrainVista | Subj. | BH-AR | BrainVista | Subj. | BH-AR | BrainVista | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 01 | 0.2526 | 0.2652 | 0.0126 | 11 | 0.2640 | 0.2810 | 0.0170 | 21 | 0.2851 | 0.3048 | 0.0197 |
| 02 | 0.2507 | 0.2639 | 0.0133 | 12 | 0.2727 | 0.2900 | 0.0173 | 22 | 0.2901 | 0.3100 | 0.0199 |
| 03 | 0.2517 | 0.2657 | 0.0140 | 13 | 0.2747 | 0.2923 | 0.0176 | 23 | 0.2973 | 0.3174 | 0.0202 |
| 04 | 0.2479 | 0.2626 | 0.0147 | 14 | 0.2793 | 0.2971 | 0.0178 | 24 | 0.2956 | 0.3160 | 0.0205 |
| 05 | 0.2547 | 0.2696 | 0.0149 | 15 | 0.2764 | 0.2944 | 0.0180 | 25 | 0.2939 | 0.3146 | 0.0208 |
| 06 | 0.2566 | 0.2719 | 0.0154 | 16 | 0.2771 | 0.2954 | 0.0183 | 26 | 0.2969 | 0.3180 | 0.0212 |
| Dataset | Win count | Mean | Sign test | Wilcoxon |
|---|---|---|---|---|
| Algonauts 2025 | ||||
| CineBrain | ||||
| HAD |
| Algonauts 2025 | CineBrain | HAD | |||||||
| Model | MSE | 2v2 | MSE | 2v2 | MSE | 2v2 | |||
| TRIBE | 1.372 | 0.110 | 0.729 | 0.405 | 0.779 | 0.972 | 1.215 | 0.241 | 0.736 |
| MindSimulator | 1.243 | 0.115 | 0.731 | 0.382 | 0.783 | 0.974 | 1.154 | 0.249 | 0.739 |
| BrainLM | 1.486 | 0.072 | 0.695 | 0.428 | 0.751 | 0.968 | 1.284 | 0.198 | 0.711 |
| BrainJEPA | 1.401 | 0.083 | 0.702 | 0.446 | 0.728 | 0.969 | 1.063 | 0.224 | 0.742 |
| Brain Harmony | 1.058 | 0.140 | 0.760 | 0.338 | 0.790 | 0.981 | 0.741 | 0.239 | 0.811 |
| Dataset | Model | ||||
|---|---|---|---|---|---|
| Algonauts 2025 | BH-AR | 0.204 | 0.083 | 0.071 | 0.192 |
| BrainVista | 0.247 | 0.067 | 0.058 | 0.163 | |
| CineBrain | BH-AR | 0.598 | 0.061 | 0.052 | 0.158 |
| BrainVista | 0.641 | 0.048 | 0.043 | 0.132 | |
| HAD | BH-AR | 0.279 | 0.074 | 0.066 | 0.177 |
| BrainVista | 0.298 | 0.063 | 0.056 | 0.151 |
| Context length | |||
|---|---|---|---|
| 10 | 0.585 | 0.287 | 0.213 |
| 20 | 0.607 | 0.326 | 0.269 |
| 40 | 0.621 | 0.339 | 0.295 |
| 60 | 0.622 | 0.340 | 0.294 |
| Dataset | Recon. MSE | Recon. | Explained variance |
|---|---|---|---|
| Algonauts 2025 | 0.041 | 0.958 | 0.914 |
| CineBrain | 0.047 | 0.951 | 0.901 |
| HAD | 0.044 | 0.955 | 0.908 |
| Normalization | |||
|---|---|---|---|
| Training-set per-network | 0.621 | 0.339 | 0.295 |
| Subject-level | 0.614 | 0.331 | 0.287 |
| Global | 0.606 | 0.321 | 0.279 |
| Run-wise per-network diagnostic | 0.618 | 0.336 | 0.292 |
| Challenge | BrainVista component | Diagnostic evidence |
|---|---|---|
| Functional heterogeneity across cortical systems | Network-wise Tokenizers | Tokenization granularity ablation and network-level rollout fidelity. |
| Temporal leakage under mismatched stimulus–BOLD timescales | Stimulus-to-Brain mask | Mask ablation comparing bidirectional, vanilla causal, and S2B-causal attention. |
| Error accumulation in recursive forecasting | Conditional autoregressive rollout with past brain-state feedback | Horizon breakdown and context-length sensitivity. |
| Cross-network interaction without over-smoothing | Spatial Mixer Head | Spatial Mixer ablation and model-based attention diagnostics. |
| Dependence on multimodal sensory drive | Time-aligned multimodal stimulus tokens | Input-source and modality ablations. |
| Variant | Cls@Top10 | ||
|---|---|---|---|
| Continuous fMRI input | 0.151 | 0.118 | 0.673 |
| Shared full-brain tokenizer | 0.184 | 0.166 | 0.721 |
| Network-wise tokenizer | 0.218 | 0.209 | 0.762 |
| Mask | |||||
|---|---|---|---|---|---|
| Bidirectional attention | 0.545 | 0.361 | 0.245 | 0.169 | 0.113 |
| Vanilla causal mask | 0.552 | 0.380 | 0.270 | 0.195 | 0.142 |
| S2B mask | 0.560 | 0.403 | 0.310 | 0.251 | 0.209 |
| S2B relative gain |
| Variant | |||
|---|---|---|---|
| w/o Spatial Mixer | 0.548 | 0.294 | 0.181 |
| Linear Mixer | 0.553 | 0.302 | 0.195 |
| Spatial Mixer Head | 0.560 | 0.310 | 0.209 |
| Training strategy | |||
|---|---|---|---|
| End-to-end training | 0.574 | 0.246 | 0.173 |
| Pretrained tokenizer + frozen encoder | 0.621 | 0.339 | 0.295 |
| Family | Objective | |
|---|---|---|
| MLP | AE | 0.834 |
| MLP | VAE | 0.507 |
| BrainVAE | AE | 0.814 |
| BrainVAE | VAE | 0.797 |
| Tokenizer | Recon. | Recon. MSE | |||
|---|---|---|---|---|---|
| VQ-AE | 0.916 | 0.080 | 0.5384 | 0.2887 | 0.1938 |
| -VAE | 0.934 | 0.064 | 0.5521 | 0.3021 | 0.2031 |
| AE + SoftCLIP | 0.947 | 0.052 | 0.5613 | 0.3062 | 0.2044 |
| Reconstruction only | 0.958 | 0.041 | 0.5600 | 0.3100 | 0.2090 |
| Variant | |||
|---|---|---|---|
| All (V+A+T) | 0.560 | 0.310 | 0.209 |
| w/o Video (A+T) | 0.510 | 0.270 | 0.184 |
| w/o Audio (V+T) | 0.540 | 0.295 | 0.200 |
| w/o Text (V+A) | 0.538 | 0.293 | 0.198 |
| Video only (V) | 0.485 | 0.255 | 0.175 |
| Audio only (A) | 0.455 | 0.235 | 0.165 |
| Frozen stimulus encoder(s) | Cls@Top10 | ||
|---|---|---|---|
| Main paper encoders | 0.2183 | 0.2090 | 0.7619 |
| ImageBind | 0.2110 | 0.1990 | 0.7420 |
| Qwen Omni 2.5 | 0.2020 | 0.1880 | 0.7240 |
| Training / evaluation setting | BH-AR | BrainVista | |
|---|---|---|---|
| Within-subject training | |||
| Single-source cross-subject | |||
| Leave-one-subject-out |
| Test subject | Training subject | |||
|---|---|---|---|---|
| Sub-01 | Sub-02 | Sub-03 | Sub-05 | |
| Sub-01 | 0.1814 | 0.1628 | 0.1660 | 0.1542 |
| Sub-02 | 0.1605 | 0.1920 | 0.1640 | 0.1558 |
| Sub-03 | 0.1650 | 0.1586 | 0.1856 | 0.1516 |
| Sub-05 | 0.1533 | 0.1560 | 0.1518 | 0.1958 |
| Test subject | Training subject | |||
|---|---|---|---|---|
| Sub-01 | Sub-02 | Sub-03 | Sub-05 | |
| Sub-01 | 0.1970 | 0.1850 | 0.1881 | 0.1717 |
| Sub-02 | 0.1824 | 0.2109 | 0.1859 | 0.1761 |
| Sub-03 | 0.1875 | 0.1813 | 0.2074 | 0.1738 |
| Sub-05 | 0.1762 | 0.1790 | 0.1758 | 0.2207 |
| Subject | Visual | SomMot | DorsAttn | VentrAttn | Limbic | FPN/Control | Default |
|---|---|---|---|---|---|---|---|
| sub01 | 0.3512 | 0.2830 | 0.3131 | 0.2673 | 0.1807 | 0.2690 | 0.3237 |
| sub02 | 0.3381 | 0.2605 | 0.2977 | 0.2505 | 0.1695 | 0.2595 | 0.3092 |
| sub03 | 0.3420 | 0.2726 | 0.3020 | 0.2581 | 0.1715 | 0.2592 | 0.3127 |
| sub05 | 0.3275 | 0.2520 | 0.2835 | 0.2415 | 0.1585 | 0.2465 | 0.3015 |
| Feature | ARI | NMI |
|---|---|---|
| Ground-truth brain feature | 0.318 | 0.392 |
| BrainHarmony-AR | 0.138 | 0.214 |
| BrainVista | 0.214 | 0.286 |
| Emotion category | BrainHarmony-AR | BrainVista |
|---|---|---|
| Happy | 0.46 | 0.58 |
| Sad | 0.39 | 0.50 |
| Neutral | 0.58 | 0.67 |
| Angry | 0.42 | 0.54 |
| Worried | 0.34 | 0.44 |
| Surprise | 0.39 | 0.49 |
| Data | Units/subject | Anchors/subject | Relative | |||
|---|---|---|---|---|---|---|
| 25% | 7 | 2.9k | 0.5174 | 0.2589 | 0.1698 | 81.2% |
| 50% | 14 | 5.9k | 0.5408 | 0.2818 | 0.1874 | 89.7% |
| 75% | 21 | 8.8k | 0.5528 | 0.2978 | 0.2003 | 95.8% |
| 100% | 28 | 11.8k | 0.5600 | 0.3100 | 0.2090 | 100% |
| Model | GPU-h/subject | |||
|---|---|---|---|---|
| Small | 1.4 | 0.5417 | 0.2913 | 0.1938 |
| Base | 2.1 | 0.5600 | 0.3100 | 0.2090 |
| Large | 4.9 | 0.5641 | 0.3134 | 0.2118 |
| Model | Train | Validation | Gap | Best epoch/40 | Test |
|---|---|---|---|---|---|
| Small | 0.2178 | 0.1902 | 0.0276 | 38 | 0.1938 |
| Base | 0.2611 | 0.2047 | 0.0564 | 34 | 0.2090 |
| Large | 0.3094 | 0.2069 | 0.1025 | 27 | 0.2118 |
| Component / metric | Algonauts 2025 | CineBrain | HAD |
|---|---|---|---|
| Raw brain dimension | 1,000 | 18,946 | fsLR surface |
| Model-space input dimension | 1,000 | 1,024 | 1,000 |
| Concatenated brain latent | 803 | 820 | 803 |
| Transformer hidden size | 256 | 256 | 256 |
| Brain projection | |||
| Peak training memory | 24.0 GB | 26.8 GB | 24.3 GB |
Neuroimaging
Neuroimaging uses brain measurements and computational methods to map anatomy, activity, connectivity, and disease.
See all 249 papers