Samples, Sources, Space: Decomposing Data Scale in Spatially Structured Representation Learning of Human Brain Microarchitecture
Authors: Christian Schiffer, Mathis Bode, Thomas Lippert, Katrin Amunts, Timo Dickscheid
Organizations: Institute of Neuroscience and Medicine (INM-1), Research Centre Jülich, Jülich, Germany · Helmholtz AI, Research Centre Jülich, Jülich, Germany · Jülich Supercomputing Centre, Research Centre Jülich, Jülich, Germany · Frankfurt Institute for Advanced Studies, Goethe University Frankfurt, Frankfurt am Main, Germany · C´ecile & Oscar Vogt Institute for Brain Research, University Hospital Düsseldorf, Düsseldorf, Germany · Computer Vision, Institute for Computational Visualistics, University of Koblenz, Koblenz, Germany
Scaling studies typically represent training data by a single count of samples. For hierarchically and spatially structured data, however, the same number of samples can be drawn from few or many sources and distributed differently across the underlying domain. We therefore study data scaling as an allocation problem, separating unique sample count, source diversity, and spatial coverage. We study this decomposition in microscopic whole-brain histology, where a source is an individual brain, and a sample is an image patch at a specific spatial location. Across 93 controlled pretraining runs of a contrastive model that uses spatial proximity for supervision, we vary data allocation, compute, and model capacity over 11.6 million spatially anchored image patches from 21 human brains. Performance improves with more unique samples, broader spatial coverage, additional compute, and larger model capacity. At fixed sample count, distributing samples across one to 18 subjects produces no detectable improvement, even though representations generalize substantially better to subjects encountered during pretraining. Inter-subject variation therefore strongly affects generalization, but additional subjects provide no benefit when a fixed sample budget is distributed across more sources. These results establish sample count, source diversity, and spatial coverage as distinct axes of data scaling in spatially structured representation learning.
Figures & tables
Figure 1: Experimental decomposition of data scale and its instantiation in whole-brain histology. E1–E4 vary the composition of the training data by redistributing a fixed number of samples across more sources, increasing samples together with sources, increasing samples within fixed sources, or changing spatial coverage at fixed sample and source counts. The lower pictograms show the corresponding manipulations for brain histology, where sources are individual subjects and samples are spatially anchored image patches. E5–E8 provide complementary controls over training compute, model capacity, cross-subject contributions to the SpatialNCE objective, and the choice of held-out evaluation subjects. See table 1 for details. The data hierarchy on the right maps the abstract units to the imaging pipeline from whole brains, to histological section, and image patches.
Experiment
Manipulation
Levels
Runs
E1
Fixed samples, more sources
redistribute Nunique=450,000$$ across sources
B∈{1,2,4,8,18}
9
E2
More samples, more sources
retain selected sections as B increases
B∈{1,2,4,8,18}×{ all, every 10th }
30
E3
More samples, fixed sources
vary retained-section fraction at fixed B
B∈{1,9,18}×{ all, every 3rd, every 10th }
5
E4
Coverage
arrange fixed Nunique into k section blocks
B∈{1,4,13} , k∈{ every 10th, 16, 4, 1 }
19
E5
Compute
vary optimization steps
0.5× , 1× , 2×
7
E6
Capacity
vary encoder size
ViT-B, ViT-L, ViT-H
14
Table 1: Overview of the eight experiments and 93 pretraining runs. Run counts are incremental, with reused configurations counted only once. Unless stated otherwise, runs use ViT-B and the default presentation budget. In this study, a source corresponds to a subject, and a section is an ordered spatial subdivision of that source along the anatomical domain.
Figure 2: Subject inclusion in pretraining dominates performance. A : Macro-F1 score gain when a subject was included in the pretraining pool of a model vs. when it was not; averaged over n models (gray line: MDE 0.057 ), at identical evaluation samples, labels, and class denominator. B : Score distribution by generalization level: unseen subjects are absent from pretraining and probe training, transfer subjects only from probe training, and held-out-section subjects contribute other sections to both. C : Familiarity gain against Nunique/B , the mean number of locations a model saw per subject. Each point is one model from E1 , E2 , E3 , E6 , giving its subject-centered macro-F1 gain on pretraining subjects over other subjects.
Figure 3: Scaling subject count B without scaling Nunique does not improve representations. A : Macro-F1 against subject count B at fixed Nunique≈450,000$$ ( E1 in table 1 ), with solid lines giving the mean per capacity P (color) and the MDE band anchored at the B=1 mean (dashed line). The curve is flat at every capacity. B : The same axis overlaid with E2 , where Nunique grows linearly with B . C : 95% intervals of coefficients per doubling on the logit scale. Intervals for βB and the capacity interaction γBP include zero, while performance improves with sample count βN and compute βC .
Figure 4: Scaling samples Nunique , compute C , and capacity P improve representations. A : Macro-F1 against Nunique across the grid ( βN=0.09 / doubling, dashed line), with marker color, style, and size giving capacity P , coverage ρ , and subject count B . B : Macro-F1 against compute C , subject count B (color), and coverage ρ (marker style) across the 7 settings of E5 .
Figure 5: Coverage geometry, positive weight, and subject gating at fixed sample count. A : Adjusted logit macro-F1 differences among the coverage conditions in E4 relative to every-10th-section sampling, with 95% confidence intervals, ordered by spatial concentration. B : Positive weight M for the corresponding E4 conditions, with every-10th-section sampling as the reference. C : Effect of subject gating in E7 , which sets all cross-subject SpatialNCE weights to zero. Each gated configuration is paired with an ungated configuration using the same data composition. Annotations indicate the fraction of M contributed by cross-subject pairs before gating. At B=2 , where cross-subject pairs contribute 49% of M , the gating contrast is unresolved. At B=18 , where cross-subject pairs contribute 94% of M , gating reduces macro-F1 by 0.057 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
subject
locations
sections
annotated
plane
role
B01
644224
468
332
coronal
pretraining
B02
522551
462
253
coronal
pretraining
B03
610975
506
298
coronal
pretraining
B04
570373
475
372
coronal
transfer
B05
461204
451
355
coronal
pretraining
B06
571496
469
385
coronal
pretraining
Appendix
Table 2: Dataset overview. For each subject, we report the number of sampling locations, retained sections, annotated sections, cutting plane, and role in the study. Subjects are identified by numeric IDs.
Variant
Width
Heads
Layers
Accelerators
ViT-B
768
12
12
64
ViT-L
1024
16
24
128
ViT-H
1280
16
32
256
Appendix
Table 3: Encoder dimensions. Depth, width, and head count follow the standard ViT variants, while token grid and register count are identical across capacities.
fit
term
VIF
full grid
βN
1.92
full grid
βB
1.73
full grid
βC
1.00
full grid
βM
1.14
fixed sample
βB
1.37
fixed sample
βM
1.37
Appendix
Table 4: Variance inflation factors for the continuous predictors in the full-grid model. All VIFs are below 2 .
Figure 6: Simulated power for detecting a capacity-by-subject-count interaction γBP under E1. Power is shown as a function of the imposed interaction magnitude and residual variance. At the fitted residual variance, approximately 80% power is reached for an interaction of 0.05 logit units per doubling of subject count.
condition ρ
sections
A–P span (mm)
largest gap (mm)
all
467
180
1.0
every 10th
47
180
5.2
16 blocks
78
175
11.9
4 blocks
52
142
42.5
loc. posterior
97
37
0.5
loc. center
31
11
0.5
Appendix
Table 5: Geometry of the coverage conditions ρ in E4 at B=13 , averaged across the 13 coronal subjects. All conditions use the same sample budget but differ in anatomical extent and spatial fragmentation.
Figure 7: Positive weight across E1. A : Total positive weight M remains approximately constant as subject count increases from one to 18. B : The fraction of M contributed by cross-subject pairs increases from zero to 94.2% over the same range. At fixed Nunique , E1 therefore preserves the overall positive weight while changing the composition of contributing pairs.
B
sampling
gated
subsets
Nunique
M
cross-subject (%)
1
every 10th
✗
5
51709
15.23
0.0
2
every 10th
✗
4
108881
16.31
49.0
4
every 10th
✗
5
219266
15.95
74.0
8
every 10th
✗
2
438674
15.85
86.9
9
every 10th
✗
1
472504
15.66
88.4
13
every 10th
✗
1
701132
16.15
92.0
Appendix
Table 6: Positive weight M and its cross-subject contribution for every pretraining configuration, averaged across subject subsets at each level. M is the covariate Mc used in the mixed-effects model and is computed from the configuration before training.
Figure 8: Macro-F1 on held-out sections from familiar subjects as a function of distance to the nearest section seen during pretraining in the B=13 runs of E4. Each line corresponds to one coverage condition ρ . Macro-F1 decreases with distance from the nearest pretraining section in five of the six coverage conditions.
Figure 9: Training trajectories for five B=18 full-coverage configurations from E5–E7 and their shared baseline. The top panel shows SpatialNCE loss, and the bottom panel shows the gradient of the smoothed loss trajectory. The loss continues to decrease at the doubled budget of 146400 steps. See table 8 for the corresponding configurations.
Figure 10: Macro-F1 for the B=18 configurations of E1 and E2 under two alternative unseen-subject assignments. Absolute performance varies across assignments, while the direction and approximate magnitude of the difference between E1 and E2 are preserved.
Encoder
Reference
Macro-F1
Uniform random guess
—
0.004
DINOv2 (ViT-B/14)
Oquab et al. (2024)
0.168
GigaPath (ViT-G/16)
Xu et al. (2024)
0.220
UNI (ViT-L/16)
Chen et al. (2024)
0.225
Virchow2 (ViT-H/14)
Zimmermann et al. (2024)
0.251
ViT-B/32
ours
0.300
Appendix
Table 7: Reference encoders evaluated as frozen feature extractors using the same probe and unseen-subject evaluation protocol. Our rows correspond to the B=18 configurations of E6 at each model capacity. The comparison provides an absolute performance reference across models with different architectures, pretraining data, and computational budgets. The chance row states where the macro-F1 scale begins under the same per-subject class denominator.
exp.
run
arch
B
sampling
Nunique
steps
gated
fit
E1
vit_b_p32_b01_subset03_iso_unique
B
1
fixed budget
449924
73200
✗
✓
E1
vit_b_p32_b01_subset04_iso_unique
B
1
fixed budget
450792
73200
✗
✓
E1
vit_b_p32_b02_subset01_iso_unique
B
2
fixed budget
449883
73200
✗
✓
E1
vit_b_p32_b02_subset02_iso_unique
B
2
fixed budget
450825
73200
✗
✓
E1
vit_b_p32_b04_subset01_iso_unique
B
4
fixed budget
450404
73200
✗
✓
E1
vit_b_p32_b04_subset02_iso_unique
B
4
fixed budget
449245
73200
✗
✓
Appendix
Table 8: Inventory of all 93 pretraining runs, grouped by experiment. Steps gives the completed optimization budget. The seven runs at 146400 steps also contribute a retained checkpoint at half that budget. Gated marks the runs whose objective admits no cross-subject positives. Fit marks inclusion in the full-grid model. The four runs on alternative held-out triples are analyzed only in section D.5 .
Scaling laws have guided large-model development in computer vision and natural language processing, but the relationships among data, model size, and compute remain unclear for functional magnetic resonance imaging (fMRI) foundation models. Here, we conduct a controlled empirical study using pretraining data from more than 200 source datasets and over 10,000 GPU-hours of experiments. Holding the pretraining framework and downstream protocol fixed, we vary pretraining data size, model size, and training duration. Downstream performance generally improves with compute, yet models using similar compute can perform substantially differently. Additional pretraining data bring larger gains at larger model sizes, suggesting that data and model size should be scaled together. At matched compute, increasing pretraining data benefits more tasks than increasing model size, although the pattern varies across tasks. We then use in-distribution (ID) downstream performance to select the combination of pretraining data size, model size, and training duration at two fixed compute budgets. The resulting models are locked before out-of-distribution (OOD) evaluation. They achieve the highest average performance across the evaluated OOD tasks among the compared fMRI foundation models while using less pretraining compute. Overall, our results show that compute alone does not characterize fMRI scaling: performance depends on how pretraining data, model size, and training duration are combined.
Wenhao Ye, Xuanye Pan, Junfeng Xia +3
Southern University of Science and Technology · Shenzhen University · SLAI +1
If a person can solve a task, can measuring their brain make it easier to train a model to solve that task too? Recent NeuroAI work suggests that supplementing task training with neural recordings can modestly improve model performance and robustness. However, it is unclear when there should be a benefit from using neural data and how much benefit to expect. We formulate this question mathematically, and begin to address it theoretically using a simple, analytically tractable linear gaussian model of task targets and neural recordings. For a multimodal estimator trained on both brain data and task labels, we derive scaling laws for how performance scales with the numbers of brain and task samples. From these laws we derive relative value and exchange rates between brain samples and task samples, quantifying how much extra task samples neural data is worth as a function of task-brain alignment, neural and task noise, latent dimension, and brain data sample size. We also analyze test distribution shift, to identify conditions where brain-regularized learning can produce substantial robustness gains through learned invariances. Finally, under a fixed collection budget, we characterize the regimes in which brain data is worth collecting. Our results provide a foundation for understanding how valuable brain data could be for improving machine learning.
Lane Lewis, Zhixin Wang, David Schwab +1
Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA · Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · NSF AI Institute for Artificial and Natural Intelligence (ARNI) +2
Spatial Transcriptomics (ST) profiles thousands of gene expression values at discrete spots with precise coordinates on tissue sections, preserving spatial context essential for clinical and pathological studies. With rising sequencing throughput and advancing platforms, the expanding data volumes motivate large-scale ST pretraining. However, the fundamental unit for pretraining, i.e., what constitutes a single training sample, remains ill-posed. Existing choices fall into two camps: (1) treating each spot as an independent sample, which discards spatial dependencies and collapses ST into single-cell transcriptomics; and (2) treating an entire slide as a single sample, which produces prohibitively large inputs and drastically fewer training examples, undermining effective pretraining. To address this gap, we propose treating spatial transcriptomics as croppable images. Specifically, we define a multi-channel image representation with fixed spatial size by cropping patches from raw slides, thereby preserving spatial context while substantially increasing the number of training samples. Along the channel dimension, we define gene subset selection rules to control input dimensionality and improve pretraining stability. Extensive experiments show that the proposed image-like dataset construction for ST pretraining consistently improves downstream performance, outperforming conventional pretraining schemes. Ablation studies verify that both spatial patching and channel design are necessary, establishing a unified, practical paradigm for organizing ST data and enabling large-scale pretraining.
Yishun Zhu, Jiaxin Qi, Jian Wang +2
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of the Chinese Academy of Sciences, Zhejiang, China