Samples, Sources, Space: Decomposing Data Scale in Spatially Structured Representation Learning of Human Brain Microarchitecture

Christian Schiffer; Mathis Bode; Thomas Lippert; Katrin Amunts; Timo Dickscheid

cs.AISep 25, 2026

Samples, Sources, Space: Decomposing Data Scale in Spatially Structured Representation Learning of Human Brain Microarchitecture

Authors: Christian Schiffer, Mathis Bode, Thomas Lippert, Katrin Amunts, Timo Dickscheid

Organizations: Institute of Neuroscience and Medicine (INM-1), Research Centre Jülich, Jülich, Germany · Helmholtz AI, Research Centre Jülich, Jülich, Germany · Jülich Supercomputing Centre, Research Centre Jülich, Jülich, Germany · Frankfurt Institute for Advanced Studies, Goethe University Frankfurt, Frankfurt am Main, Germany ·

Abstract

Scaling studies typically represent training data by a single count of samples. For hierarchically and spatially structured data, however, the same number of samples can be drawn from few or many sources and distributed differently across the underlying domain. We therefore study data scaling as an allocation problem, separating unique sample count, source diversity, and spatial coverage. We study this decomposition in microscopic whole-brain histology, where a source is an individual brain, and a sample is an image patch at a specific spatial location. Across 93 controlled pretraining runs of a contrastive model that uses spatial proximity for supervision, we vary data allocation, compute, and model capacity over 11.6 million spatially anchored image patches from 21 human brains. Performance improves with more unique samples, broader spatial coverage, additional compute, and larger model capacity. At fixed sample count, distributing samples across one to 18 subjects produces no detectable improvement, even though representations generalize substantially better to subjects encountered during pretraining. Inter-subject variation therefore strongly affects generalization, but additional subjects provide no benefit when a fixed sample budget is distributed across more sources. These results establish sample count, source diversity, and spatial coverage as distinct axes of data scaling in spatially structured representation learning.

Figures & tables

Figure 1: Experimental decomposition of data scale and its instantiation in whole-brain histology. E1–E4 vary the composition of the training data by redistributing a fixed number of samples across more sources, increasing samples together with sources, increasing samples within fixed sources, or changing spatial coverage at fixed sample and source counts. The lower pictograms show the corresponding manipulations for brain histology, where sources are individual subjects and samples are spatially anchored image patches. E5–E8 provide complementary controls over training compute, model capacity, cross-subject contributions to the SpatialNCE objective, and the choice of held-out evaluation subjects. See table 1 for details. The data hierarchy on the right maps the abstract units to the imaging pipeline from whole brains, to histological section, and image patches.

	Experiment	Manipulation	Levels	Runs
E1	Fixed samples, more sources	redistribute $N_{\mathrm{unique}}=$ 450,000$$ across sources	$B\in\{1,2,4,8,18\}$	9
E2	More samples, more sources	retain selected sections as $B$ increases	$B\in\{1,2,4,8,18\}\times\{$ all, every 10th $\}$	30
E3	More samples, fixed sources	vary retained-section fraction at fixed $B$	$B\in\{1,9,18\}\times\{$ all, every 3rd, every 10th $\}$	5
E4	Coverage	arrange fixed $N_{\mathrm{unique}}$ into $k$ section blocks	$B\in\{1,4,13\}$ , $k\in\{$ every 10th, 16, 4, 1 $\}$	19
E5	Compute	vary optimization steps	$0.5\times$ , $1\times$ , $2\times$	7
E6	Capacity	vary encoder size	ViT-B, ViT-L, ViT-H	14

Table 1: Overview of the eight experiments and 93 pretraining runs. Run counts are incremental, with reused configurations counted only once. Unless stated otherwise, runs use ViT-B and the default presentation budget. In this study, a source corresponds to a subject, and a section is an ordered spatial subdivision of that source along the anatomical domain.

Figure 2: Subject inclusion in pretraining dominates performance. A : Macro-F1 score gain when a subject was included in the pretraining pool of a model vs. when it was not; averaged over $n$ models (gray line: MDE $0.057$ ), at identical evaluation samples, labels, and class denominator. B : Score distribution by generalization level: unseen subjects are absent from pretraining and probe training, transfer subjects only from probe training, and held-out-section subjects contribute other sections to both. C : Familiarity gain against $N_{\mathrm{unique}}/B$ , the mean number of locations a model saw per subject. Each point is one model from E1 , E2 , E3 , E6 , giving its subject-centered macro-F1 gain on pretraining subjects over other subjects.

Figure 3: Scaling subject count

B

without scaling

N_{\mathrm{unique}}

does not improve representations. A : Macro-F1 against subject count

B

at fixed

N_{\mathrm{unique}}\!\approx\!

450,000$$ ( E1 in table 1 ), with solid lines giving the mean per capacity

P

(color) and the MDE band anchored at the

B{=}1

mean (dashed line). The curve is flat at every capacity. B : The same axis overlaid with E2 , where

N_{\mathrm{unique}}

grows linearly with

B

. C : 95% intervals of coefficients per doubling on the logit scale. Intervals for

\beta_{B}

and the capacity interaction

\gamma_{BP}

include zero, while performance improves with sample count

\beta_{N}

and compute

\beta_{C}

Figure 4: Scaling samples $N_{\mathrm{unique}}$ , compute $C$ , and capacity $P$ improve representations. A : Macro-F1 against $N_{\mathrm{unique}}$ across the grid ( $\beta_{N}=0.09$ / doubling, dashed line), with marker color, style, and size giving capacity $P$ , coverage $\rho$ , and subject count $B$ . B : Macro-F1 against compute $C$ , subject count $B$ (color), and coverage $\rho$ (marker style) across the 7 settings of E5 .

Figure 5: Coverage geometry, positive weight, and subject gating at fixed sample count. A : Adjusted logit macro-F1 differences among the coverage conditions in E4 relative to every-10th-section sampling, with 95% confidence intervals, ordered by spatial concentration. B : Positive weight

M

for the corresponding E4 conditions, with every-10th-section sampling as the reference. C : Effect of subject gating in E7 , which sets all cross-subject SpatialNCE weights to zero. Each gated configuration is paired with an ungated configuration using the same data composition. Annotations indicate the fraction of

M

contributed by cross-subject pairs before gating. At

B=2

, where cross-subject pairs contribute

49\%

M

, the gating contrast is unresolved. At

B=18

, where cross-subject pairs contribute

94\%

M

, gating reduces macro-F1 by

0.057

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

subject	locations	sections	annotated	plane	role
B01	$644\,224$	468	332	coronal	pretraining
B02	$522\,551$	462	253	coronal	pretraining
B03	$610\,975$	506	298	coronal	pretraining
B04	$570\,373$	475	372	coronal	transfer
B05	$461\,204$	451	355	coronal	pretraining
B06	$571\,496$	469	385	coronal	pretraining

Appendix

Table 2: Dataset overview. For each subject, we report the number of sampling locations, retained sections, annotated sections, cutting plane, and role in the study. Subjects are identified by numeric IDs.

Variant	Width	Heads	Layers	Accelerators
ViT-B	768	12	12	64
ViT-L	1024	16	24	128
ViT-H	1280	16	32	256

Appendix

Table 3: Encoder dimensions. Depth, width, and head count follow the standard ViT variants, while token grid and register count are identical across capacities.

fit	term	VIF
full grid	$\beta_{N}$	1.92
full grid	$\beta_{B}$	1.73
full grid	$\beta_{C}$	1.00
full grid	$\beta_{M}$	1.14
fixed sample	$\beta_{B}$	1.37
fixed sample	$\beta_{M}$	1.37

Appendix

Table 4: Variance inflation factors for the continuous predictors in the full-grid model. All VIFs are below

2

Figure 6: Simulated power for detecting a capacity-by-subject-count interaction

\gamma_{BP}

under E1. Power is shown as a function of the imposed interaction magnitude and residual variance. At the fitted residual variance, approximately

80\%

power is reached for an interaction of

0.05

logit units per doubling of subject count.

condition $\rho$	sections	A–P span (mm)	largest gap (mm)
all	467	180	1.0
every 10th	47	180	5.2
16 blocks	78	175	11.9
4 blocks	52	142	42.5
loc. posterior	97	37	0.5
loc. center	31	11	0.5

Appendix

Table 5: Geometry of the coverage conditions

\rho

in E4 at

B=13

, averaged across the 13 coronal subjects. All conditions use the same sample budget but differ in anatomical extent and spatial fragmentation.

Figure 7: Positive weight across E1. A : Total positive weight

M

remains approximately constant as subject count increases from one to 18. B : The fraction of

M

contributed by cross-subject pairs increases from zero to

94.2\%

over the same range. At fixed

N_{\mathrm{unique}}

, E1 therefore preserves the overall positive weight while changing the composition of contributing pairs.

$B$	sampling	gated	subsets	$N_{\mathrm{unique}}$	$M$	cross-subject (%)
1	every 10th	✗	5	$51\,709$	15.23	0.0
2	every 10th	✗	4	$108\,881$	16.31	49.0
4	every 10th	✗	5	$219\,266$	15.95	74.0
8	every 10th	✗	2	$438\,674$	15.85	86.9
9	every 10th	✗	1	$472\,504$	15.66	88.4
13	every 10th	✗	1	$701\,132$	16.15	92.0

Appendix

Table 6: Positive weight

M

and its cross-subject contribution for every pretraining configuration, averaged across subject subsets at each level.

M

is the covariate

M_{c}

used in the mixed-effects model and is computed from the configuration before training.

Figure 8: Macro-F1 on held-out sections from familiar subjects as a function of distance to the nearest section seen during pretraining in the

B=13

runs of E4. Each line corresponds to one coverage condition

\rho

. Macro-F1 decreases with distance from the nearest pretraining section in five of the six coverage conditions.

Figure 9: Training trajectories for five

B=18

full-coverage configurations from E5–E7 and their shared baseline. The top panel shows SpatialNCE loss, and the bottom panel shows the gradient of the smoothed loss trajectory. The loss continues to decrease at the doubled budget of

146\,400

steps. See table 8 for the corresponding configurations.

Figure 10: Macro-F1 for the

B=18

configurations of E1 and E2 under two alternative unseen-subject assignments. Absolute performance varies across assignments, while the direction and approximate magnitude of the difference between E1 and E2 are preserved.

Encoder	Reference	Macro-F1
Uniform random guess	—	$0.004$
DINOv2 (ViT-B/14)	Oquab et al. (2024)	$0.168$
GigaPath (ViT-G/16)	Xu et al. (2024)	$0.220$
UNI (ViT-L/16)	Chen et al. (2024)	$0.225$
Virchow2 (ViT-H/14)	Zimmermann et al. (2024)	$0.251$
ViT-B/32	ours	$0.300$

Appendix

Table 7: Reference encoders evaluated as frozen feature extractors using the same probe and unseen-subject evaluation protocol. Our rows correspond to the

B=18

configurations of E6 at each model capacity. The comparison provides an absolute performance reference across models with different architectures, pretraining data, and computational budgets. The chance row states where the macro-F1 scale begins under the same per-subject class denominator.

exp.	run	arch	$B$	sampling	$N_{\mathrm{unique}}$	steps	gated	fit
E1	vit_b_p32_b01_subset03_iso_unique	B	1	fixed budget	$449\,924$	$73\,200$	✗	✓
E1	vit_b_p32_b01_subset04_iso_unique	B	1	fixed budget	$450\,792$	$73\,200$	✗	✓
E1	vit_b_p32_b02_subset01_iso_unique	B	2	fixed budget	$449\,883$	$73\,200$	✗	✓
E1	vit_b_p32_b02_subset02_iso_unique	B	2	fixed budget	$450\,825$	$73\,200$	✗	✓
E1	vit_b_p32_b04_subset01_iso_unique	B	4	fixed budget	$450\,404$	$73\,200$	✗	✓
E1	vit_b_p32_b04_subset02_iso_unique	B	4	fixed budget	$449\,245$	$73\,200$	✗	✓

Appendix

Table 8: Inventory of all 93 pretraining runs, grouped by experiment. Steps gives the completed optimization budget. The seven runs at

146\,400

steps also contribute a retained checkpoint at half that budget. Gated marks the runs whose objective admits no cross-subject positives. Fit marks inclusion in the full-grid model. The four runs on alternative held-out triples are analyzed only in section D.5 .

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 813 papers

Sep 23, 2026

A Scaling Study for fMRI Foundation Models

Scaling laws have guided large-model development in computer vision and natural language processing, but the relationships among data, model size, and compute remain unclear for functional magnetic resonance imaging (fMRI) foundation models. Here, we conduct a controlled empirical study using pretraining data from more than 200 source datasets and over 10,000 GPU-hours of experiments. Holding the pretraining framework and downstream protocol fixed, we vary pretraining data size, model size, and training duration. Downstream performance generally improves with compute, yet models using similar compute can perform substantially differently. Additional pretraining data bring larger gains at larger model sizes, suggesting that data and model size should be scaled together. At matched compute, increasing pretraining data benefits more tasks than increasing model size, although the pattern varies across tasks. We then use in-distribution (ID) downstream performance to select the combination of pretraining data size, model size, and training duration at two fixed compute budgets. The resulting models are locked before out-of-distribution (OOD) evaluation. They achieve the highest average performance across the evaluated OOD tasks among the compared fMRI foundation models while using less pretraining compute. Overall, our results show that compute alone does not characterize fMRI scaling: performance depends on how pretraining data, model size, and training duration are combined.

Wenhao Ye, Xuanye Pan, Junfeng Xia +3

Southern University of Science and Technology · Shenzhen University · SLAI +1

Medical Imaging Foundation Models fMRI Foundation Models Neural Scaling Laws

Ask PDF

May 10, 2026

How Much is Brain Data Worth for Machine Learning?

If a person can solve a task, can measuring their brain make it easier to train a model to solve that task too? Recent NeuroAI work suggests that supplementing task training with neural recordings can modestly improve model performance and robustness. However, it is unclear when there should be a benefit from using neural data and how much benefit to expect. We formulate this question mathematically, and begin to address it theoretically using a simple, analytically tractable linear gaussian model of task targets and neural recordings. For a multimodal estimator trained on both brain data and task labels, we derive scaling laws for how performance scales with the numbers of brain and task samples. From these laws we derive relative value and exchange rates between brain samples and task samples, quantifying how much extra task samples neural data is worth as a function of task-brain alignment, neural and task noise, latent dimension, and brain data sample size. We also analyze test distribution shift, to identify conditions where brain-regularized learning can produce substantial robustness gains through learned invariances. Finally, under a fixed collection budget, we characterize the regimes in which brain data is worth collecting. Our results provide a foundation for understanding how valuable brain data could be for improving machine learning.

Lane Lewis, Zhixin Wang, David Schwab +1

Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA · Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · NSF AI Institute for Artificial and Natural Intelligence (ARNI) +2

Neural Scaling Laws Data-Efficient Learning

Ask PDF

Mar 13, 2026

Spatial Transcriptomics as Images for Large-Scale Pretraining

Spatial Transcriptomics (ST) profiles thousands of gene expression values at discrete spots with precise coordinates on tissue sections, preserving spatial context essential for clinical and pathological studies. With rising sequencing throughput and advancing platforms, the expanding data volumes motivate large-scale ST pretraining. However, the fundamental unit for pretraining, i.e., what constitutes a single training sample, remains ill-posed. Existing choices fall into two camps: (1) treating each spot as an independent sample, which discards spatial dependencies and collapses ST into single-cell transcriptomics; and (2) treating an entire slide as a single sample, which produces prohibitively large inputs and drastically fewer training examples, undermining effective pretraining. To address this gap, we propose treating spatial transcriptomics as croppable images. Specifically, we define a multi-channel image representation with fixed spatial size by cropping patches from raw slides, thereby preserving spatial context while substantially increasing the number of training samples. Along the channel dimension, we define gene subset selection rules to control input dimensionality and improve pretraining stability. Extensive experiments show that the proposed image-like dataset construction for ST pretraining consistently improves downstream performance, outperforming conventional pretraining schemes. Ablation studies verify that both spatial patching and channel design are necessary, establishing a unified, practical paradigm for organizing ST data and enabling large-scale pretraining.

Yishun Zhu, Jiaxin Qi, Jian Wang +2

Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of the Chinese Academy of Sciences, Zhejiang, China

Spatial Transcriptomics

Ask PDF

Abstract

Representation Learning

Explore similar work

A Scaling Study for fMRI Foundation Models

How Much is Brain Data Worth for Machine Learning?

Spatial Transcriptomics as Images for Large-Scale Pretraining