Structure, Dimensionality, and Distribution Shape - What Makes Pretraining Data Transferable, and Why the Input Pipeline Decides Which

Sangyoon Bae; Jiook Cha

cs.LGNov 3, 2026

Structure, Dimensionality, and Distribution Shape - What Makes Pretraining Data Transferable, and Why the Input Pipeline Decides Which

Authors: Sangyoon Bae, Jiook Cha

Organizations: Department of Psychology, Seoul National University, Seoul, 08826, South Korea

Abstract

What makes a pretraining corpus transferable? We describe a corpus by three properties: its structure, its dimensionality, and the distribution shape of each unit's values. We propose the Transfer-Invariance Principle: the same property can be irrelevant to transfer under one input transform and decisive under another, so the model's input transform, not the data modality, decides which properties can matter. To test it we build TransferForge, a generator that places a synthetic population at any prescribed structure, dimensionality, and distribution shape. KernelSynth, the synthetic prior of Chronos, cannot, because it draws independent series. When the model standardizes each input window (PatchTST on UCR), the distribution shape is inert and the structural setting governs. Synthetic pretraining then matches a real corpus of the same size and exceeds it once scaled. When the model encodes calcium fluorescence signals without normalization (POYO Plus), the shape becomes a governing constraint, and a shape-matched synthetic corpus replaces real pretraining with zero real neural activity. Flipping only the input normalization, with the model and data fixed, changes the effect of a shape manipulation by a factor of 5.3, so the pipeline, not the modality, sets the switch. The principle yields a recipe that chooses what to synthesize before any training. Pretraining data then becomes something to manufacture for the model, most valuably where real data is scarce, expensive, or private.

Figures & tables

Figure 1: The Transfer-Invariance Principle. A pretraining corpus is a point on three axes (top): two structural (structure, dimensionality) and the per-unit distribution shape . The model’s input pipeline sets which axes can govern transfer (middle). Under per-window normalization, synthetic data exceeds a real corpus once scaled (bottom left, PatchTST on UCR). Under unnormalized input, shape-matched synthetic data replaces real pretraining (bottom right, POYO+ on calcium imaging).

Modality	Structure (embeddedness)	Dimensionality (effective rank)	Tail (excess kurtosis)	SNR
			Distribution shape
fMRI (task)	0.54	2.6	$-0.4$	3.4
EEG	0.47	3.2	$0.1$	3.7
fMRI (resting)	0.40	4.3	$-0.1$	3.9
weather	0.32	31	$0.4$	3.5
C. elegans	0.28	7.0	$-0.1$	6.5

Table 1: The three axes measured across modalities. Tail and SNR are computed per unit and summarized by the median. They are defined only for time-ordered data (vision and language: “–”).

Synthetic
Pretraining corpus	Classification accuracy
Ours ( $\sim$ 10k windows)	0.842 $\pm$ 0.006
KernelSynth ( $\sim$ 10k windows)	0.831 $\pm$ 0.008
Ours ( $1776$ windows)	0.813 $\pm$ 0.014
KernelSynth ( $1776$ windows)	0.808 $\pm$ 0.003
Real

Table 2: UCR classification accuracy after pretraining. PatchTST is pretrained on each corpus, then a frozen probe is trained on each UCR dataset; mean

\pm

SD over 3 seeds (10 for the

\sim

10k synthetic rows), averaged over 22 UCR datasets. Ours is TransferForge at the interior optimum (

n_{\text{roots}}=4

). Rows with the same window count pretrain for the same number of steps;

1776

is the size of the real corpus.

Figure 2: UCR classification accuracy vs. synthetic corpus size. PatchTST, frozen probe, at the interior optimum ( $n_{\text{roots}}=4$ ); mean $\pm$ SD over 3 seeds, averaged over 22 UCR datasets. Blue: our corpus. Orange: KernelSynth at the same sizes and seeds. Dashed: the fixed real corpus ( $1776$ windows). Dotted: random init.

Synthetic only (0 real traces)
Pretraining corpus	Decoding accuracy
Calcium-shaped synthetic	0.407 $\pm$ 0.015
Gaussian synthetic	0.353 $\pm$ 0.105
Synthetic $+$ scarce real
Calcium-shaped synthetic $+$ scarce real	0.343 $\pm$ 0.052
Gaussian synthetic $+$ scarce real	0.220 $\pm$ 0.106

Table 3: Drifting-gratings decoding accuracy after pretraining. POYO+, 15-epoch full finetune; held-out test accuracy, mean

\pm

SD. Chance

=0.125

. Rows are over 3 seeds, except the three rows that use scarce real data (5 seeds each, including chance-stalled runs, which widens their spread).

Figure 3: Calcium drifting-gratings decoding accuracy (chance

0.125

) as one property is varied. (a) structural setting (

n_{\text{roots}}

), (b) window effective rank of the pretraining pool (real sessions span

2.1

–

16.8

, 5th–95th percentile), (c) per-unit distribution shape match (gauss/shape/SNR/both). Held-out test accuracy. The distribution shape is re-matched in (a) and (b). (a) is single-seed at a 5-epoch budget, so its levels are not comparable to the other panels; (b) and (c) show mean

\pm

SD over 3 seeds after 10-epoch pretraining; drifting-gratings finetuning ran 15 epochs, except runs that hit the time limit and were rerun at 12 (7 of the 12 runs in (b); the seed-43 and seed-44 runs of shape and SNR in (c)). The dotted line is the 15-epoch no-pretraining level.

Figure 4: Distribution-shape effect with and without per-window normalization. Each panel plots the transfer change from the Gaussian anchor, oriented so down

=

shape hurts . The grey band is within seed noise. (a) PatchTST, per-window normalization on vs. replaced by a corpus-wide scale (10 seeds). (b) Chronos with its native tokenizer, no intervention (3 seeds). (c) Chronos, an SNR sweep with tail shape held Gaussian (3 seeds). Panels use different tasks and metrics.

Modality	Typical input transform	Tail	Candidate governing axis
EEG, fMRI, weather	per-window normalization	$\approx 0$	structure
spikes	unnormalized counts	$3.7$	distribution shape (weaker)
calcium	unnormalized $\Delta F/F$	$12.0$	distribution shape (shown here)

Table 4: Candidate governing axis. Tail is the excess kurtosis of Table 1 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Dataset	Unit	Units	Length
fMRI (task)	HCP EMOTION task ( Van Essen et al., 2013 )	voxel	$24\times 24$ subj.	88 TRs
EEG	TUH abnormal EEG (TUAB) ( Obeid and Picone, 2016 )	channel	$40\times 23$ windows	2000 samples
fMRI (resting)	HCP S1200 resting, 360 ROIs ( Van Essen et al., 2013 )	ROI	$24\times 40$ subj.	1200 TRs
weather	WEATHER-5K, hourly stations ( Han et al., 2024 )	station $\times$ variable	2500	8760 h
C. elegans	DANDI:000565, whole-brain calcium ( Dunn et al., 2024 )	neuron	431 (4 sessions)	1500 frames
vision	ImageNet-1k validation, $16\times 16$ grayscale ( Deng et al., 2009 )	pixel	256	–

Appendix

Table 5: Data behind Table 1 . Units and lengths are those used for tail and SNR. For spikes and calcium, structure and dimensionality are computed on larger session sets (

24

IBL sessions;

432

Allen sessions).

Figure 5: Matching the calcium distribution shape. Left: pooled

\Delta F/F

value distribution (log scale); real and calcium-transformed synthetic share the heavy sparse-spiky tail, while Gaussian synthetic does not. Right: per-unit summary statistics (skew, excess kurtosis, SNR); calcium synthetic is close to the calibration session on all three, though heavier-tailed, while Gaussian synthetic matches none.

Figure 6: Calcium sweeps (drifting-gratings decoding accuracy, chance

0.125

). (A) Decoding accuracy vs.

n_{\text{roots}}

(5-epoch, single seed). (B) coupling range of the generator vs. the built corpus after the calcium distribution-shape transform. (C) Decoding accuracy as the per-unit distribution match is varied (gauss / shape-only / SNR-only / both

=

calcium; mean

\pm

SD over 3 seeds, held-out test accuracy).

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 434 papers

Is spectral predictability enough? Testing Ω-matching for synthetic pretraining under distribution shift
Oct 22, 2026Edric Castel C. Hao, Juan Raphael S. Seña, Neon S. De JesusSynthetic Data Pretraining Time Series Forecasting
Analog Devices, Inc.
Conditional Transfer from Controlled Pretraining Mixtures to Code
Oct 8, 2026Ohad RubinSynthetic Data Pretraining Data Mixture Optimization
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
May 11, 2026Xu Guo, Runyu Peng, Jian Tong +4Language Model Pretraining Synthetic Data Pretraining
Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute

Oct 22, 2026J/K move · Enter open · S save

Is spectral predictability enough? Testing Ω-matching for synthetic pretraining under distribution shift

Edric Castel C. Hao, Juan Raphael S. Seña, Neon S. De Jesus

Analog Devices, Inc.

Synthetic Data Pretraining Time Series Forecasting Synthetic-to-Real Domain Adaptation

Open paper AskPDF arXiv

Synthetic pretraining under distribution shift requires deciding what synthetic data to use. A natural criterion is to match the spectral predictability

Ω

of the source to the target, but is

Ω

sufficient? We test this hypothesis, which we call predictability-aware pretraining, across three architectures (LSTM, PatchTST, TCN) and three seeds. Under ideal synthetic-to-synthetic conditions,

Ω

-matched pretraining can outperform even an oracle baseline (+46.8% MSE reduction, LSTM), though the benefit is already architecture-dependent. On the real-world ETTm1 benchmark, however, this relationship does not hold: sine-plus-noise pretraining at matched

Ω

yields only a marginal mean improvement +2.3% averaged over all three architectures, up to +65.5% on individual channels), whereas

Ω

matched chaotic ODE trajectories systematically degrade performance -13.2%). The two sources agree on the direction of transfer for only 53% of configurations across all three architectures (Pearson r=-0.02), indicating that

Ω

carries little information about which source will transfer beneficially. We further observe an architecture-dependent asymmetry in the response to the direction of the

Ω

gap. We conclude that

Ω

matching can help in controlled settings but is not sufficient to predict transfer across sources, and we discuss additional signal properties that a source-selection criterion may need to consider.

Abstract

Transfer Learning

Explore similar work

Is spectral predictability enough? Testing Ω-matching for synthetic pretraining under distribution shift

Conditional Transfer from Controlled Pretraining Mixtures to Code

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data