Is spectral predictability enough? Testing Ω-matching for synthetic pretraining under distribution shift

Edric Castel C. Hao; Juan Raphael S. Seña; Neon S. De Jesus

cs.LGOct 22, 2026

Is spectral predictability enough? Testing Ω-matching for synthetic pretraining under distribution shift

Authors: Edric Castel C. Hao, Juan Raphael S. Seña, Neon S. De Jesus

Organizations: Analog Devices, Inc.

Abstract

Synthetic pretraining under distribution shift requires deciding what synthetic data to use. A natural criterion is to match the spectral predictability $Ω$ of the source to the target, but is $Ω$ sufficient? We test this hypothesis, which we call predictability-aware pretraining, across three architectures (LSTM, PatchTST, TCN) and three seeds. Under ideal synthetic-to-synthetic conditions, $Ω$ -matched pretraining can outperform even an oracle baseline (+46.8% MSE reduction, LSTM), though the benefit is already architecture-dependent. On the real-world ETTm1 benchmark, however, this relationship does not hold: sine-plus-noise pretraining at matched $Ω$ yields only a marginal mean improvement +2.3% averaged over all three architectures, up to +65.5% on individual channels), whereas $Ω$ matched chaotic ODE trajectories systematically degrade performance -13.2%). The two sources agree on the direction of transfer for only 53% of configurations across all three architectures (Pearson r=-0.02), indicating that $Ω$ carries little information about which source will transfer beneficially. We further observe an architecture-dependent asymmetry in the response to the direction of the $Ω$ gap. We conclude that $Ω$ matching can help in controlled settings but is not sufficient to predict transfer across sources, and we discuss additional signal properties that a source-selection criterion may need to consider.

Figures & tables

Strategy	LSTM	PatchTST	TCN
random_synthetic	$+$ 46.7%	$-$ 3.4%	$+$ 6.4%
oracle	$+$ 16.0%	$-$ 10.8%	$-$ 6.4%
pred_aware	$+$ 46.8%	$+$ 1.3%	$-$ 2.5%

Table 1: Verification (synthetic

\to

synthetic). MSE reduction vs. no_aug at

\Delta\omega=+0.4

(down-gap

\omega_{\mathrm{pre}}0.7\!\to\!\omega_{\mathrm{tgt}}0.3

), mean over dataset sizes. Positive

=

improvement; bold

=

best per model.

Direction	LSTM	PatchTST	TCN
Down-gap ( $0.7\to 0.3$ )	$+46.8\%$	$+1.3\%$	$-2.5\%$
Up-gap ( $0.3\to 0.7$ )	$-65.7\%$	$+86.4\%$	$-1.7\%$

Table 2: Directional asymmetry at

|\Delta\omega|=0.4

. MSE reduction (%) vs. no_aug . Positive

=

improvement.

	$T$ =2000				$T$ =10000
Ch.	$\omega$ .3	$\omega$ .5	$\omega$ .7	$\omega$ .9	$\omega$ .3	$\omega$ .5	$\omega$ .7	$\omega$ .9
HUFL	1.059	1.247	0.988	1.155	1.096	0.947	0.819	0.762
HULL	0.977	1.050	1.047	1.012	0.905	0.924	0.930	0.991
LUFL	1.088	1.004	0.930	0.915	0.982	1.005	1.001	1.012
LULL	0.896	0.991	1.010	0.912	0.919	0.884	0.858	0.999
MUFL	0.595	1.058	0.912	1.398	0.938	1.003	0.531	0.637

Table 3: LSTM normalized MSE under sine-plus-noise pretraining ( pred_aware / no_aug ,

<1.0=

improvement, bold) by channel and pretrain

\omega

(seed-averaged).

	$T$ =2000				$T$ =10000
Ch.	$\omega$ .3	$\omega$ .5	$\omega$ .7	$\omega$ .9	$\omega$ .3	$\omega$ .5	$\omega$ .7	$\omega$ .9
HUFL	0.978	0.839	0.844	0.753	0.813	0.799	0.697	0.345
HULL	1.013	0.978	1.019	0.945	0.936	0.920	0.916	0.997
LUFL	0.992	1.066	1.034	0.994	0.915	0.923	0.883	0.920
LULL	0.981	0.981	0.942	0.951	0.919	0.946	1.004	1.020
MUFL	0.749	0.940	0.835	0.995	0.914	0.748	1.341	0.891

Table 4: PatchTST normalized MSE under sine-plus-noise pretraining ( pred_aware / no_aug ,

<1.0=

improvement, bold) by channel and pretrain

\omega

(seed-averaged).

	$T$ =2000				$T$ =10000
Ch.	$\omega$ .3	$\omega$ .5	$\omega$ .7	$\omega$ .9	$\omega$ .3	$\omega$ .5	$\omega$ .7	$\omega$ .9
HUFL	0.990	1.126	1.240	0.868	1.027	0.832	0.949	1.020
HULL	1.067	1.037	1.008	1.016	1.054	0.986	1.042	1.062
LUFL	0.993	1.037	1.058	1.082	1.039	1.047	1.028	1.057
LULL	1.067	1.025	1.040	1.059	1.017	1.005	1.009	1.035
MUFL	0.960	1.149	1.068	0.983	0.871	1.088	1.011	1.044

Table 5: TCN normalized MSE under sine-plus-noise pretraining ( pred_aware / no_aug ,

<1.0=

improvement, bold) by channel and pretrain

\omega

(seed-averaged).

	B1 (sine+noise)		B2 (dysts ODE)
Model	$T$ =2000	$T$ =10000	$T$ =2000	$T$ =10000
LSTM	$-$ 0.5	$+$ 7.0	$-$ 2.2	$-$ 18.8
PatchTST	$+$ 3.9	$+$ 8.0	$-$ 12.6	$-$ 16.7
TCN	$-$ 3.3	$-$ 1.2	$-$ 7.6	$-$ 21.5

Table 6: Mean improvement (%) vs. no_aug by model and

T

, seed-averaged. Positive

=

improvement. B1: sine-plus-noise. B2: dysts ODE.

Figure 1: Sine-plus-noise vs. dysts ODE per-configuration improvement at matched $\Omega$ ( $2\times 3$ grid over source/model). The near-random scatter ( $r=-0.02$ , $53\%$ directional agreement) shows $\Omega$ matching alone does not determine transfer quality.

Figure 2: Architecture-dependent response. Mean MSE improvement (%) vs. no_aug by model and dataset size $T$ for each synthetic source. The three architectures show three distinct patterns: LSTM and PatchTST benefit from spectrally simple sine-plus-noise (a), TCN stays roughly neutral, yet all three degrade under $\Omega$ -matched chaotic ODE data (b). Matching $\Omega$ does not equalize these outcomes.

Synthetic Data Pretraining

Synthetic data pretraining uses generated, simulated, or transformed examples to initialize models when real training data are scarce or costly.

See all 119 papers

Conditional Transfer from Controlled Pretraining Mixtures to Code
Oct 8, 2026Ohad RubinSynthetic Data Pretraining Data Mixture Optimization
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
May 11, 2026Xu Guo, Runyu Peng, Jian Tong +4Language Model Pretraining Synthetic Data Pretraining
Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute
Mix, Don't Pick: Why Synthetic Corpus Composition Matters for Time Series Foundation Model Pretraining
Jun 6, 2026Aaryan Nagpal, Debdeep Sanyal, Murari Mandal +2Synthetic Data Pretraining Time Series Forecasting
Birla AI Labs, Mumbai, India · KIIT, Bhubaneswar, India · BITS Pilani, Pilani, India

Oct 8, 2026J/K move · Enter open · S save

Conditional Transfer from Controlled Pretraining Mixtures to Code

Ohad Rubin

Synthetic Data Pretraining Data Mixture Optimization Synthetic Data Evaluation Code Generation

Open paper AskPDF arXiv

Synthetic tasks are increasingly used both as probes of language-model capability and as pretraining data. Both uses are often justified by loss reduction: falling loss is treated as informative, and faster loss reduction with more sampling as evidence that a task is worth sampling. We separate three signals. A task is diagnostic when its loss tracks global pretraining progress; it is teachable when its loss responds to its own token budget; and a data source transfers when including it improves a downstream target. We study controlled pretraining in which 70% of the corpus is fixed general Python and the remaining 30% is a simplex over three source families: OpenCodeInstruct, a curated suite of 12 code-adjacent synthetic tasks, and 15 literature-derived probe tasks. Across a task-budget sweep we detect teachability for 14 of 27 tasks, with a sharp asymmetry between the two synthetic families (10/12 curated versus 4/15 literature-derived). Teachability and downstream transfer give different rankings. On the mixture-simplex edge between the curated suite and OpenCodeInstruct, HumanEval pass@20 after a fixed fine-tuning stage rises from 15.9 at pure curated data to its highest observed value, 22.6, at a mixture that is 75% OpenCodeInstruct, then falls to 19.5 at pure OpenCodeInstruct. Curated synthetic data therefore has conditional value: it contributes as a limited share of a mixture that a target-aligned source still dominates. Finally, a loss-based adaptive scheduler exposes the mismatch between residual loss reducibility and downstream transfer. Across three 60k-step free-ratio runs, Ado drives the OpenCodeInstruct share below 5% within the first 5k steps and to 1.2--1.4% by the end of training, and underperforms its matched fixed-mixture controls by 2.4--11.0 percentage points. Optimizing near-term task-loss reduction moves the mixture away from the region that transfers.

Abstract

Synthetic Data Pretraining

Explore similar work

Conditional Transfer from Controlled Pretraining Mixtures to Code

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

Mix, Don't Pick: Why Synthetic Corpus Composition Matters for Time Series Foundation Model Pretraining