Why Does Train-Validation Separation Emerge? Update-Pressure Density Dynamics in Pretrained Backbones

Yuchen Li; Mingyu Du; Zongqi Fan; Ken-Tye Yong; Nguyen H. Tran

cs.LGOct 1, 2026

Why Does Train-Validation Separation Emerge? Update-Pressure Density Dynamics in Pretrained Backbones

Authors: Yuchen Li, Mingyu Du, Zongqi Fan, Ken-Tye Yong, Nguyen H. Tran

Organizations: School of Computer Science, The University of Sydney, Sydney, Australia · School of Computer Science and Engineering, UNSW Sydney, Sydney, Australia

Abstract

Train-validation separation is the evolving difference between performance on observed training examples and a finite held-out validation set. We propose a dynamic structural account of how this gap develops during adaptation of pretrained models: continued fitting can shift update demand from broadly reusable support toward narrower support with weaker held-out transfer. A conditional local model links this shift to increasing heterogeneity in gradient allocation and train-validation separation. Fixed training probes make this structural evolution observable without validation examples entering the readouts; held-out performance is used separately to evaluate its relation to the gap. In a constructed hierarchy implemented with a residual multilayer perceptron (ResMLP), increasing the target share of example-private features from $p=.3$ to $.5$ to $.7$ , while preserving the relative mixture $1{:}2{:}3{:}4$ among the four shared feature levels, increases the final mean accuracy gap from $.185$ to $.331$ to $.527$ across five runs per condition. Masked-input losses measured separately on training and validation examples expose the corresponding transfer asymmetry. The natural language processing (NLP) analysis uses 10-epoch runs of RoBERTa, DeBERTa, and Qwen on six datasets (90 runs): the training-probe-weighted within-class and overall dispersion readouts each have positive raw and smoothed level correlations with the accuracy gap in all 90 runs. Raw changes paired at approximately one-epoch intervals remain positively associated in 86/90 and 87/90 runs, respectively. A 40-epoch ResNet-18 study tests both readouts on three vision datasets. Together, controlled simulation, NLP, and vision support the dynamic structural account across settings, with real-model evidence testing its observable predictions under the specified monitors.

Figures & tables

Figure 1: Conceptual illustration of support hierarchy and gradient-allocation dispersion. Panel A uses face recognition as an analogy. General topology is broadly shared support; community patterns are reused by subsets; private marks may fit one training image without transferring to validation. Panel B sketches the qualitative pattern probed by $D_{\mathrm{den},t}$ : overlapping allocations imply lower dispersion and heterogeneous allocations higher dispersion. These are schematic profiles, not measured trajectories or an assumed Gaussian model.

Figure 2: Schematic motivation for applying gradient magnitude before normalization; all distributions and curves are explanatory rather than measured trajectories. Panel A shows two label-conditioned modes with approximately antipodal centers in signed-gradient space, so strong conditional structure can cancel at the pooled signed center. Panel B maps sign-equivalent gradients to the same normalized magnitude-allocation representation, $q=|g|/\|g\|_{1}$ . Panel C shows a complementary trajectory-level case in which nearby checkpoints cross a local zero-gradient surface within the same activation region: the direction reverses while the magnitude allocation remains stable. Panel D shows that genuine redistribution across coordinates is not removed by the map and therefore remains visible in ideal pooled dispersion.

Figure 3: Schematic illustration of why the accuracy gap is the primary behavioral endpoint; the curves are explanatory rather than measured experimental trajectories. Panel A shows training accuracy continuing to improve after validation argmax predictions have largely saturated, producing train–validation separation. Panel B shows that validation cross-entropy can turn upward into a U-shaped trajectory over the same period because confidence and logit scale continue to change after most predicted classes are stable. The inset gives a separate single-example illustration, not a point on the aggregate curves: for $y=1$ , decreasing $p_{y}$ from $0.49$ to $0.01$ leaves the argmax prediction incorrect and its accuracy contribution unchanged, while increasing cross-entropy from approximately $0.71$ to $4.61$ .

Symbol	Meaning	Status
$g_{i}(t)$	Signed monitored per-sample gradient for probe $i$	Observed
$q_{i}(t),\bar{q}(t)$	Normalized magnitude allocation and its cross-probe mean	Observed
$D_{\mathrm{den}}^{\mathrm{ideal}}(t)$	Exact pooled dispersion of normalized probe allocations	Theoretical
$D_{\mathrm{den}}(t)$	Directly recorded overall dispersion readout	Observed
$A(t),B(t)$	Remaining demand addressable by shared and heterogeneous residual support	Latent
$u(t),v_{i}(t)$	Shared and probe-dependent residual pressure shapes	Latent

Table 1: Notation for the local support-narrowing dynamics.

Figure 4: Training accuracy, validation accuracy, and their difference for five ResMLP runs at each target private share. Curves are run means at the 25 recorded epoch endpoints.

Private $p$	Final accuracy gap	Final loss gap	$D_{\mathrm{within}}^{\mathrm{wt}}$ : raw / LOWESS (positive)	$D_{\mathrm{den}}$ : raw / LOWESS (positive)
.3	.185	1.398	.703 / .757 (5/5)	.911 / .951 (5/5)
.5	.331	2.251	.632 / .679 (5/5)	.777 / .812 (5/5)
.7	.527	3.878	.891 / .911 (5/5)	.972 / .984 (5/5)

Table 2: Controlled hierarchy, normalized readouts. Final accuracy and surrogate-loss gaps are five-run means at epoch 25. Correlations are equal-weight means of within-run Pearson

r

between each readout and the accuracy-gap level; entries give raw / LOWESS (

\mathrm{frac}=.28

) and raw-positive runs.

D_{\mathrm{within}}^{\mathrm{wt}}

is count weighted and

D_{\mathrm{den}}

is the direct overall readout.

Private $p$	$\Delta U_{\mathrm{within}}^{\mathrm{wt}}$ : mean / median	$\Delta U_{\mathrm{total}}$ : mean / median
.3	152.011 / 60.082	156.691 / 62.886
.5	302.045 / 120.888	312.150 / 126.111
.7	639.187 / 514.815	677.525 / 539.137

Table 3: Magnitude-sensitive unnormalized parameter-gradient companions measured on the same 25-epoch training runs. Entries are the mean / median epoch-25 minus epoch-1 increment across five runs; each increment is positive in all five runs at every private share. These amplitudes do not measure correlation with the accuracy gap.

Figure 5: Train–validation differences in private masked-loss share within the five-level composition. Rows correspond to $p=.3,.5,.7$ ; left and right panels show training and validation shares as 100% stacked areas. Comparing the purple private bands at the same epoch within each row reveals how their shares separate during training. Each layer averages raw masked losses across the five matching runs at an epoch, then divides by that split’s sum over the five levels. Table 4 quantifies the change in this private-share disparity.

Private $p$	Train private change	Validation private change	Increase in val–train disparity
.3	+15.13 pp	+22.25 pp	+7.12 pp
.5	+7.70 pp	+15.26 pp	+7.56 pp
.7	$-3.19$ pp	+13.10 pp	+16.28 pp

Table 4: Change in the validation–training private masked-loss-share disparity shown in Figure 5 , from epoch 1 to epoch 25. The first two result columns give each split’s private-share change; the last gives the increase in their disparity, computed as validation change minus training change before rounding. Values are percentage points (pp). The columns are rounded independently, so subtraction of displayed entries may differ by 0.01 pp.

Backbone	Readout	Raw $r$	LOWESS $r$	Joint positive
RoBERTa	$D_{\mathrm{within}}^{\mathrm{wt}}$	.785	.872	30/30
RoBERTa	direct $D_{\mathrm{den}}$	.879	.922	30/30
DeBERTa	$D_{\mathrm{within}}^{\mathrm{wt}}$	.789	.888	30/30
DeBERTa	direct $D_{\mathrm{den}}$	.894	.954	30/30
Qwen3-1.7B	$D_{\mathrm{within}}^{\mathrm{wt}}$	.836	.884	30/30
Qwen3-1.7B	direct $D_{\mathrm{den}}$	.820	.865	30/30

Table 5: Main 10-epoch NLP cohort. Mean within-run Pearson correlations with the overall accuracy-gap level, raw / LOWESS, and runs jointly positive under both views.

D_{\mathrm{within}}^{\mathrm{wt}}

uses training-probe class counts;

D_{\mathrm{den}}

is the directly recorded overall readout.

Backbone	Readout	Linear time partial	Raw epoch difference	LOWESS epoch difference
RoBERTa	$D_{\mathrm{within}}^{\mathrm{wt}}$	.674 (29/30)	.466 (29/30)	.661 (29/30)
	direct $D_{\mathrm{den}}$	.828 (30/30)	.649 (30/30)	.832 (30/30)
DeBERTa	$D_{\mathrm{within}}^{\mathrm{wt}}$	.738 (30/30)	.523 (29/30)	.769 (29/30)
	direct $D_{\mathrm{den}}$	.811 (30/30)	.622 (29/30)	.832 (30/30)
Qwen3-1.7B	$D_{\mathrm{within}}^{\mathrm{wt}}$	.791 (30/30)	.559 (28/30)	.733 (29/30)
	direct $D_{\mathrm{den}}$	.822 (30/30)	.592 (28/30)	.791 (30/30)

Table 6: Temporal checks in the main 10-epoch NLP cohort. Linear-time partial

r

correlates residuals after regressing each series on an intercept and linear epoch. Increment columns correlate matched approximately one-epoch differences; raw differences are the primary local check, while differenced LOWESS is a descriptive smooth view. Each entry is mean within-run

r

(positive runs).

Dataset	Readout	Raw $r$	LOWESS $r$	Joint positive
CIFAR-10	$D_{\mathrm{within}}^{\mathrm{wt}}$	.936	.964	5/5
CIFAR-10	direct $D_{\mathrm{den}}$	.890	.955	5/5
CIFAR-100	$D_{\mathrm{within}}^{\mathrm{wt}}$	.972	.982	5/5
CIFAR-100	direct $D_{\mathrm{den}}$	.918	.964	5/5
STL-10	$D_{\mathrm{within}}^{\mathrm{wt}}$	.865	.983	5/5
STL-10	direct $D_{\mathrm{den}}$	.743	.956	5/5

Table 7: ResNet-18 association with the overall accuracy gap over 40 epochs.

D_{\mathrm{within}}^{\mathrm{wt}}

uses all observed training-probe classes and direct

D_{\mathrm{den}}

is the recorded overall readout. Entries give raw / LOWESS Pearson

r

and runs jointly positive.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Dataset	$D_{0}$	$D_{1}$	$W$	direct $D_{\mathrm{den}}$
RoBERTa	BoolQ	$.763/.854$ (5/5)	$.736/.803$ (5/5)	$.859/.910$ (5/5)	$.912/.937$ (5/5)
RoBERTa	CoLA	$.530/.646$ (5/5)	$.537/.689$ (5/5)	$.759/.849$ (5/5)	$.850/.902$ (5/5)
RoBERTa	MRPC	$.610/.625$ (5/5)	$.453/.566$ (5/5)	$.620/.773$ (5/5)	$.854/.909$ (5/5)
RoBERTa	RTE	$.688/.797$ (5/5)	$.752/.859$ (5/5)	$.822/.900$ (5/5)	$.934/.963$ (5/5)
RoBERTa	SST-2	$.656/.801$ (5/5)	$.455/.628$ (5/5)	$.742/.822$ (5/5)	$.799/.851$ (5/5)
RoBERTa	WiC	$.802/.934$ (5/5)	$.742/.912$ (5/5)	$.906/.976$ (5/5)	$.923/.969$ (5/5)

Appendix

Table 8: 10-epoch NLP results by dataset. Each cell is mean raw/LOWESS Pearson

r

with the paired accuracy-gap level, followed by the number of five runs positive under both views.

W

is weighted by the effective training-probe class counts; the overall

D_{\mathrm{den}}

is the direct export.

Backbone	Dataset	$\Delta W$ vs. $\Delta G$	$\Delta D_{\mathrm{den}}$ vs. $\Delta G$	$D_{\mathrm{den}}$ vs. $G^{\mathrm{loss}}$	$\Delta D_{\mathrm{den}}$ vs. $\Delta G^{\mathrm{loss}}$
RoBERTa	BoolQ	$.552$ (5/5)	$.730$ (5/5)	$.934$ (5/5)	$.477$ (4/5)
RoBERTa	CoLA	$.221$ (5/5)	$.552$ (5/5)	$.946$ (5/5)	$.636$ (5/5)
RoBERTa	MRPC	$.244$ (4/5)	$.482$ (5/5)	$.929$ (5/5)	$.720$ (5/5)
RoBERTa	RTE	$.438$ (5/5)	$.720$ (5/5)	$.927$ (5/5)	$.585$ (5/5)
RoBERTa	SST-2	$.621$ (5/5)	$.674$ (5/5)	$.457$ (5/5)	$.182$ (5/5)
RoBERTa	WiC	$.722$ (5/5)	$.733$ (5/5)	$.876$ (5/5)	$.783$ (5/5)

Appendix

Table 9: Temporal and surrogate-loss checks for the 10-epoch NLP cohort, by dataset. Cells are mean within-run Pearson

r

(positive runs out of five). The first two columns correlate raw approximately one-epoch differences with the overall accuracy-gap difference; the last two correlate direct

D_{\mathrm{den}}

with the loss-gap level and its paired raw one-epoch difference.

$K$	$W$ raw	$W$ LOWESS	$D_{\mathrm{den}}$ raw	$D_{\mathrm{den}}$ LOWESS
2	.276 (9/15)	.318 (11/15)	.366 (11/15)	.425 (11/15)
4	.371 (10/15)	.421 (11/15)	.486 (11/15)	.570 (12/15)
6	.451 (12/15)	.541 (14/15)	.565 (14/15)	.674 (14/15)
8	.516 (15/15)	.639 (15/15)	.617 (14/15)	.726 (15/15)
10	.562 (15/15)	.690 (15/15)	.645 (15/15)	.753 (15/15)

Appendix

Table 10: Parameter-gradient monitor-depth sensitivity. Cells are mean within-run raw or LOWESS correlation with the overall accuracy gap (positive runs among 15).

Readout	Private $p$	$N=64$	$N=128$	$N=256$	$N=512$
$W$	.3	.698/.749	.732/.789	.730/.790	.734/.795
	.5	.669/.716	.632/.679	.626/.675	.615/.662
	.7	.893/.919	.891/.911	.873/.895	.872/.895
$D_{\mathrm{den}}$	.3	.911/.950	.894/.937	.907/.948	.905/.946
	.5	.795/.831	.777/.812	.784/.819	.776/.812
	.7	.970/.983	.972/.984	.970/.983	.968/.981

Appendix

Table 11: Training-probe-count sensitivity. Each cell is mean raw / LOWESS within-run correlation; all displayed raw and LOWESS cells have 5/5 positive runs.

W

is normalized count-weighted within-class dispersion;

D_{\mathrm{den}}

is the directly logged normalized overall readout.

Additional exclusion	Mean raw $r$	Mean LOWESS $r$
0%	.6447	.7528
1%	.6433	.7521
5%	.6445	.7557
10%	.6427	.7575

Appendix

Table 12: Parameter-monitor

K=10

overall association after additional within-checkpoint exclusion of the lowest monitored-mass probes. All 15 trajectories remain positive under raw and LOWESS analyses. The 0% row matches Table 10 .

Private share $p$	Run	$D_{\mathrm{within}}^{\mathrm{wt}}$	$D_{\mathrm{den}}$	$U_{\mathrm{within}}^{\mathrm{wt}}$	$U_{\mathrm{total}}$
.3	1	$1.892\mathrm{e}{-7}\!\to\!2.355\mathrm{e}{-6}$	$2.498\mathrm{e}{-7}\!\to\!4.472\mathrm{e}{-6}$	$1.76179\mathrm{e}{-5}\!\to\!423.104$	$4.12154\mathrm{e}{-5}\!\to\!432.453$
.3	2	$1.829\mathrm{e}{-7}\!\to\!2.362\mathrm{e}{-6}$	$3.402\mathrm{e}{-7}\!\to\!4.545\mathrm{e}{-6}$	$2.99457\mathrm{e}{-5}\!\to\!46.2702$	$1.42022\mathrm{e}{-4}\!\to\!48.6398$
.3	3	$2.063\mathrm{e}{-7}\!\to\!1.871\mathrm{e}{-6}$	$3.169\mathrm{e}{-7}\!\to\!3.735\mathrm{e}{-6}$	$4.42614\mathrm{e}{-5}\!\to\!5.57298$	$1.44605\mathrm{e}{-4}\!\to\!5.77365$
.3	4	$1.757\mathrm{e}{-7}\!\to\!2.369\mathrm{e}{-6}$	$3.113\mathrm{e}{-7}\!\to\!1.414\mathrm{e}{-5}$	$1.38323\mathrm{e}{-5}\!\to\!225.024$	$7.17385\mathrm{e}{-5}\!\to\!233.703$
.3	5	$1.878\mathrm{e}{-7}\!\to\!1.934\mathrm{e}{-6}$	$3.198\mathrm{e}{-7}\!\to\!3.786\mathrm{e}{-6}$	$2.68954\mathrm{e}{-5}\!\to\!60.0816$	$8.22350\mathrm{e}{-5}\!\to\!62.8865$
.5	1	$2.384\mathrm{e}{-7}\!\to\!2.753\mathrm{e}{-6}$	$2.870\mathrm{e}{-7}\!\to\!4.477\mathrm{e}{-6}$	$1.73819\mathrm{e}{-5}\!\to\!1132.05$	$3.50456\mathrm{e}{-5}\!\to\!1171.83$

Appendix

Table 13: Individual ResMLP dispersion levels in the controlled 25-epoch intervention. Each cell gives the raw epoch-1 value followed by the epoch-25 value. Normalized

D

records allocation dispersion; unnormalized

U

retains gradient magnitude. The run labels pair the same runs across all four columns.

Figure 6: Run-level checkpoint scatterplots of overall dispersion versus the accuracy gap. Rows are $p=.3,.5,.7$ ; columns are normalized direct $D_{\mathrm{den}}$ and the magnitude-sensitive unnormalized companion. Each color denotes one of the five matched training runs.

Figure 7: Run-level trajectories from the controlled hierarchy. Rows are $p=.3,.5,.7$ ; columns show the direct normalized overall readout and accuracy gap at the 25 epoch endpoints. Colors match the five training runs within each row.

Study / backbone	Depth rule	Monitored trainable region
ResMLP-24	$k=10$	Hybrid parameter/activation pressure at selected final-block shared-output interfaces
RoBERTa-base	$k=4$	Attention-output and feed-forward-output projections in the final encoder blocks
DeBERTa-base	$k=6$	Attention-output and feed-forward-output projections in the final encoder blocks
Qwen3-1.7B	$k=2$	Trainable LoRA parameters attached to attention-output and feed-forward-output projections in the final decoder blocks
ResNet-18	$k=8$	Parameter-gradient monitor used in the 40-epoch vision runs

Appendix

Table 14: Fixed monitoring depths. Here

k

counts final backbone blocks closest to the task output. Each NLP backbone retains its BoolQ-selected depth across all six datasets and all independent runs.

Item	Setting
Independent runs	Five per reported condition, except the one-run-per-share FP32 and BF16 precision controls
Gap target	$\mathrm{GapAcc}(t)=\mathrm{TrainAcc}(t)-\mathrm{ValAcc}(t)$
Temporal views	Raw records and LOWESS-smoothed records; 10-epoch NLP uses local-linear span 0.28 with no robust iterations
Primary structural readouts	Overall $D_{\mathrm{den},t}$ and training-probe-weighted within-class $D_{\mathrm{within},t}^{\mathrm{wt}}$
Probe counts	NLP: 128; main ResMLP: 128 (32 per class); ResMLP probe-count grid: 64/128/256/512; vision: 100 for CIFAR-10/STL-10 and 200 for CIFAR-100
Probe sampling	NLP uses stratified training probes; ResMLP and vision supplements use class-balanced training probes

Appendix

Table 15: Common analysis and logging settings used for the reported experiments.

Explore similar work

Sep 30, 2026cs.LG

Misalignment of Low-Loss Regions Causes Grokking

Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism remains unsettled. In this work, we develop an analysis framework based on mode connectivity and the geometry of low-loss regions. The framework predicts that the standard modular-arithmetic setting does not always produce grokking: under a symmetry-preserving train/validation split, we observe a stable anti-grokking case in which validation performance does not recover. This counterexample challenges several existing correlational explanations of grokking. More broadly, our analysis framework and results further suggest that grokking arises when the low-loss regions induced by the training and validation partitions are misaligned. Once these regions become well aligned, training hyperparameters alone cannot produce grokking and the observed dynamics collapse to either trainable or non-trainable behavior.

Yongding Tian, Zaid Al-Ars, Maksim Kitsak +1

Computer Engineering Lab Delft University of Technology Delft, NL · HDL TypeTech Delft, NL · Network and Architecture Service Delft University of Technology Delft, NL +1

Grokking Overfitting Symmetry Region

Ask PDF

May 12, 2026cs.LG

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabilities, we study how upstream training choices - that is, the manner in which a capability is acquired - shape how robustly that capability is retained. We investigate this question in a controlled three-stage language-model pipeline: pretraining, post-training to acquire a target capability, and downstream fine-tuning on a new objective. Across 135M and 1B models, two post-training domains, and two downstream fine-tuning tasks, we find that immediate post-training performance does not reliably predict retention after subsequent fine-tuning: training recipes that look equivalent immediately after post-training can retain the target capability very differently after subsequent fine-tuning. In particular, early exposure - mixing post-training data into pretraining - consistently improves the frontier between retained upstream performance and downstream performance. In compute-matched experiments, where the target data must be allocated between pretraining and post-training, we find that the optimum lies at neither extreme. Together with our other empirical and theoretical findings, this supports the view that post-training drives immediate specialization while early exposure improves robustness to later forgetting. Replay and dropout, typically used to mitigate forgetting as it occurs during fine-tuning, provide complementary gains to early exposure when applied during post-training. Our findings suggest that robustness to subsequent fine-tuning should be treated as a first-class objective of upstream training, addressed preventatively through choices like early exposure rather than reactively during fine-tuning itself.

Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer +2

Carnegie Mellon University

Model Fine-Tuning Retention Forgetting Early Layers Exposure

Ask PDF

Jun 14, 2026cs.LG

A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt

Test-time training (TTT) adapts a pretrained model to each prompt via parameter updates, improving accuracy under pretraining-to-test distribution shifts. Yet, its performance often suffers from instability and sensitivity to hyperparameters such as update steps and subspace. We explain this behavior through a decision-theoretic lens, treating TTT as implicit Bayesian inference in the kernel regime. Under a Gaussian process benchmark, we show that TTT reduces prediction error when updates are spectrally matched to the prompt's signal-to-noise ratio and aligned with query-relevant eigen-directions. This perspective underpins the following results: (1) we show when fixed update steps and subspaces fail under distribution shifts, motivating adaptive strategies; (2) we prove that selecting update steps via prompt evidence admits a PAC-Bayes guarantee against overfitting; and (3) we characterize the Bayes-optimal update subspace under a linear-Gaussian correction model, yielding a scoring rule for selecting Transformer blocks and heads. Our theory helps explain the empirical instability of TTT, taking a step toward principled guidance for when, how far, and which directions to adapt.

Tomoya Wakayama

RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan.

Stable Test-Time Adaptation Test-Time Training Distribution Shifts Gaussian Process

Ask PDF