Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks

Nicolás Ratier-Werbin; Samuel Lozano; Miguel Ruiz-García

cs.LGOct 19, 2026

Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks

Authors: Nicolás Ratier-Werbin, Samuel Lozano, Miguel Ruiz-García

Organizations: Departamento de Estructura de la Materia, Física Térmica y Electrónica, Universidad Complutense de Madrid, 28040 Madrid, Spain · Grupo Interdisciplinar de Sistemas Complejos, Universidad Complutense de Madrid, 28040 Madrid, Spain

Abstract

We investigate whether cyclically varying the class composition of training batches can modify both the performance and the internal organization of neural networks. Motivated by nonequilibrium systems, in which cycling between related sub-goals can make switching between their solutions increasingly efficient, we introduce Dynamical Batching (DB), a sampling protocol that periodically emphasizes each class while leaving the loss and the optimizer unchanged, and compare it with Uniform Batching (UB), in which every mini-batch has equal class proportions. In the regime where standard training fails, DB changes the outcome: on a three-arm spiral, a capacity-limited MLP that plateaus far below perfect accuracy under UB fits the training data under DB. Tracking individual weights, we find that under DB the change between class-specialized states is carried by a smaller set of weights than under UB, and that this set is markedly more persistent from cycle to cycle, on every synthetic dataset we test and in a wider network that fits the data under both protocols. This organization is a signature of the protocol rather than a predictor of its benefit: it appears equally where DB leaves accuracy unchanged. On CIFAR-10, the same protocol modestly improves the test accuracy of a channel-narrowed ResNet-10, a standard residual architecture, and of a second capacity-limited convolutional network. These results show that how class emphasis is distributed in time can shape both the solution a network reaches and the internal structure of the network.

Figures & tables

Figure 1: Learning by Dynamical Batching (DB). (a) Class proportion

p_{i}(t)

, the fraction of each mini-batch drawn from class

i

, over two training cycles (

C=3

). At low amplitude (top,

A=5

) every batch still contains all three classes: the focal-class proportion rises from

1/3

0.71

, while the other two fall to

0.14

. At high amplitude (bottom,

A=70

) the emphasized class dominates the batch, its proportion saturating near

1

, while the other two nearly vanish. A cycle is

C

consecutive periods, one sweep through the classes. (b) Schematic of the parameter space (illustrative, not data). Each class-specialized objective is satisfied in an extended, cone-shaped region (one cone per class, colored as in (a)), and the cones meet only near a common apex (Optimum, star). The shape only encodes that each class alone is satisfied in a large region of parameter space, while all classes together are satisfied only in a much smaller region. In the under-parametrized regime, Uniform Batching (UB, dotted gray) moves from initialization (circle) to a point that satisfies no class fully (Suboptimal, cross). DB (solid green) alternates between the cones, each period pulling the trajectory toward one class’s region before the next pulls it toward another.

Figure 2: Dynamical Batching (DB) on the three-arm spiral. (a) Accuracy gain in percentage points over the Uniform Batching (UB, $A=1$ ) run with the same period $T$ , across oscillation amplitude $A$ and period $T$ , for the small ( $H=50$ ) and large ( $H=500$ ) networks; training and test gain shown separately. Color scales differ between rows. Each cell averages 5 seeds; the yellow and green boxes mark the UB baseline ( $A=1$ ) and the best DB configuration, $(A,T)=(20,320)$ , used in (b) and in Sec. IV . For the small network, a broad region of positive gain reaches $+37.87$ points in test accuracy, next to a failure region at large $A$ and small $T$ . The large network already reaches $99.00\%$ train and $95.93\%$ test accuracy under UB in this grid, and DB improves test accuracy slightly (up to $+1.74$ points) in a narrow band at low amplitude, just above the failure region. (b) Decision boundaries for a single UB run ( $A=1$ ) and a single DB run at $(A,T)=(20,320)$ : the network is evaluated on a grid and shaded by predicted class, with training (circles) and test (crosses) samples colored by true label. UB produces a fragmented boundary that does not follow the arms (test accuracy $56.60\%$ ); DB recovers the spiral structure (test accuracy $94.47\%$ ).

Figure 3: Class switching concentrates on a small, persistent set of weights, and the distance between class states shrinks over training. Spiral, small network,

(A,T)=(20,320)

B=50

. One cycle is

3\times 320=960

steps, giving

130

cycles, of which the first

122

are dynamical and the rest fall in the uniform final

5\%

and are excluded from the analysis. Solid green: Dynamical Batching (DB); dashed amber: Uniform Batching (UB,

A=1

). (a) Switching-set size

|\mathcal{S}|/n

: the fraction of all weights needed to carry

80\%

of the class-to-class variation. (b) Persistence of the switching set: the fraction of weights switching in either cycle that switch in both. (c) Switching distance

d

(Eq. ( 4 )), the mean of

\lVert\boldsymbol{\theta}_{i}-\boldsymbol{\theta}_{j}\rVert_{2}

over the three class pairs. (d) Full-dataset training accuracy (black, left axis) and training loss (coral red, right axis; note the different loss scales) over one focal-class period (

T=320

steps; the horizontal axis counts steps from the start of that period), early and late in one DB run. The dashed vertical line marks the middle of the period (

t=T/2=160

), where the focal class is most emphasized. The step-like changes of the loss coincide with the steps at which the number of samples of each non-focal class per batch changes (Eq. ( 2 )). (e) Weight-level schematic (no data): early in training the class-to-class change is spread over many weights and redrawn every cycle; late in training it is carried by a small set of weights (highlighted) that is largely the same in every cycle, while the remaining scaffold barely oscillates. In (a)–(c), each point is the mean over 20 seeds and bands are

\pm 1

s.d. across seeds.

Method	Train acc. (%)	Test acc. (%)	$\|\mathcal{S}\|/n$	Persistence $J$	$d$
UB	$70.5\pm 8.9$	$59.2\pm 7.7$	$0.32\pm 0.04$	$0.33\pm 0.06$	$0.24\pm 0.19$
DB	$99.9\pm 0.3$	$95.1\pm 1.3$	$0.19\pm 0.06$	$0.59\pm 0.13$	$0.64\pm 0.14$

Table 1: Spiral, small network,

(A,T)=(20,320)

, mean

\pm

s.d. over 20 seeds. Accuracies are measured at the end of training (after the uniform final phase); switching measures at the last cycle of each protocol’s regime (the last dynamical cycle for DB, the last cycle for UB).

|\mathcal{S}|/n

: switching-set size;

J

: persistence of the switching set;

d

: switching distance (Eq. ( 4 )).

Figure 4: Accuracy change on a channel-narrowed ResNet-10. Gain in training (left) and test (right) accuracy, in percentage points, relative to the Uniform Batching (UB) run ( $A=1$ ) with the same period $T$ (UB averaged over all periods: train $100.00\%$ , test $79.84\%$ ), across amplitude $A$ and period $T$ , for a ResNet-10 with channel widths scaled by $0.15$ , trained on CIFAR-10. Each cell averages 5 seeds. The color scale is clipped at $-30$ points: dark blue cells are runs that collapse, mostly to chance level. Axes are categorical; only a subset of the tested values is labeled.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Train acc. (%)	Test acc. (%)	$\|\mathcal{S}\|/n$	Persistence $J$	$d$
UB	$75.3\pm 11.3$	$62.4\pm 10.8$	$0.29\pm 0.08$	$0.34\pm 0.06$	$0.18\pm 0.13$
DB	$100.0\pm 0.0$	$94.9\pm 1.4$	$0.15\pm 0.07$	$0.55\pm 0.19$	$0.38\pm 0.17$

Appendix

Table 2: Spiral, small network,

(A,T)=(20,320)

, trained for

250{,}000

steps (twice the budget used elsewhere), mean

\pm

s.d. over 20 seeds. Columns as in Table 1 .

Figure 5: Training curves on the spiral. Full-dataset training (dark) and test (light) accuracy over training for UB (amber) and DB (green); lines are the mean and bands

\pm 1

s.d. across 20 seeds. The final

5\%

of training is uniform for both protocols. (a) Small network (

H=50

(A,T)=(20,320)

. After an initial rise, UB levels off, gaining less than

10

points of training accuracy over the last two thirds of training, and ends at

70.5\pm 8.9\%

training and

59.2\pm 7.7\%

test accuracy. DB overtakes it after

\approx 20{,}000

steps and ends at

99.9\pm 0.3\%

and

95.1\pm 1.3\%

. The fast oscillation of the DB curves follows the periodic class emphasis. (b) Large network (

H=500

(A,T)=(10,150)

. Both protocols fit the training set (

99.9\pm 0.5\%

UB,

100.0\pm 0.0\%

DB), UB faster than DB; test accuracy ends at

96.6\pm 1.5\%

(UB) and

98.2\pm 1.2\%

(DB).

Figure 6: Grid search on the spiral at a larger batch size ( $B=200$ ). Accuracy gain in percentage points over the UB run ( $A=1$ ) with the same period $T$ , across oscillation amplitude $A$ and period $T$ , for the small ( $H=50$ ) and large ( $H=500$ ) networks; training and test gain are shown separately. As for $B=50$ in the main text, a broad region of positive gain unlocks the capacity-constrained network, bordered by a failure region at extreme amplitudes and short periods where the emphasis shifts faster than the network can consolidate. Each cell represents the average of 5 independent seeds.

Figure 7: The five synthetic datasets. Training samples (circles) and test samples (crosses), colored by class, for (a) Spiral, (b) Blobs, (c) Checkerboard, (d) Dartboard and (e) Rings, each with 3 classes and 100 training and 100 test points per class, all within the region

[-1,1]\times[-1,1]

. Each panel shows one realization (seed

0

) generated from the exact configuration used for training; every training seed draws a new realization.

Dataset	Method	Train acc. (%)	Test acc. (%)	$\|\mathcal{S}\|/n$	$J$	$d$
Spiral	UB	$70.5\pm 8.9$	$59.2\pm 7.7$	$0.32\pm 0.04$	$0.33\pm 0.06$	$0.24\pm 0.19$
	DB	$99.9\pm 0.3$	$95.1\pm 1.3$	$0.19\pm 0.06$	$0.59\pm 0.13$	$0.64\pm 0.14$
Blobs	UB	$99.8\pm 0.4$	$97.8\pm 1.1$	$0.20\pm 0.08$	$0.35\pm 0.10$	$0.19\pm 0.06$
	DB	$99.8\pm 0.4$	$96.7\pm 1.6$	$0.16\pm 0.10$	$0.71\pm 0.10$	$0.35\pm 0.08$
Checkerboard	UB	$97.8\pm 2.7$	$89.7\pm 4.1$	$0.26\pm 0.07$	$0.33\pm 0.06$	$0.17\pm 0.09$
	DB	$97.6\pm 1.4$	$90.9\pm 1.7$	$0.16\pm 0.08$	$0.73\pm 0.14$	$0.53\pm 0.12$

Appendix

Table 3: Accuracy and weight-level measures across five toy datasets,

(A,T)=(20,320)

, mean

\pm

s.d. over 20 seeds. Accuracies are measured at the end of training (after the uniform final phase); switching measures at the last cycle of each protocol’s regime.

|\mathcal{S}|/n

: switching-set size;

J

: persistence of the switching set;

d

: switching distance (Sec. IV ).

Figure 8: Additional measures on the spiral. Solid green: DB; dashed amber: UB; bands

\pm 1

s.d. across 20 seeds. (a) Mean pairwise cosine between the per-class probe gradients

\nabla\mathcal{L}_{k}

, each computed on the training examples of a single class, with all three gradients evaluated at the same parameters (Appendix B.2 ), small network (

H=50

). Under UB it sits at

-1/(C-1)=-0.5

(dotted) throughout training, while training plateaus near

71\%

(Fig. 5 a). Under DB it starts at

\approx-0.39

and settles at

\approx-0.45

after

\approx 40

cycles: the class gradients remain strongly opposed but do not cancel. (b) The same for the large network (

H=500

). UB again sits close to

-0.5

(between

-0.50

and

-0.48

), although it fits the training set (

\geq 99.8\%

): the arrangement is generic under UB, not a signature of failure. DB stays near

-0.40

for the first

\approx 25

cycles and settles at

\approx-0.45

after

\approx 50

cycles. (c) Participation ratio

\mathrm{PR}_{ij}=(\sum_{k}\Delta_{k}^{2})^{2}/\sum_{k}\Delta_{k}^{4}

\Delta=\boldsymbol{\theta}_{i}-\boldsymbol{\theta}_{j}

[ 41 ] , the effective number of weights carrying the displacement, small network. Under DB it falls from

90

28

, agreeing in direction with

|\mathcal{S}|/n

(Fig. 3 a); under UB it drops within the first cycles and then rises to

40

, with large seed-to-seed variability, so the two protocols separate only after cycle

75

–

80

. (d) Switching-set size

|\mathcal{S}|/n

under DB (solid green) and UB (dashed amber) for energy thresholds of

50\%

(thin, lower),

80\%

(thick, main text) and

90\%

(thin, upper). After the first few cycles, each DB curve lies below the UB curve of the same threshold, ending at

0.07

0.19

and

0.30

against

0.10

0.32

and

0.47

: DB needs fewer weights than UB to carry the same fraction of the class-to-class variation, whichever fraction is chosen.

Method	Train acc. (%)	Test acc. (%)	$\|\mathcal{S}\|/n$	Persistence $J$	$d$
UB	$99.9\pm 0.5$	$96.6\pm 1.5$	$0.15\pm 0.06$	$0.34\pm 0.14$	$0.13\pm 0.11$
DB	$100.0\pm 0.0$	$98.2\pm 1.2$	$0.03\pm 0.03$	$0.70\pm 0.20$	$0.18\pm 0.12$

Appendix

Table 4: Spiral, large network (

H=500

(A,T)=(10,150)

, mean

\pm

s.d. over 20 seeds. Columns as in Table 1 .

Figure 9: Studying the structure of the network (analogous to Fig. 3 a–c) for the large network (

H=500

), where both protocols fit the training set.

(A,T)=(10,150)

, giving

277

cycles, of which

264

are dynamical. (a) Switching-set size

|\mathcal{S}|/n

: DB drops to

0.06

within a few cycles and ends at

0.03

; UB stays at

0.15

–

0.25

and ends at

0.15

. The concentration is stronger than in the small network (Fig. 3 a). (b) Persistence of the switching set (Jaccard overlap of consecutive cycles): DB starts below UB (

0.1

) while the switching set is being formed, rises to

0.7

by cycle

\approx 35

and stays at

0.55

–

0.75

; UB stays at

0.30

–

0.34

. (c) Switching distance: DB falls from

3.6

0.18

, about twentyfold, closely tracking the drift of ordinary training over the same windows (UB, from

2.8

0.13

Architecture	Fits train (UB)?	Base train	Base test	Best test	$(A,T)$	Train at best	Test gain
ResNet-10 (width 0.15) (main text)	Yes	$100.00\pm 0.00$	$79.84\pm 0.49$	$81.27\pm 0.38$	$(125,200)$	$94.65\pm 0.58$	Yes ( $+1.43\pm 0.62$ )
Plain CNN + FC bottleneck (50)	No	$98.30\pm 0.20$	$75.97\pm 0.75$	$77.96\pm 0.60$	$(75,1500)$	$90.07\pm 0.35$	Yes ( $+2.00\pm 0.96$ )
Myrtle5 (base width 64)	Yes	$100.00\pm 0.00$	$85.10\pm 0.47$	$85.64\pm 0.59$	$(5,15)$	$100.00\pm 0.00$	No ( $+0.53\pm 0.75$ )
ResNet-10 (width 0.5)	Yes	$100.00\pm 0.00$	$85.98\pm 0.48$	$86.20\pm 0.07$	$(10,50)$	$100.00\pm 0.00$	No ( $+0.22\pm 0.48$ )

Appendix

Table 5: Architectures tested on CIFAR-10 and their response to DB. Test-accuracy gains beyond seed-to-seed noise appear only in the two capacity-limited architectures (top two rows). “Base” values are UB (

A=1

) means over the

15

cells of the

A=1

column; “Best test” is the maximum over the DB configurations (

A>1

) of the grid, and “Train at best” the training accuracy of that same configuration. Every configuration is run with

5

seeds. Values are mean

\pm

s.d. over seeds (accuracies in %); the test gain is given with the combined s.d.

\sqrt{s_{\mathrm{base}}^{2}+s_{\mathrm{best}}^{2}}

Panel of Figure 10 — Figure 10: Grid searches for the additional CNN architectures. Gain in training (left) and test (right) accuracy, in percentage points, relative to the UB run ( $A=1$ ) with the same period $T$ , across amplitude $A$ and period $T$ , on CIFAR-10. Each cell averages $5$ seeds. The color scale is clipped at $-30$ points; dark blue cells are runs that collapse. Axes are categorical, as in Fig. 4 . (a) Plain CNN with a $50$ -unit fully connected bottleneck (UB: train $98.30\%$ , test $75.97\%$ ): a broad region of positive test gain at intermediate to large amplitudes and periods of about a hundred steps or more, with reduced training accuracy. (b) Myrtle5 (UB: $100.00\%$ , $85.10\%$ ). (c) ResNet-10 at width $0.5$ (UB: $100.00\%$ , $85.98\%$ ). In (b) and (c) the test accuracy outside the failure region is flat to within seed-to-seed noise, and the training set is fitted wherever training does not collapse.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 516 papers

Aug 16, 2026

Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study

Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-horizon predictability as a measure of temporal redundancy: where, when, and under which training conditions recent updates contain information about near-future parameter motion. We combine three complementary probe families, displacement-direction, subspace-residual, and predictor-based probes, with convention-aware, null-calibrated group-level readouts, and apply them to multi-pass vision training on CIFAR and public Pythia pretraining checkpoints. Across both regimes, vector-like tensors such as normalization parameters and biases (auxiliary parameters) exhibit simpler short-horizon dynamics than matrix-like feature-transforming weights (bulk parameters), whose predictable behavior concentrates in localized, time-varying pockets. Agreement within and across probe families, and with independent trajectory diagnostics, indicates that these measurements capture intrinsic trajectory structure, while probe differences distinguish complementary forms of temporal organization. Controlled CIFAR comparisons further show that architecture and training recipe systematically modulate the measured structure. A Pythia-70M case study further exposes a sequence of role-, depth-, and scale-dependent events, including bulk ESA falling below the random sign-agreement level and the emergence and redistribution of predictable qkv pockets across layers. These results position short-horizon predictability as a retrospective, parameter-resolved diagnostic of training dynamics.

Fanqi Wang, Weisheng Tang, Hairong Qi

Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville

Neural Network Training Dynamics

Ask PDF

Jul 17, 2026

Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training

Common practice when training Convolutional Neural Networks (CNNs) is to use randomly shuffled mini-batches. This creates two limitations: slower convergence, and a diminishing learning signal, since many samples are quickly classified as easy during training. We address these inefficiencies with A*-Inspired Batch Selection (A*-BS), a lightweight, model-agnostic strategy that formulates mini-batch scheduling as a heuristic search problem. Each batch is treated as a node in a search space and ranked using an A*-like score combining a loss-based difficulty measure with a reuse penalty. This encourages informative gradient updates and batch diversity throughout training, without modifying network architectures or optimization algorithms, so it integrates seamlessly into existing pipelines. We evaluate A*-BS on the twelve 2D classification tasks of the MedMNIST-v2 benchmark, using a deliberately simple architecture of approximately 2.25x10^5 parameters, compared against the ResNet-18 and ResNet-50 baselines reported by the benchmark. On half of these tasks, the lightweight model with A*-BS reaches higher accuracy and AUC than both ResNet baselines, with relative gains of up to 15%. An ablation under identical architecture and hyperparameters shows A*-BS outperforms random batch shuffling on all twelve tasks. Wall-clock measurements further show the lightweight CNN with A*-BS trains substantially faster than ResNet-18 and ResNet-50 on identical hardware. These results indicate that intelligent batch ordering can partially compensate for reduced architectural complexity, offering a computationally efficient alternative to deeper models, with reliability reinforced by strong performance even against deeper, more sophisticated architectures.

Anxhelo Shehu, Enes Stastoli, Arben Cela

AI Laboratory, University Metropolitan Tirana, Tirana, Albania. · Laboratory of Images Signals and Intelligent Systems, ESIEE Paris, Paris, France.

Deep Learning Optimization Convolutional Neural Networks

Ask PDF

Oct 17, 2026

Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks

Recent work has shown that deep neural networks exhibit an implicit low-rank simplicity bias. In this paper, we investigate the questions that naturally arise from this observation. What are the temporal consequences of low-rank simplicity bias during optimization? How does representation geometry evolve throughout optimization rather than only at convergence? Do different neural architectures exhibit distinct geometric optimization behaviors? To answer these questions, we conduct a large-scale empirical study tracking representational geometry throughout training. We find that neural networks follow three recurring geometric trajectories: persistent high-rank, collapse-and-recovery, and persistent low-rank. We further identify Destructive Representation Collapse (DRC) as a pathological regime in which representations collapse to rank-one and fail to recover, leading to a sustained loss of representational capacity. While collapse-and-recovery is consistently associated with successful optimization in controlled settings, experiments on real-world models show that the relationship between geometric trajectories and optimization outcomes is configuration dependent. Overall, our findings suggest that representation geometry evolves dynamically during training and provide a new perspective on how deep neural networks learn.

Shivaprasad B. Gowda, Sparsh Mittal

Indian Institute of Information Technology, Nagpur · Indian Institute of Technology, Roorkee

Low-Rank Representation Learning Neural Representation Geometry Neural Network Training Dynamics Representation Learning Neural Network Optimization Representation Collapse

Ask PDF

Abstract

Neural Network Training Dynamics

Explore similar work

Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study

Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training

Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks