cs.LGOct 19, 2026
SaveDynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks
Organizations: Departamento de Estructura de la Materia, Física Térmica y Electrónica, Universidad Complutense de Madrid, 28040 Madrid, Spain · Grupo Interdisciplinar de Sistemas Complejos, Universidad Complutense de Madrid, 28040 Madrid, Spain
Abstract
We investigate whether cyclically varying the class composition of training batches can modify both the performance and the internal organization of neural networks. Motivated by nonequilibrium systems, in which cycling between related sub-goals can make switching between their solutions increasingly efficient, we introduce Dynamical Batching (DB), a sampling protocol that periodically emphasizes each class while leaving the loss and the optimizer unchanged, and compare it with Uniform Batching (UB), in which every mini-batch has equal class proportions. In the regime where standard training fails, DB changes the outcome: on a three-arm spiral, a capacity-limited MLP that plateaus far below perfect accuracy under UB fits the training data under DB. Tracking individual weights, we find that under DB the change between class-specialized states is carried by a smaller set of weights than under UB, and that this set is markedly more persistent from cycle to cycle, on every synthetic dataset we test and in a wider network that fits the data under both protocols. This organization is a signature of the protocol rather than a predictor of its benefit: it appears equally where DB leaves accuracy unchanged. On CIFAR-10, the same protocol modestly improves the test accuracy of a channel-narrowed ResNet-10, a standard residual architecture, and of a second capacity-limited convolutional network. These results show that how class emphasis is distributed in time can shape both the solution a network reaches and the internal structure of the network.
Figures & tables
Figure 1: Learning by Dynamical Batching (DB). (a) Class proportion , the fraction of each mini-batch drawn from class , over two training cycles ( ). At low amplitude (top, ) every batch still contains all three classes: the focal-class proportion rises from to , while the other two fall to . At high amplitude (bottom, ) the emphasized class dominates the batch, its proportion saturating near , while the other two nearly vanish. A cycle is consecutive periods, one sweep through the classes. (b) Schematic of the parameter space (illustrative, not data). Each class-specialized objective is satisfied in an extended, cone-shaped region (one cone per class, colored as in (a)), and the cones meet only near a common apex (Optimum, star). The shape only encodes that each class alone is satisfied in a large region of parameter space, while all classes together are satisfied only in a much smaller region. In the under-parametrized regime, Uniform Batching (UB, dotted gray) moves from initialization (circle) to a point that satisfies no class fully (Suboptimal, cross). DB (solid green) alternates between the cones, each period pulling the trajectory toward one class’s region before the next pulls it toward another.
Figure 2: Dynamical Batching (DB) on the three-arm spiral. (a) Accuracy gain in percentage points over the Uniform Batching (UB, ) run with the same period , across oscillation amplitude and period , for the small ( ) and large ( ) networks; training and test gain shown separately. Color scales differ between rows. Each cell averages 5 seeds; the yellow and green boxes mark the UB baseline ( ) and the best DB configuration, , used in (b) and in Sec. IV . For the small network, a broad region of positive gain reaches points in test accuracy, next to a failure region at large and small . The large network already reaches train and test accuracy under UB in this grid, and DB improves test accuracy slightly (up to points) in a narrow band at low amplitude, just above the failure region. (b) Decision boundaries for a single UB run ( ) and a single DB run at : the network is evaluated on a grid and shaded by predicted class, with training (circles) and test (crosses) samples colored by true label. UB produces a fragmented boundary that does not follow the arms (test accuracy ); DB recovers the spiral structure (test accuracy ).
Figure 3: Class switching concentrates on a small, persistent set of weights, and the distance between class states shrinks over training. Spiral, small network, , . One cycle is steps, giving cycles, of which the first are dynamical and the rest fall in the uniform final and are excluded from the analysis. Solid green: Dynamical Batching (DB); dashed amber: Uniform Batching (UB, ). (a) Switching-set size : the fraction of all weights needed to carry of the class-to-class variation. (b) Persistence of the switching set: the fraction of weights switching in either cycle that switch in both. (c) Switching distance (Eq. ( 4 )), the mean of over the three class pairs. (d) Full-dataset training accuracy (black, left axis) and training loss (coral red, right axis; note the different loss scales) over one focal-class period ( steps; the horizontal axis counts steps from the start of that period), early and late in one DB run. The dashed vertical line marks the middle of the period ( ), where the focal class is most emphasized. The step-like changes of the loss coincide with the steps at which the number of samples of each non-focal class per batch changes (Eq. ( 2 )). (e) Weight-level schematic (no data): early in training the class-to-class change is spread over many weights and redrawn every cycle; late in training it is carried by a small set of weights (highlighted) that is largely the same in every cycle, while the remaining scaffold barely oscillates. In (a)–(c), each point is the mean over 20 seeds and bands are s.d. across seeds.
| Method | Train acc. (%) | Test acc. (%) | Persistence | ||
|---|---|---|---|---|---|
| UB | |||||
| DB |
Table 1: Spiral, small network, , mean s.d. over 20 seeds. Accuracies are measured at the end of training (after the uniform final phase); switching measures at the last cycle of each protocol’s regime (the last dynamical cycle for DB, the last cycle for UB). : switching-set size; : persistence of the switching set; : switching distance (Eq. ( 4 )).
Figure 4: Accuracy change on a channel-narrowed ResNet-10. Gain in training (left) and test (right) accuracy, in percentage points, relative to the Uniform Batching (UB) run ( ) with the same period (UB averaged over all periods: train , test ), across amplitude and period , for a ResNet-10 with channel widths scaled by , trained on CIFAR-10. Each cell averages 5 seeds. The color scale is clipped at points: dark blue cells are runs that collapse, mostly to chance level. Axes are categorical; only a subset of the tested values is labeled.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Train acc. (%) | Test acc. (%) | Persistence | ||
|---|---|---|---|---|---|
| UB | |||||
| DB |
Table 2: Spiral, small network, , trained for steps (twice the budget used elsewhere), mean s.d. over 20 seeds. Columns as in Table 1 .
Figure 5: Training curves on the spiral. Full-dataset training (dark) and test (light) accuracy over training for UB (amber) and DB (green); lines are the mean and bands s.d. across 20 seeds. The final of training is uniform for both protocols. (a) Small network ( ), . After an initial rise, UB levels off, gaining less than points of training accuracy over the last two thirds of training, and ends at training and test accuracy. DB overtakes it after steps and ends at and . The fast oscillation of the DB curves follows the periodic class emphasis. (b) Large network ( ), . Both protocols fit the training set ( UB, DB), UB faster than DB; test accuracy ends at (UB) and (DB).
Figure 6: Grid search on the spiral at a larger batch size ( ). Accuracy gain in percentage points over the UB run ( ) with the same period , across oscillation amplitude and period , for the small ( ) and large ( ) networks; training and test gain are shown separately. As for in the main text, a broad region of positive gain unlocks the capacity-constrained network, bordered by a failure region at extreme amplitudes and short periods where the emphasis shifts faster than the network can consolidate. Each cell represents the average of 5 independent seeds.
Figure 7: The five synthetic datasets. Training samples (circles) and test samples (crosses), colored by class, for (a) Spiral, (b) Blobs, (c) Checkerboard, (d) Dartboard and (e) Rings, each with 3 classes and 100 training and 100 test points per class, all within the region . Each panel shows one realization (seed ) generated from the exact configuration used for training; every training seed draws a new realization.
| Dataset | Method | Train acc. (%) | Test acc. (%) | |||
|---|---|---|---|---|---|---|
| Spiral | UB | |||||
| DB | ||||||
| Blobs | UB | |||||
| DB | ||||||
| Checkerboard | UB | |||||
| DB |
Table 3: Accuracy and weight-level measures across five toy datasets, , mean s.d. over 20 seeds. Accuracies are measured at the end of training (after the uniform final phase); switching measures at the last cycle of each protocol’s regime. : switching-set size; : persistence of the switching set; : switching distance (Sec. IV ).
Figure 8: Additional measures on the spiral. Solid green: DB; dashed amber: UB; bands s.d. across 20 seeds. (a) Mean pairwise cosine between the per-class probe gradients , each computed on the training examples of a single class, with all three gradients evaluated at the same parameters (Appendix B.2 ), small network ( ). Under UB it sits at (dotted) throughout training, while training plateaus near (Fig. 5 a). Under DB it starts at and settles at after cycles: the class gradients remain strongly opposed but do not cancel. (b) The same for the large network ( ). UB again sits close to (between and ), although it fits the training set ( ): the arrangement is generic under UB, not a signature of failure. DB stays near for the first cycles and settles at after cycles. (c) Participation ratio of [ 41 ] , the effective number of weights carrying the displacement, small network. Under DB it falls from to , agreeing in direction with (Fig. 3 a); under UB it drops within the first cycles and then rises to , with large seed-to-seed variability, so the two protocols separate only after cycle – . (d) Switching-set size under DB (solid green) and UB (dashed amber) for energy thresholds of (thin, lower), (thick, main text) and (thin, upper). After the first few cycles, each DB curve lies below the UB curve of the same threshold, ending at , and against , and : DB needs fewer weights than UB to carry the same fraction of the class-to-class variation, whichever fraction is chosen.
| Method | Train acc. (%) | Test acc. (%) | Persistence | ||
|---|---|---|---|---|---|
| UB | |||||
| DB |
Table 4: Spiral, large network ( ), , mean s.d. over 20 seeds. Columns as in Table 1 .
Figure 9: Studying the structure of the network (analogous to Fig. 3 a–c) for the large network ( ), where both protocols fit the training set. , giving cycles, of which are dynamical. (a) Switching-set size : DB drops to within a few cycles and ends at ; UB stays at – and ends at . The concentration is stronger than in the small network (Fig. 3 a). (b) Persistence of the switching set (Jaccard overlap of consecutive cycles): DB starts below UB ( ) while the switching set is being formed, rises to by cycle and stays at – ; UB stays at – . (c) Switching distance: DB falls from to , about twentyfold, closely tracking the drift of ordinary training over the same windows (UB, from to ).
| Architecture | Fits train (UB)? | Base train | Base test | Best test | Train at best | Test gain | |
|---|---|---|---|---|---|---|---|
| ResNet-10 (width 0.15) (main text) | Yes | Yes ( ) | |||||
| Plain CNN + FC bottleneck (50) | No | Yes ( ) | |||||
| Myrtle5 (base width 64) | Yes | No ( ) | |||||
| ResNet-10 (width 0.5) | Yes | No ( ) |
Table 5: Architectures tested on CIFAR-10 and their response to DB. Test-accuracy gains beyond seed-to-seed noise appear only in the two capacity-limited architectures (top two rows). “Base” values are UB ( ) means over the cells of the column; “Best test” is the maximum over the DB configurations ( ) of the grid, and “Train at best” the training accuracy of that same configuration. Every configuration is run with seeds. Values are mean s.d. over seeds (accuracies in %); the test gain is given with the combined s.d. .
Figure 10: Grid searches for the additional CNN architectures. Gain in training (left) and test (right) accuracy, in percentage points, relative to the UB run ( ) with the same period , across amplitude and period , on CIFAR-10. Each cell averages seeds. The color scale is clipped at points; dark blue cells are runs that collapse. Axes are categorical, as in Fig. 4 . (a) Plain CNN with a -unit fully connected bottleneck (UB: train , test ): a broad region of positive test gain at intermediate to large amplitudes and periods of about a hundred steps or more, with reduced training accuracy. (b) Myrtle5 (UB: , ). (c) ResNet-10 at width (UB: , ). In (b) and (c) the test accuracy outside the failure region is flat to within seed-to-seed noise, and the training set is fitted wherever training does not collapse.
More in this topic
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 516 papers