Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networks
Authors: Jaeyong Bae, Hawoong Jeong
Organizations: Department of Physics, Korea Advanced Institute of Science and Technology, Daejeon, Korea · Center of Complex Systems, Korea Advanced Institute of Science and Technology, Daejeon, Korea
Finite datasets can share the same size and low-order statistics while differing strongly in structural complexity. We connect this dataset complexity to loss-landscape geometry by pairing local label mixing across neighborhood scales with local entropy around trained neural-network solutions. Adapted from the Franz--Parisi construction in spin-glass theory, local entropy measures the effective volume of low-loss, solution-like parameter configurations at each distance from a reference. We estimate it in finite networks using adaptive sequential Monte Carlo. In a controlled synthetic sweep, greater dataset complexity produces a larger decrease in local entropy near the reference. Farther away, its radial derivative becomes weak and nearly common across conditions. Dataset complexity therefore changes where the effective solution volume contracts, rather than making it decrease uniformly faster. Experiments on real image data show the same qualitative trend, with label randomization further amplifying the effect. These results show that dataset structure shapes how low-loss neighborhoods are organized across finite distances from trained solutions.
Figures & tables
Figure 1: Schematic overview of our study. We examine how dataset complexity affects finite-distance loss geometry around trained neural-network solutions.
Figure 2: Full radial-shell local entropy in the controlled perceptron benchmark. The dashed black curve shows the independently calculated thermodynamic-limit RS result ΔΦRSshell(r) . Colored curves show the SMC estimates ΔΦNshell(r) for N∈{40,80,160,320} trainable weights. Shaded bands denote one standard error across datasets after averaging 10 references within each of 10 independent datasets. Each radius uses 215 particles.
Figure 3: Controlled organization of the synthetic labels. (a) Four representative observed configurations spanning the controlled sweep. The colored regions and black interfaces visualize the organization of the sampled labels. (b) Mean CMS at every βdata , with standard errors across 60 independent datasets.
Figure 4: Local entropy across the controlled synthetic sweep. (a) Centered local entropy ΔΦE(r) . More negative values correspond to a smaller effective solution volume relative to r0 . (b) Its radial derivative gE(r) . The dotted line marks r=1 . (c) Particle-weighted, sample-wise training accuracy among the configurations contributing to the r=1 shell, plotted against CMS . (d) Total radial variation ATV plotted against CMS . Curves show means at each βdata after averaging ten references within each of 60 independent datasets. Shaded bands and error bars denote standard errors across datasets. Color denotes CMS .
Figure 5: Dataset organization in the MNIST label-noise sweep. (a) Endpoint visualizations in shared UMAP coordinates for the clean endpoint ( η=0 ) and randomized endpoint ( η=0.5 ). Colors show the observed labels, and black contours summarize their organization in the embedding. UMAP is used only for visualization. (b) Mean CMS at each noise probability, with standard errors across ten paired datasets. The complexity measure is calculated in the standardized 100-dimensional input space.
Figure 6: Local entropy under MNIST label noise with fixed inputs. (a) Centered local entropy ΔΦE(r) . More negative values correspond to a smaller effective solution volume relative to r0 . (b) Its radial derivative gE(r) . The dotted line marks r=1 . (c) Particle-weighted, sample-wise training accuracy among the configurations contributing to the r=1 shell, plotted against CMS . (d) Total radial variation ATV plotted against CMS . Curves average ten retained references within each of ten paired datasets. Shaded bands and error bars denote standard errors across datasets. Color denotes CMS .
Figure 7: Dataset organization in the selected MNIST digit-pair tasks. (a) UMAP visualizations of the lower-complexity 0/1 and higher-complexity 4/9 endpoints. (b) Mean CMS for the pairs at ranks 1,5,9,…,45 after ranking all 45 tasks by their mean across ten balanced datasets per pair. Error bars denote standard errors across those datasets.
Figure 8: Local entropy across the selected MNIST digit-pair tasks. (a) Centered local entropy ΔΦE(r) . (b) Its radial derivative gE(r) . The dotted line marks r=1 . (c) Particle-weighted, sample-wise training accuracy among the configurations contributing to the r=1 shell, plotted against CMS . (d) Total radial variation ATV plotted against CMS . Curves average ten references within each of ten balanced datasets per task. Shaded bands and error bars denote standard errors across datasets, and color denotes mean CMS .
Figure 9: Dimensionless rescaled radial response and a schematic corridor interpretation. (a) The horizontal axis is shell radius r , and the vertical axis is h∗=(h−hmin)/(hmax−hmin) , with h=−gE/r . Each higher/lower pair uses joint extrema over the displayed r range and the same range for both standard-error bands. This scaling compares radial shape and turning location, not absolute response magnitude. Panels (b) and (c) illustrate the constraint-crowding hypothesis. The higher-complexity sketch places the main bottleneck closer to the reference and concentrates the loss of solution-like directions into a narrower radial interval.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
System
ρstep
ρpool
Min. Δt
MH sweeps
Synthetic DNN
0.75
0.75
10−4
1
MNIST label noise
0.95
0.50
10−4
2
MNIST digit pairs
0.95
0.50
10−4
2
Perceptron
0.85
0.85
5×10−5
2
Appendix
Table 1: Sampling controls used for the reported local-entropy estimates. Every row uses direct-transition testing, systematic resampling, and a vMF random-walk concentration κmove equal to 80 times the parameter dimension.
Modern deep neural networks often contain far more parameters than needed to fit their training data, yet they achieve impressive generalization. A common explanation for this success is the implicit bias of stochastic gradient descent (SGD). An alternative volume hypothesis posits that, within low training-loss regions, loss-landscape basins leading to strong generalization occupy much larger regions of weight space than basins that generalize poorly, and therefore SGD is simply more likely to land in the former. Recent experimental explorations of this idea present seemingly contradictory results. While in one set of experiments randomly sampling the network weights until achieving zero training error yielded poor generalization, molecular dynamics density estimates supported the volume hypothesis. We observe that these experiments were performed at different dataset size regimes, and explore an intermediate regime using the Replica Exchange Wang-Landau algorithm to estimate the joint density of states over training and test accuracies in binary networks. Across several architectures and datasets, we show that the generalization advantage of gradient learning over random sampling training generally diminishes as the training data size grows, suggesting a resolution of the paradox.
Ari Pakman, Lior Kreimer, Yakir Berchenko
Department of Industrial Engineering and Management, Ben-Gurion University of the Negev, Beer Sheva, Israel.
Deep neural networks are expressive enough to contain worst-case targets that can be evaluated in polynomial time but cannot be learned in polynomial time by gradient descent. For practical tasks they nonetheless learn well, raising the question of what non-generic structure of real-world targets enables this. Existing surrogate models cannot pose this question because they either lack hard-to-learn targets entirely (deep linear networks) or cannot evaluate such targets efficiently (kernel methods, infinite-width limits). We study tree tensor networks (TTNs), a model class that generalizes deep linear networks and Tucker decompositions. We show they embed arbitrary read-once Boolean formulas, and thus contain polynomial-size targets that cannot be learned by gradient descent in polynomial time under the same mechanism as neural networks. Despite this, we prove that their loss landscapes are conditionally benign for every realizable target: every local minimum that is minimum-norm is global. Thus, surprisingly, bad local minima are not what distinguishes between typical and worst-case problems in TTNs. Instead, learning difficulty in TTNs can arise from high-order degenerate saddle points, which we show are caused by rank-deficiency. This is explored through a case study of the parity function, illustrating the potential for TTNs to relate landscape geometry to computational hardness.
LayerNorm and RMSNorm impose fundamentally different geometric constraints on their outputs - and this difference has a precise, quantifiable consequence for model complexity. We prove that LayerNorm's mean-centering step, by confining data to a linear hyperplane (through the origin), reduces the Local Learning Coefficient (LLC) of the subsequent weight matrix by exactly m/2 (where m is its output dimension); RMSNorm's projection onto a sphere preserves the LLC entirely. This reduction is structurally guaranteed before any training begins, determined by data manifold geometry alone. The underlying condition is a geometric threshold: for the codimension-one manifolds we study, the LLC drop is binary -- any non-zero curvature, regardless of sign or magnitude, is sufficient to preserve the LLC, while only affinely flat manifolds cause the drop. At finite sample sizes this threshold acquires a smooth crossover whose width depends on how much of the data distribution actually experiences the curvature, not merely on whether curvature exists somewhere. We verify both predictions experimentally with controlled single-layer scaling experiments using the wrLLC framework. We further show that Softmax simplex data introduces a "smuggled bias" that activates the same m/2 LLC drop when paired with an explicit downstream bias, proved via the affine symmetry extension of the main theorem and confirmed empirically.