Neural Expansion: A Unified Mechanism for How Deep Neural Networks Generalize

Chashi Mahiul Islam; Samuel Jacob Chacko; Mao Nishino; Canlin Zhang; Xiuwen Liu

cs.LGOct 16, 2026

Neural Expansion: A Unified Mechanism for How Deep Neural Networks Generalize

Authors: Chashi Mahiul Islam, Samuel Jacob Chacko, Mao Nishino, Canlin Zhang, Xiuwen Liu

Organizations: Department of Computer Science, Florida State University · Department of Mathematics, Florida State University · Independent

Abstract

Deep neural networks demonstrate remarkable generalization despite being highly over-parameterized, challenging traditional statistical learning theories. While several theoretical paradigms such as margin-based bounds, PAC-Bayesian analyses, and neural tangent kernels provide global explanations, they often lack mechanistic insight into how generalization emerges from training dynamics. We introduce Neural Expansion, a unified geometric framework that characterizes generalization through the expansion of directionally-selective ray coverage in the input space. Using Generalization Intervals (GIs), we quantify direction-specific regions around training samples where the model's predictions remain stable. Our extensive empirical studies across diverse architectures and datasets reveal that training leads to progressive expansion of ray coverage, particularly along directions with low Jacobian singular values. This mechanism helps explain a large fraction of correctly classified test inputs, adversarial examples, and even mislabeled or out-of-distribution samples through the geometry of ray coverage rather than instance-level memorization. By connecting input-space curvature to predictive behavior, our framework provides a mechanistic foundation that complements existing global theories of deep learning.

Figures & tables

Figure 1: Evolution of ray coverage during training of an MNIST MLP over 10 epochs. Left: Mean positive GI growth between same-class training samples, with percentage of GIs exceeding 1.0. Middle: Average GI along RSV directions. Right: Average GI along random directions.

Panel of Figure 2 — Figure 2: (a) GI magnitudes across RSV directions in early- vs. late-stage models (epoch 1 vs. epoch 50). Training increases GIs selectively along high-index (low-curvature) directions. (b) Comparison of linear approximation quality along RSV directions before and after training.

Model	Condition	Test Acc. (%)	Mean GI	Ray Coverage (%)
MNIST MLP	Control	$87.38\pm 0.08$	$1.872\pm 0.045$	$78.33\pm 1.04$
	Expand	$89.04\pm 0.08$	$1.943\pm 0.054$	$81.17\pm 1.26$
	Restrict	$75.18\pm 1.18$	$1.498\pm 0.025$	$60.67\pm 0.76$
Fashion-MNIST CNN	Control	$85.27\pm 0.18$	$2.005\pm 0.017$	$73.83\pm 0.29$
	Expand	$86.37\pm 0.29$	$2.037\pm 0.011$	$74.33\pm 0.58$
	Restrict	$77.86\pm 1.29$	$1.492\pm 0.092$	$54.33\pm 3.21$

Table 1: Test accuracy, mean GI, and ray coverage under control, expansion, and restriction conditions. Values are reported as mean

\pm

standard deviation over three random seeds at the final training epoch.

Figure 4: Left: Random Label Learning with GI ray coverage for RANDOM-10 and RANDOM-100 models. The gray dashed line (as defined in Figure 3 ) indicates the 1% directional coverage threshold. Middle: LT2 Misclassification Distribution showing percentage distribution from true class frequency to predicted class frequency. Right: LT2 GI Coverage showing average percentage of GIs exceeding 1.0 across different frequency categories.

Figure 5: Adversarial Examples Analysis (misclassified samples indicate successful attacks; correct classifications indicate failures). The heatmap shows the percentage of adversarial examples with GI > 1 to training samples of the predicted class across different dataset-model combinations and attack methods (CW, IFGSM, PGD).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Epoch-wise accuracy curves across models and datasets for vanilla as well as LT1, LT2, Random-10 and Random-100 variations.

Dataset	Architecture	GPU(s)	Runtime (hh:mm)	Memory Used (GB)
MNIST (6000 pairs)	MLP	A5000	00:30	3.0
	CNN	A5000	00:55	5.0
	ViT	A5000	01:25	6.5
CIFAR-10 (2000 pairs)	ResNet20	A5000	00:40	4.2
CIFAR-10 (2000 pairs)	VGG16-BN	A5000	00:55	5.1
ImageNette (100 pairs)	VGG16	A5000	01:40	6.8

Appendix

Table 2: Compute resources and runtime details for test-to-train GI coverage experiments using a step-wise directional search.

Figure 7: Relative GI growth along RSV vs. random directions during training. Both direction types show similar growth patterns through epoch 3, after which RSV directions exhibit substantially accelerated expansion, reaching 1220% growth by epoch 10 compared to 940% for random directions.

Epoch	Batch	Correct					Incorrect
		Count	Mean GI	% GI > 1	Avg GI	Median GI	Count	Mean GI	% GI > 1	Avg GI	Median GI
0	0	50	1.1798	48.21	1.1638	0.41	50	1.1922	48.49	1.1816	0.49
0	200	50	1.4496	79.60	1.4708	1.38	50	1.5397	84.00	1.5635	1.48
0	399	50	1.3748	80.06	1.3850	1.21	50	1.3375	82.87	1.3553	1.28
1	399	50	1.4358	74.89	1.4523	1.23	50	1.2922	81.24	1.3065	1.30
3	200	50	1.6066	79.80	1.6202	1.36	50	1.2574	85.17	1.2656	1.20

Appendix

Table 3: Summary of GI statistics across training checkpoints for correctly and incorrectly classified training sample pairs. Mean GI is the arithmetic mean of all GI values including those at the boundary (

\mathrm{GI}=0

); Avg GI is the mean computed only over sample pairs with

\mathrm{GI}>0

(i.e., where a nonzero interval exists).

Model	Untrained, same	Untrained, diff.	Trained, same	Trained, diff.
MLP, MNIST	46.8%	30.6%	88.9%	1.2%
CNN, MNIST	35.6%	42.7%	96.3%	0.3%
CNN, Fashion-MNIST	32.3%	34.6%	88.5%	1.0%

Appendix

Table 4: Ray coverage (%) under a 2

\times

2 untrained/trained

\times

same/different-class design. Training substantially increases same-class coverage, and trained same-class coverage substantially exceeds trained different-class coverage, confirming coverage is class-discriminative rather than an artifact of the measurement.

Architecture	Factor	$n$ runs	Mean	Std	CV (%)	Min–max
MNIST MLP	Seed	5	88.91	0.07	0.1	88.82–89.02
MNIST MLP	Optimizer	3	91.20	1.64	1.8	89.04–93.03
MNIST CNN	Seed	5	96.86	0.11	0.1	96.74–97.04
MNIST CNN	Optimizer	3	98.52	1.22	1.2	96.79–99.43

Appendix

Table 5: Final-epoch ray coverage (%) is nearly seed-invariant and only mildly optimizer-sensitive, whereas substantial variability is observed among untrained baselines (13–99.96% MLP, 24–95% CNN).

Model	Median hop distance	Median tube radius	Median ratio (tube/hop)
MNIST CNN	17.9	23.1	1.39 $\times$
Fashion-MNIST CNN	16.2	15.4	1.01 $\times$

Appendix

Table 6: Orthogonal tube test: orthogonal stability radius around high-coverage rays, normalized by the ray’s own hop distance. A ratio

\geq 1

indicates the stable region is at least as wide as the ray is long, in every perpendicular direction tested.

Panel of Figure 8 — Figure 8: Predicted class trajectories along paths from training to test samples (Train $\to$ Test Class 6) with GI $>$ 1.0. Each subplot shows class-stable behavior across the entire interpolation path, indicating the test point lies within the ray coverage of the training sample.

Panel of Figure 9 — Figure 9: Predicted class trajectories along paths from training to test samples (Train $\to$ Test Class 6) with GI $<$ 1.0. Multiple class transitions are observed along the interpolation, indicating unstable or fragmented class regions.

Setting	Correctly Classified	Misclassified
RANDOM-10 (MNIST-MLP)	17%	24%
RANDOM-100 (MNIST-MLP)	18%	22%

Appendix

Table 7: Ray coverage percentages for correctly and incorrectly classified test samples under random label supervision.

Class	0	1	2	3	4	5	6	7	8	9	Ratio
LT1	4717	2866	2370	2015	1806	1590	1346	1034	886	730	6.5:1
Band	H	H	H	M	M	M	M	L	L	L
LT2	4716	2504	2026	1464	1096	761	383	218	93	60	78.6:1
Band	H	H	H	M	M	M	L	L	L	L

Appendix

Table 8: Class distributions in MNIST long-tailed variants. H=High, M=Medium, L=Low frequency bands based on 33rd and 66th percentiles.

Figure 10: Gradient & Image $\Delta$ Projections on Top-k Right Singular Vectors. Blue lines show the cumulative percentage of the loss gradient projected onto the top-k RSVs. Orange lines show the projection of the image difference vector between target and source images.

Figure 11: Box plot illustrating the GI ray coverage for out-of-distribution (OOD) samples from CIFAR-10, evaluated on an MLP trained on MNIST. The gray dashed line (as defined in Figure 3 ) indicates the 1% directional coverage threshold.

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 299 papers

Pointwise Generalization in Deep Neural Networks
May 18, 2026Shaojie Li, Yunbei XuNeural Network Generalization Representation Learning
National University of Singapore
Understanding Generalization through Decision Pattern Shift
May 13, 2026Huiqi Deng, Yibo Li, Quanshi Zhang +3Neural Network Generalization Neural Network Interpretability
Xi’an Jiaotong University · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods
Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Neural Network Generalization Gradient Descent Dynamics
Mathematical Institute for Machine Learning and Data Science, Catholic University of Eichstätt-Ingolstadt · Department of Computer Science, RPTU Kaiserslautern-Landau · Department of Mathematics, The University of Hong Kong +1

May 18, 2026J/K move · Enter open · S save

Pointwise Generalization in Deep Neural Networks

Shaojie Li, Yunbei Xu

National University of Singapore

Neural Network Generalization Representation Learning Generalization Bounds

Open paper AskPDF arXiv

We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, representation-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.

Abstract

Neural Network Generalization

Explore similar work

Pointwise Generalization in Deep Neural Networks

Understanding Generalization through Decision Pattern Shift

Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods