stat.MLFeb 18, 2026

Conjugate Learning Theory: Uncovering the Mechanisms of Trainability and Generalization in Deep Neural Networks

Authors: Binchuan Qi

Organizations: Tongji University, Siping Street, 200092 Shanghai, China · Zhejiang Yuying College of Vocational Technology, Sihao Street, Hangzhou, 310018, Zhejiang, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In this work, we propose a notion of practical learnability grounded in finite sample settings, and develop a conjugate learning theoretical framework based on convex conjugate duality to characterize this learnability property. Building on this foundation, we demonstrate that training deep neural networks (DNNs) with mini-batch stochastic gradient descent (SGD) achieves global optima of empirical risk by jointly controlling the extreme eigenvalues of a structure matrix and the gradient energy, and we establish a corresponding convergence theorem. We further elucidate the impact of batch size and model architecture (including depth, parameter count, sparsity, skip connections, and other characteristics) on non-convex optimization. Additionally, we derive a model-agnostic lower bound for the achievable empirical risk, theoretically demonstrating that data determines the fundamental limit of trainability. On the generalization front, we derive deterministic and probabilistic bounds on generalization error based on generalized conditional entropy measures. The former explicitly delineates the range of generalization error, while the latter characterizes the distribution of generalization error relative to the deterministic bounds under independent and identically distributed (i.i.d.) sampling conditions. Furthermore, these bounds explicitly quantify the influence of three key factors: (i) information loss induced by irreversibility in the model, (ii) the maximum attainable loss value, and (iii) the generalized conditional entropy of features with respect to labels. Moreover, they offer a unified theoretical lens for understanding the roles of regularization, irreversible transformations, and network depth in shaping the generalization behavior of deep neural networks. Extensive experiments validate all theoretical predictions, confirming the framework's correctness and consistency.

Figures & tables

Figure 1: Schematic illustration of the conjugate learning framework. The diagram outlines the complete processing pipeline from raw input to learning target approximation, emphasizing the interplay among model output, conjugate transformation, and distance measurement.
Figure 2: Schematic illustration of the mapping between gradient energy and the global optimum of the empirical risk. The top subplot shows the gradient energy as a function of the model parameters θ\theta , with blue dots indicating points where the gradient energy is zero. The bottom subplot depicts the corresponding empirical risk landscape, with its global minimum marked by a pentagram. For positive definite structure matrices, zero gradient energy points coincide with the global minimum of the empirical risk, validating gradient energy minimization as a proxy for empirical risk reduction.
Figure 3: Custom-designed model architectures and configuration parameters. Gray blocks represent components where the number of repetitions can be adjusted via the parameter ndn_{d} , and model width can be tuned via the parameter nwn_{w} . Model B is a modified variant of Model A with additional skip connections. The symbol II denotes an identity transformation, which preserves the dimensionality of feature maps in skip connection pathways.
ModelLossDatasetlr
LeNetsoftmax cross entropymini CIFAR-100.005
 mini CIFAR-1000.01
 CIFAR-100.001
 CIFAR-1000.002
 MSEmini CIFAR-100.1
 mini CIFAR-1001
Table 1: Training hyperparameters for different models, losses, and datasets.
Figure 6: Training dynamics of LeNet across datasets and loss functions. The first row shows the evolution of standardized empirical risk and its upper/lower bounds during training. The second row depicts the changes in key metric terms over training iterations. The third row presents dynamic Pearson correlation coefficients between different terms.
Figure 9: Training dynamics of ResNet18 across datasets and loss functions. The first row demonstrates the evolution of standardized empirical risk and its upper/lower bounds during training. The second row shows the progression of key metrics throughout training. The third row plots dynamic Pearson correlation coefficients between different metrics over training iterations.
Figure 12: Training dynamics of ViT across datasets and loss functions. The first row exhibits the evolution of standardized empirical risk and its upper/lower bounds during training. The second row illustrates the variation of key metric terms over training iterations. The third row shows dynamic Pearson correlation coefficients between different terms throughout training.
Figure 15: Relationship between the structure matrix and model depth/width at random initialization. ∥Ax∥F\|A_{x}\|_{F} denotes the Frobenius norm of the structure matrix AxA_{x} , and DiagNorm(Ax)\mathrm{DiagNorm}(A_{x}) represents the norm of the diagonal vector of AxA_{x} . The quantity log⁡2∥Ax∥F−log⁡2DiagNorm(Ax)\log_{2}\|A_{x}\|_{F}-\log_{2}\mathrm{DiagNorm}(A_{x}) quantifies the diagonal dominance of AxA_{x} : a value of 0 indicates that AxA_{x} is a diagonal matrix.
More in this topic

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 303 papers

Explore similar work

CardsList
Aug 10, 2026

Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training

Stochastic gradient descent (SGD) and its variants train deep neural networks (DNNs) effectively in practice, yet classical optimization theory does not fully explain this success: conventional analyses rest on differentiability, convexity, and smoothness, assumptions that DNN objectives routinely violate. We close this gap by generalizing convexity and smoothness, through Legendre functions and convex conjugation, into a unified optimization framework for DNN training. We introduce H(ψ)\mathcal{H}(ψ)-convexity and H(Ψ)\mathcal{H}(Ψ)-smoothness, a single formalism that encompasses convex and non-convex as well as smooth and non-smooth objectives and reveals a natural duality between generalized convexity and smoothness. Building on these notions, we derive generalized gradient descent (GD) and generalized SGD via convex conjugation. We prove that generalized GD attains an optimal learning rate of exactly 11, and we establish rigorous gradient-energy-based convergence rates for both optimizers. We further recast DNN training as a composite optimization problem and show that its convergence hinges on jointly reducing the gradient energy and controlling the extreme eigenvalues of the structure matrix; a model capacity risk quantifies the residual error that no optimizer can remove. Experiments on six datasets (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, TREC, and SST-2) and on convolutional, residual, and Transformer architectures, with five loss functions, are consistent with the predicted training dynamics.

Binchuan Qi

College of Electronics and Information Engineering Tongji University Shanghai, 200092, China · Sihao Street Zhejiang Yuying College of Vocational Technology Zhejiang, 310018, China

Gradient DescentNeural Network OptimizationHypergradient OptimizationStochastic Gradient Descent
AskPDF
Jun 4, 2026

Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks

Recent progress has been made in understanding the statistical generalization performance of gradient descent methods for overparameterized neural networks within the neural tangent kernel (NTK) regime. However, most of the existing work on regression problems is limited to shallow network architectures, leaving a notable gap in the theory of deep neural networks. This paper addresses this gap by presenting a comprehensive generalization analysis for deep ReLU networks trained using gradient descent (GD) and stochastic gradient descent (SGD). Specifically, we establish the first known minimax-optimal rates of excess population risk for both GD and SGD with deep ReLU networks, under the assumption that the network width scales polynomially with respect to the network depth and training sample size. Our results demonstrate that with sufficient width, gradient descent methods for deep ReLU networks can achieve optimal generalization rates on par with kernel methods.

Junyu Zhou, Puyu Wang, Yunwen Lei +2

Mathematical Institute for Machine Learning and Data Science, KU Eichstätt-Ingolstadt, Ingolstadt, Germany · Department of Computer Science, RPTU Kaiserslautern-Landau, Kaiserslautern, Germany · Department of Mathematics, University of Hong Kong, Hong Kong, China +1

Neural Network GeneralizationReLU Neural NetworksGradient DescentStochastic Gradient Descent
AskPDF
May 18, 2026

Pointwise Generalization in Deep Neural Networks

We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, representation-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.

Shaojie Li, Yunbei Xu

National University of Singapore

Neural Network GeneralizationRepresentation LearningGeneralization Bounds
AskPDF