stat.MLNov 14, 2026

Generalization Bounds for Neural Networks with Sparse Connectivity

Authors: Nong Minh Hieu, Antoine Ledent

Organizations: School of Computing and Information Systems, Singapore Management University

Abstract

Empirical evidence suggests that large neural networks rarely make effective use of all available parameters, with learned solutions exhibiting pronounced sparsity. Despite this ubiquity, existing generalization theory only partially explains how such sparsity influences statistical performance. In this paper, we derive non-asymptotic generalization gap bounds for deep neural networks whose layer-wise weight matrices are restricted by ℓp\ell_p quasi-norm (0<p≤10<p\leq 1) constraints. Our analysis gives rise to a novel result that interpolates between classical norm-based generalization guarantees and parameter-counting bounds. Remarkably, in the extreme sparsity limit as the quasi-norm order pp approaches zero, our analysis reveals a transition to a square-root dependence on the number of active (non-zero) parameters, resulting in bounds that are strictly tighter than existing parameter-counting approaches. Experiments demonstrate that our bounds exhibit a milder dependence on initial architectural choice at the over-parametrization regime, effectively capturing the generalization benefit of the sparsity arising from training on natural data.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Pointwise Generalization in Deep Neural Networks

    May 18, 2026Shaojie Li, Yunbei XuNeural Network GeneralizationRepresentation Learning

  2. Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks

    Sep 8, 2026Xiaoyu Li, Zhizhou Sha, Jiaojiao Jiang +2Neural Network GeneralizationReLU Neural Networks