cs.AIAug 29, 2026

Understanding Deep Learning via Entropy Space Theory

Authors: Li Li, Tong Zhang, Wentao Yu, Zuobin Wang

Organizations: International Reserach Centre for Nano Handling and Manufacturing of China, Changchun University of Science and Technology, Weixing Road, Changchun City, 130022, Jilin Province, China. · School of Electronic Information Engineering, Changchun University of Science and Technology, Weixing Road, Changchun City, 130022, Jilin Province, China. · Zhongshan Institute of Changchun University of Science and Technology, Huizhan East Road, Zhongshan City, 528437, Guangdong Province, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deep learning is often criticized for its theoretical research lagging behind practice. To make deep learning easier to understand, the entropy space theory is first introduced here. The entropy space can cover all the possibilities of any deep learning model by topological structure. It is independent of network parameters. Through the designed fundamental operations and norm, entropy space is proven to be a normed space within the formal axiomatic framework. Based on the theory, a unified coordinate system is proposed. It can coordinatize every state of a model and rank them by compression of the maximal value of information entropy. The theory offers a novel priori framework for mathematical fundamentals of deep learning.

Figures & tables

Identity element∀ \mutlufontΩx, \mutlufontΩy, \mutlufontΩz⊂\mutlufontΩR\displaystyle\begin{gathered}\forall\,\text{{\mutlufont Ω}}_{x},\,\text{{\mutlufont Ω}}_{y},\,\text{{\mutlufont Ω}}_{z}\subset\text{{\mutlufont Ω}}_{\mathbb{R}}\end{gathered}\mutlufontΩ∅, s.t., \mutlufontΩx+\mutlufontΩ∅=\mutlufontΩ∅+\mutlufontΩx=\mutlufontΩx\text{{\mutlufont Ω}}_{\emptyset},\ \mathrm{s.t.},\ \text{{\mutlufont Ω}}_{x}+\text{{\mutlufont Ω}}_{\emptyset}=\text{{\mutlufont Ω}}_{\emptyset}+\text{{\mutlufont Ω}}_{x}=\text{{\mutlufont Ω}}_{x}
Inverse element s.t., \mutlufontΩx−\mutlufontΩx=\mutlufontΩ∅\mathrm{s.t.},\ \text{{\mutlufont Ω}}_{x}-\text{{\mutlufont Ω}}_{x}=\text{{\mutlufont Ω}}_{\emptyset}
Commutative law s.t., \mutlufontΩx+\mutlufontΩy=\mutlufontΩy+\mutlufontΩx\mathrm{s.t.},\ \text{{\mutlufont Ω}}_{x}+\text{{\mutlufont Ω}}_{y}=\text{{\mutlufont Ω}}_{y}+\text{{\mutlufont Ω}}_{x}
Associative law s.t., \mutlufontΩx+(\mutlufontΩy+\mutlufontΩz)=(\mutlufontΩx+\mutlufontΩy)+\mutlufontΩz\mathrm{s.t.},\ \text{{\mutlufont Ω}}_{x}+(\text{{\mutlufont Ω}}_{y}+\text{{\mutlufont Ω}}_{z})=(\text{{\mutlufont Ω}}_{x}+\text{{\mutlufont Ω}}_{y})+\text{{\mutlufont Ω}}_{z}
Table 1: Laws of addition of entropy space
Identity element∀ a,b∈R,∀ \mutlufontΩx, \mutlufontΩy⊂\mutlufontΩR\displaystyle\begin{gathered}\forall\,a,b\in\mathbb{R},\\[10.0pt] \forall\,\text{{\mutlufont Ω}}_{x},\,\text{{\mutlufont Ω}}_{y}\subset\text{{\mutlufont Ω}}_{\mathbb{R}}\end{gathered}1, s.t.,1⋅\mutlufontΩx=\mutlufontΩx1,\ \mathrm{s.t.},1\cdot\text{{\mutlufont Ω}}_{x}=\text{{\mutlufont Ω}}_{x}
Distributive law over entropy space addition s.t.,a⋅(\mutlufontΩx+\mutlufontΩy)=a⋅\mutlufontΩx+a⋅\mutlufontΩy\mathrm{s.t.},a\cdot(\text{{\mutlufont Ω}}_{x}+\text{{\mutlufont Ω}}_{y})=a\cdot\text{{\mutlufont Ω}}_{x}+a\cdot\text{{\mutlufont Ω}}_{y}
Distributive law over scalar addition s.t.,(a+b)⋅\mutlufontΩx=a⋅\mutlufontΩx+b⋅\mutlufontΩx\mathrm{s.t.},(a+b)\cdot\text{{\mutlufont Ω}}_{x}=a\cdot\text{{\mutlufont Ω}}_{x}+b\cdot\text{{\mutlufont Ω}}_{x}
Associative law s.t.,a⋅(b⋅\mutlufontΩx)=(a⋅b)⋅\mutlufontΩx\mathrm{s.t.},a\cdot(b\cdot\text{{\mutlufont Ω}}_{x})=(a\cdot b)\cdot\text{{\mutlufont Ω}}_{x}
Table 2: Laws of multiplication of entropy space
Distinct regionsρ(Ωx,Ωy)\bm{\rho(\Omega_{x},\Omega_{y})}ρ(Ωx,Ωz)+ρ(Ωz,Ωy)\begin{gathered}\bm{\rho(\Omega_{x},\Omega_{z})+}\\ \bm{\rho(\Omega_{z},\Omega_{y})}\end{gathered}Whether ρ(Ωx,Ωy)≤ρ(Ωx,Ωz)+ρ(Ωz,Ωy)\rho(\Omega_{x},\Omega_{y})\leq\rho(\Omega_{x},\Omega_{z})+\rho(\Omega_{z},\Omega_{y})
① Ωz⊆Ωx∪Ωy,\Omega_{z}\subseteq\Omega_{x}\cup\Omega_{y}, and (Ωx⊆Ωy or Ωy⊆Ωx)(\Omega_{x}\subseteq\Omega_{y}\text{ or }\Omega_{y}\subseteq\Omega_{x})  Yes, and yes when (Ωx,Ωy)→(Ωy,Ωx)(\Omega_{x},\Omega_{y})\rightarrow(\Omega_{y},\Omega_{x}) .
   Yes, and yes when (Ωx,Ωy)→(Ωy,Ωx)(\Omega_{x},\Omega_{y})\rightarrow(\Omega_{y},\Omega_{x}) .
   Yes, and yes when (Ωx,Ωy)→(Ωy,Ωx)(\Omega_{x},\Omega_{y})\rightarrow(\Omega_{y},\Omega_{x}) .
   Yes, and yes when (Ωx,Ωy)→(Ωy,Ωx)(\Omega_{x},\Omega_{y})\rightarrow(\Omega_{y},\Omega_{x}) .
② Ωz⊆Ωx∪Ωy,\Omega_{z}\subseteq\Omega_{x}\cup\Omega_{y}, and (Ωx⊈Ωy and Ωy⊈Ωx)(\Omega_{x}\nsubseteq\Omega_{y}\text{ and }\Omega_{y}\nsubseteq\Omega_{x})  Yes, and yes when (Ωx,Ωy)→(Ωy,Ωx)(\Omega_{x},\Omega_{y})\rightarrow(\Omega_{y},\Omega_{x}) . Also yes when Ωx∩Ωy=∅\Omega_{x}\cap\Omega_{y}=\varnothing .
   Yes, and yes when (Ωx,Ωy)→(Ωy,Ωx)(\Omega_{x},\Omega_{y})\rightarrow(\Omega_{y},\Omega_{x}) .
Table 3: Enumeration of all the possible cases of triangle inequality
Figure 1: Schematic of entropy space based coordinate system

Explore similar work

CardsList

  1. Informational Frustration in Neural Manifolds: Shannon Bottlenecks and the Limits of Learnability

    Jun 29, 2026Srinivasa Rao P., Vangmayi P ReddyStatistical Physics of LearningNeural Network Generalization

    Curlvee Technolabs, India, Former Scientist, C-DAC · Former Scientist, C-DAC · Indian Institute of Technology Madras, India

  2. There Will Be a Scientific Theory of Deep Learning

    Apr 23, 2026Jamie Simon, Daniel Kunin, Alexander Atanasov +11Neural Network Training DynamicsStatistical Learning Theory

    UC Berkeley and Imbue · UC Berkeley · Harvard University +6

  3. On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks

    Jun 15, 2026Luis Contreras, Marco Nahas, Tejas KotwalComplex-Valued Neural NetworksDeep Linear Networks