We characterize the geometry of the set of all weight vectors for which a fully-connected linear neural network computes the same linear transformation W. Said another way, the set of all sequences of matrices (of predetermined sizes) whose product is W. This set of weight vectors is an algebraic variety in the Euclidean weight space, called the fiber of W under the matrix multiplication map. Sometimes this variety is a manifold, but usually not. The rank stratification of a fiber is a natural partition of the fiber into smooth manifolds of various dimensions called strata. We derive the dimensions of these strata and characterize how they are connected to each other. They satisfy the frontier condition: if a stratum intersects the closure of another stratum, then the former stratum is a subset of the closure of the latter stratum. This subset relationship can be expressed as a partial order with a single minimal element, a stratum that is included in the closure of every other stratum. Hence the strata can be arranged in a hierarchy represented by a directed acyclic graph that represents that partial order; we give an algorithm that computes that graph. A key result describes the relationship between this partial order and the ranks of certain matrices in the network. We derive the tangent and normal subspaces of the strata, and we construct elegant bases for those subspaces. Each stratum represents a different pattern of information flow through the network, expressed as a barcode, borrowed from the study of persistent homology. The topology of a stratum depends solely on its barcode. So does its geometry, up to a linear transformation in weight space. Connections among the strata are best understood through "barcode moves," which can be expressed as simple modifications of the barcodes.
Figures & tables
Figure 1: The fiber μ−1([1]) for the network W3W2W1=[θ3][θ2][θ1]=[1]=W .
Figure 2: At left is the fiber μ−1([00]) for the network W2W1=[θ2][θ1θ1′]=[00]=W , partitioned into three strata: S00 is the origin; S10 (blue) is the θ2 -axis with the origin removed; and S01 (pink) is the plane spanned by the θ1 - and θ1′ -axes with the origin removed. At right, the strata are arranged in a stratum dag , organized as a table indexed by the ranks of W1 and W2 . Each dag vertex specifies the dimension of the stratum (dim), the number of degrees of freedom of motion on the fiber (dof), and the number of rank-increasing degrees of freedom (rdof), which stay on the fiber but move off the stratum and onto a higher-dimensional stratum. Always, dof = dim + rdof. A directed path from one stratum to another implies that the former is a subset of the closure of the latter.
Figure 3: Schematic of a linear neural network with four edge layers ( L=4 ).
Akji=nullWk+1◀j∩colWj◀i flow subspace of Rdj for unit layer j (note: Aj−1,j,i={0}=Ak,j,−1 )
akji∈Akji↓(Ak,j,i−1+Ak−1,j,i) prebasis subspace of Rdj for unit layer j
Akji={ak′ji′={0}:k′∈[j,k],i′∈[0,i]} prebasis for Akji (set of prebasis subspaces spanning Akji )
Aj=ALjj prebasis for Rdj for unit layer j (set of prebasis subspaces spanning Rdj )
Bkji=rowWk◀j∩nullWj◀i−1⊤ flow subspace of Rdj for transpose network ( BL+1,j,i={0}=Bk,j,j+1 )
bkji∈Bkji↓(Bk,j,i+1+Bk+1,j,i) prebasis subspace of Rdj for unit layer j of the transpose network
Bkji={bk′ji′={0}:k′∈[k,L],i′∈[i,j]} prebasis for Bkji (set of prebasis subspaces spanning Bkji )
Table 1: Notation used in this paper. Continued in Tables 2 and 3 . See also Table 6 .
αkji=dimAkji=rkWj◀i−rkWk+1◀i=∑t=jk∑s=0iωts dimension of flow subspace Akji
βkji=dimBkji=rkWk◀j−rkWk◀i−1=∑t=kL∑s=ijωts dimension of flow subspace Bkji
η(ML,ML−1,…,M1)=(JLMLJL−1−1,JL−1ML−1JL−2−1,…,J1M1J0−1) a linear transformation of Rdθ
θ=(WL,WL−1,…,W1)∈Rdθ weight vector representing a neural network
Δθ=(ΔWL,ΔWL−1,…,ΔW1)∈Rdθ displacement applied to a weight vector θ
θ′=θ+Δθ=(WL′,WL−1′,…,W1′) weight vector after a move from θ to θ′
θ~=(I~L,I~L−1,…,I~1) the canonical weight vector (for some specified rank list r )
Table 2: More notation used in this paper. Continued in Table 3 . See also Table 6 .
r≤zrk◀i≤zk◀i for all L≥k≥i≥0
r<zr≤z and r=z (at least one of the inequalities holds strictly)
Sˉ closure of set S⊆Rdθ with respect to weight space Rdθ
Q⊥ orthogonal complement of subspace Q (subspace of all vectors orthogonal to all vectors in Q )
MP={Mv:v∈P} product of a matrix M and a subspace P
Z=P⊕Q direct sum decomposition: Z=P+Q (vector sum of subspaces) and P∩Q={0}
Z=⨁i=1mPi direct sum decomposition: Z=∑i=1mPi and for every i∈[1,m] , Pi∩∑j=iPj={0}
Table 3: Still more notation used in this paper. See also Table 6 .
Figure 4: At left is the variety of solutions to W3W2W1=[θ3][θ2][θ1]=[0]=W , partitioned into seven strata: S000 is the origin; S001 , S010 , and S100 are the three coordinate axes with the origin removed; and S011 , S101 , and S110 are the three coordinate planes with the coordinate axes removed. At right is the stratum dag, organized as a three-dimensional table indexed by the ranks of W3 , W2 , and W1 .
Table 4: Some of this paper’s main results about subspaces of the weight space Rdθ and their dimensions. S is the stratum of the fiber μ−1(W) containing the point θ=(WL,WL−1,…,W1) , TθS is the tangent space of S at θ ( TθS passes through the origin, not through θ ), NθS is the normal space of S at θ (the orthogonal complement of TθS ), and dμ(θ) is the (linear) differential map of the matrix multiplication map μ at θ , defined in Section 8 . In our illustrations of stratum dags, “dim” refers to dimS and “dof” refers to dimnulldμ(θ) .
rkW2=5
dim: 31
dof: 34
→
dim: 34
dof: 34
S51
rdof: 3
S52
rdof: 0
↑
↑
rkW2=4
dim: 29
dof: 37
→
dim: 33
dof: 36
→
dim: 35
dof: 35
S41
rdof: 8
S42
rdof: 3
S43
rdof: 0
↑
↑
↑
rkW2=3
dim: 25
dof: 40
→
dim: 30
dof: 38
→
dim: 33
dof: 36
→
dim: 34
dof: 34
Figure 5: Stratum dag representing the stratification of μ−1(W) for W=W2W1 , W2∈R5×6 , W1∈R6×4 , and rkW=1 . For every pair of strata Sts and St′s′ with t≤t′ and s≤s′ , Sts⊆Sˉt′s′ .
Figure 6: The top half is an annotated barcode that illustrates the flow of the “fundamental” subspaces akji through the network. Double boxes represent subspaces of dimension 2 and triple boxes represent subspaces of dimension 3. The bottom half relates the interval counts, the layer sizes, and the subsequence matrix ranks. The number of units dj in unit layer j is the sum of the counts ωts of the intervals that touch layer j (i.e., the dimensions of the subspaces atjs ). Each matrix rank rkWk◀i is the sum of the counts of the intervals that touch both layers k and i .
Rd2
k=4
A4,2,−1={0}
⊆
A420
⊆
A421
⊆
A422=Rd2
⊆
⊆
⊆
⊆
nullW4W3
k=3
A3,2,−1={0}
⊆
A320
⊆
A321
⊆
A322
⊆
⊆
⊆
⊆
nullW3
k=2
A2,2,−1={0}
⊆
A220
⊆
A221
⊆
A222
⊆
⊆
⊆
⊆
{0}
k=1
A1,2,−1={0}
A120={0}
A121={0}
A122={0}
Table 5: The hierarchical nesting of the flow subspaces at unit layer j=2 of a network with L=4 matrices. Top: Ak2i=nullWk+1◀2∩colW2◀i for each k,i . Bottom: Bk2i=rowWk◀2∩nullW2◀i−1 for each k,i .
Figure 7: Top: an example of flow subspaces Akji . Observe that A000 is annihilated in the nullspace of W1 , whereas A300 is not entirely annihilated until it reaches the nullspace of W4 . Observe that A311 and A410 meet at an oblique angle; flow subspaces do not always meet orthogonally. Note that the subspace A322 is five-dimensional, so we cannot draw it complete. Instead, we draw a three-dimensional subspace of A322 labeled A322↓A321 such that A322 is the vector sum of the plane A321 and the space labeled A322↓A321 . Similarly, A222 is a three-dimensional subspace, but we draw a two-dimensional subspace of A222 labeled A222↓A221 such that A222 is the vector sum of the line A221 and the plane labeled A222↓A221 . Bottom: an example of corresponding prebasis subspaces akji , forming flow prebases. The prebasis subspaces for layer j span Rdj . The subspaces a211 , a410 , and a310 cannot all be pairwise orthogonal because A311 and A410 are not orthogonal.
Figure 8: At left, we reprise the annotated barcode from Figure 6 . At right, we tabulate the values of the interval counts ωts with boxes that illustrate how the four summations compute d1 , α322 , β321 , and rkW3◀1=rkW3W2 . At the bottom of the figure, we reprise the four summations for reference.
Figure 9: The canonical weight vector θ~=(I~4,I~3,I~2,I~1) and its subsequence matrices when every interval count is ωki=1 except ω20=2 . Each row of I~j is associated with some interval [i,l] that contains j , marked in shorthand with indices li , and each column of I~j is associated with some interval [h,k] that contains j−1 , marked in shorthand with indices kh . Blocks where the row indices match the column indices are identity matrices; all other component are zeros. Every rank- 1 matrix W∈R5×6 can be factored as W=J4I~4I~3I~2I~1J0−1 where the matrices I~j have the values depicted (determined solely by the barcode) but J4 and J0 depend on W and are invertible. This gives us a weight vector (J4I~4,I~3,I~2,I~1J0−1) that lies on the fiber μ−1(W) and has the same rank list as θ~ .
Figure 10: The influence of a basic move in which a factor matrix Wj undergoes a nonzero displacement ΔWj∈olkjih . The effects on the subsequence matrix Wt◀s are listed for every t and s with L≥t≥s≥0 . These tables are triangular, though it’s not obvious at first: the hatched regions represent unused zones where t<s . A yellow rectangle indicates which subsequence matrices increase in rank, constituting a barcode move if no rank decreases. The black font indicates where Wt◀s′=Wt◀s . The red font indicates where Wt◀s′=Wt◀s because the matrix Wj is not a factor in Wt◀s . The blue font indicates where Wt◀s′=Wt◀s by properties of the flow relationships. (a) Table for the case where L>l>k≥j>i>h>0 . An example of a swapping move (if no rank decreases). (b) The third row disappears if k+1=j , and the third column disappears if i=j . When k+1=j=i , the move is a connecting move. (c) The second row disappears if k≥l , and the second column disappears if i≤h . If either inequality holds, no subsequence matrix rank increases; the move is not a barcode move. The first column disappears if h=0 . (The first row disappears if l=L , though we don’t depict that case here. If h=0 and l=L , then W′=W and we move off the fiber.)
Figure 11: Two examples of connecting moves. The top example is the simplest example possible: W1 has been perturbed to increase its rank from zero to one. In the bottom example, W2 has been perturbed. In both examples, the perturbation of Wj replaces two intervals [h,j−1] and [j,l] with a single interval [h,l] . Three interval counts change, at three of the four corners of the red rectangle: ωlj and ωj−1,h decrease by one, and ωlh increases by one. The ranks of the subsequence matrices Wt◀s increase by one for all t∈[j,l] and s∈[h,j−1] (the ranks inside the red rectangle, including rkWj ). Outside the red rectangle, all interval counts and matrix ranks are unchanged.
Figure 12: Two examples of swapping moves. In the top example—the simplest example possible—either W1 or W2 may be perturbed to cause the move. In the bottom example, any of W2 , W3 , or W4 may be perturbed. The intervals [h,k] and [i,l] are replaced by an interval [h,l] , longer than both original intervals, and an interval [i,k] , shorter than both. Four interval counts change, at the four corners of the red rectangle: ωkh and ωli decrease, and ωlh and ωki increase. The ranks of the subsequence matrices Wt◀s increase for all t∈[k+1,l] and s∈[h,i−1] (the ranks inside the red rectangle). Outside the red rectangle, all interval counts and matrix ranks are unchanged.
Figure 13: Annotated barcodes for four strata in a stratum dag. (Assume that W4=0 , although W4 is not illustrated, so all four strata are on the fiber of W=0 .) Green arrows are edges of the dag; in this example they all representing swapping moves. This dag includes the directed edge (Sr,Sz) , even though there are directed paths from Sr to Sz passing through Su or Sv . Hence the dag is not a Hasse diagram. The edge (Sr,Sz) represents swapping moves whose displacements come from the subspaces ϕ31210 or ϕ31110 .
Figure 14: One-matrix subspaces for the canonical weight vector θ~=(I~4,I~3,I~2,I~1) when every ωki=1 except ω20=2 . Yellow and green backgrounds indicate basic moves that stay on the fiber (moves with displacements from subspaces in ΘOfiber such as ϕ22100 and ϕ32330 ), whereas pink and blue backgrounds indicate basic moves off the fiber (moves from subspaces in ΘOoff such as ϕ44420 and ϕ43220 , satisfying l=4 and h=0 ). Green and blue backgrounds indicate discrete moves that increase the rank of some subsequence matrix (moves from subspaces in ΘObarc such as ϕ32330 and ϕ43220 , satisfying l>k and i>h ), whereas yellow and pink backgrounds indicate moves that do not increase any rank. Hence, yellow indicates moves that stay on the closure of the stratum (moves from subspaces in ΘOstrat such as ϕ22100 ). Green indicates discrete moves to a different stratum that stay on the fiber (moves from subspaces in ΘOfiber∩ΘObarc such as ϕ32330 ); these induce edges in the stratum dag. Pink indicates moves off the fiber in which no subsequence matrix’s rank increases. Blue indicates discrete moves off the fiber.
Figure 16: Four examples of reverse barcode moves found by the algorithm FindLastMove (shown prior to performing the reverse move). In each example, the algorithm finds a red rectangle such that its bottom corner lies at a bottommost positive Δωlh , its top corner lies at a positive Δωki or between two Δd ’s, and every Δr inside the rectangle is positive. In (b) and (c) the algorithm finds a reverse connecting move ( k=i−1 ), whereas in (a) and (d) it finds a reverse swapping move ( k≥i ), with (d) being a move that changes W ’s rank. In (a) and (b), Line 10 of FindLastMove determines the top corner of the rectangle (hence k=l−1 ), whereas in (c) and (d), Lines 12 and 13 do (hence k≤l−2 ).
Figure 17: Fiber prebases at five points on the fibers from Figures 4 and 2 . Note that we have drawn each prebasis translated to the point where it is defined (to clarify how the subspaces are related to the strata), but all the subspaces actually pass through the origin, and not necessarily through the points θ , θ′ , etc.
Figure 18: Two-matrix subspaces in ΘToff∪ΘTbarc at the canonical weight vector θ~=(I~4,I~3,I~2,I~1) when every ωki=1 except ω20=2 . Each two-matrix subspace τlkjih∈ΘT has nonzero components in two blocks within two consecutive factor matrices. Pink backgrounds indicate subspaces in ΘToff ; green backgrounds indicate subspaces in ΘTbarc ; and blue backgrounds indicate subspaces in both ΘToff and ΘTbarc . Subspaces in ΘToff∪ΘTbarc (all three colors) represent directions along which the stratum is curved. Compare with Figure 14 , which illustrates the one-matrix subspaces at θ~ .
Figure 19: Visualizing prebases at the canonical weight vector θ~=(I~4,I~3,I~2,I~1) when every ωki=1 except ω20=2 . Top: visualizing the prebasis Θstrat , the tangent space TθS=spanΘstrat , and its dimension Dstrat . Each component with a yellow background indicates a dimension in a one-matrix subspace in Θstrat (such as ϕ44441 ); this component varies freely in TθS . Other background colors indicate dimensions in two-matrix subspaces in Θstrat (such as τ44320 ), dimensions that linearly link two components in weight space: pink for a direction that changes one component of I~4 and one component of I~3 ; blue for I~3 and I~2 ; and orange for I~2 and I~1 . You can see the dimension Dstrat of TθS by counting dimensions of all four colors. Center: visualizing the prebasis Θfree , the nullspace nulldμ(θ)=spanΘfree , and its dimension Dfree at θ~ . Bottom: visualizing the prebasis Θfiber . Observe that for Θfiber only, some components participate in both a one-matrix subspace and a two-matrix subspace, such as ϕ43431 and τ43331 .
ΘO={ϕlkjih={0}:L≥l≥j≥i≥0 and L≥k≥j−1≥h≥0}
\mboxDO=dθ=∑j=1Ldjdj−1 (8.19)
ΘOoff={ϕLkji0∈ΘO}={ϕLkji0={0}:L≥j>0,L≥k≥j−1, and j≥i≥0}
ΘObarc={ϕlkjih∈ΘO:l>k and i>h}={ϕlkjih={0}:L≥l≥k+1≥j≥i>h≥0}
Table 6: Sets of subspaces of Rdθ and their total dimensions. See also Table 7 in Appendix E .
Figure 20: Top: visualizing the prebasis Ψstrat , the normal space NθS=spanΨstrat , and its dimension dθ−Dstrat at the canonical weight vector θ~=(I~4,I~3,I~2,I~1) when every ωki=1 except ω20=2 . Each component with a pink background is a dimension that varies freely in NθS ; it also varies freely in some subspace in Ψstrat (such as ψ4342 ). Blue backgrounds represent a dimension in which four components vary together, one in each of the four matrices (such as the single degree of freedom in ψ4410 ). Gold and purple backgrounds represent dimensions that linearly link components in three of the four matrices. Turquoise, green, and tan backgrounds represent dimensions that linearly link components in two of the four matrices. You can see the dimension of NθS by counting dimensions of all colors. Bottom: visualizing the prebasis Ψfree , the rowspace rowdμ(θ)=spanΨfree , and its dimension rkdμ(θ)=dθ−Dfree at θ~ . Gradient descent algorithms for training neural networks choose a descent direction in rowdμ(θ) .
Figure 21: Green arrows indicate the order in which our algorithm for enumerating valid rank lists assigns ranks to the subsequence matrices. The dj ’s and rkW are fixed (black text); only the ranks in red can vary from stratum to stratum. The ≥ and ≤ symbols indicate some of the constraints that the ranks must satisfy. There are additional constraints: each interval count ωki=rkWk◀i−rkWk◀i−1−rkWk+1◀i+rkWk+1◀i−1 must be nonnegative (by the Frobenius rank inequality). For example, rkW2≤d1−rkW1+rkW2W1 .
Figure 24: A map showing the structure of coldμ(θ) , nulldμ⊤(θ) , TWDVρ , and NWDVρ for a weight vector θ , where W=μ(θ) and ρ=rkW . Here we use the standard prebases B4 and A0 (see Appendix B ) so that the subspaces ak00 are mutually orthogonal and span Rd0 , the subspaces b44i are mutually orthogonal and span Rd4 , and the subspaces eki=b44i⊗ak00 are mutually orthogonal and span Rd4×d0 .
Figure 25: To the left of the barcode, we draw the matrix G⊤ and draw green links in it to indicate which components Gki are nonzero. In the top example, dashed purple edges show some of the ways by which gradient descent can “complete a loop” and make progress by modifying W3 , W2 , or W1 . Hence this example indicates that θ is not a critical point. In the bottom example, G is nonzero but it is not possible to complete a loop by modifying just one factor matrix. Hence this example indicates that θ is a spurious critical point. (Not depicted is the case of a pure critical point, which simply has G=0 .)
Alignment, the tendency of adjacent weight matrices in deep networks to develop compatible subspace orientations, underlies gradient flow, Neural Collapse, and representation similarity across architectures. Despite extensive empirical documentation, these phenomena have resisted unified theoretical treatment: existing explanations are post-hoc, each fitted to a specific observation with whatever mathematics is at hand. We reverse this direction by deriving the mathematical structure that layerwise alignment inherently demands. Using geometric invariant theory, we prove that alignment geometry has a canonical closed, polystable stratum given by a flag variety, and that subspace intersection dimension is its unique reparameterization-invariant observable, establishing that subspace metrics are not empirical conventions but mathematical necessities. This unified framework yields two dynamical consequences: ridge regularization drives subspace alignment at an exponential rate set by weight decay, whereas nonlinear activations induce a commutator obstruction to exact basis alignment, generically present in nonlinear networks and absent in linear ones. Together these give a geometric explanation of the Level-2/3 hierarchy in Neural Collapse from first principles rather than post-hoc analysis. The commutator magnitude and head subspace overlap further serve as weight-space windows into internal alignment structure, requiring no forward passes. Experiments on multilayer perceptrons, residual networks, and pretrained language models support the proposed diagnostics and delineate their scope.
Jingchuan Xiao, Xinyi Sui, Cihan Ruan
Department of Mathematics and Computer Studies, Mary Immaculate College, Ireland · Department of Computer Science and Engineering, Santa Clara University, USA
We propose a computational tropical geometry framework for the symbolic analysis of neural networks with tropical activations. The number of linear regions of a neural network has been actively studied as a measure of the expressivity of a given architecture. To study these, we work in the setting of tropical geometry---a combinatorial and polyhedral variant of algebraic geometry---where there are known connections between tropical rational maps and feedforward neural networks. We expand this connection by developing concrete computational tools for studying the linear regions of neural networks. We present an algorithm, together with a proof of correctness, which computes the linear regions of a neural network as explicit unions of polyhedra. We further relate the computation of the number of linear regions of a tropical expression to the number of monomials that appear in it, and show how tropical expressions can often be pruned to remove redundant monomials. We introduce the Hoffman constant of a neural network's tropical expression, a geometric quantity that controls the distance from any point in the input space to the farthest linear region. We provide the open source Julia library TropicalNN.jl, which is built on top of the OSCAR computer algebra system and implements the algorithms mentioned above to analyze neural networks symbolically using their tropical representations. We present a set of proof-of-concept computational examples to demonstrate how our tropical geometric theory can be applied to reveal insights on the expressivity of a network architecture.
Paul Lezeau, Thomas Walker, Yueqi Cao +2
Department of Mathematics, Imperial College London, UK · London School of Geometry and Number Theory, UK · Department of Electrical and Computer Engineering, Rice University, USA +2
We show that neural networks with activations satisfying a Riccati-type ordinary differential equation condition, an assumption arising in recent universal approximation results in the uniform topology, produce Pfaffian outputs on analytic domains with format controlled only by the architecture. Consequently, superlevel sets, as well as Lie bracket rank drop loci for neural network parameterized vector fields, admit architecture-only bounds on topological complexity, in particular on total Betti numbers, uniformly over all weights.