We study exact representability by bias-free shallow polynomial neural networks using algebraic geometry. Over C, a width-r network with activation z↦zd computes a sum of rd-th powers of linear forms, whose Zariski closure is a Veronese secant variety. Ideal elimination therefore yields polynomial certificates of nonrepresentability. We implement this construction as a generic architecture-to-certificate pipeline. For quadratics, we recover the exact symmetric determinantal description and explain its dimension through orthogonal symmetry. In higher degree, the implementation recovers classical catalecticant and secant equations and maps the practical reach of direct elimination across a finite architecture sweep. We also derive the exact population loss floor for a rank-two quadratic network on the sphere, illustrating how an algebraic obstruction induces irreducible approximation error.
Figures & tables
Degree d
Completed
Zero
Nonzero
Timed out
2
20
12
8
5
3
6
5
1
19
4
5
3
2
20
5
2
1
1
23
Total
33
21
12
67
Table 1: Theory-versus-computation summary for the 3600-second sweep. “Zero” and “nonzero” refer to the elimination ideal returned by Macaulay2.
(n,d,r)
N
rn
K
Pred. codim.
Returned gens.
Degrees
Time (s)
(3,2,2)
6
6
12
1
1
3
0.50
(4,2,2)
10
8
18
3
10
3
0.52
(4,2,3)
10
12
22
1
1
4
0.74
(6,2,3)
21
18
39
6
105
4
1303.97
(3,3,2)
10
6
16
4
20
3
1.21
(2,4,2)
5
4
9
1
1
3
0.49
Table 2: Selected entries from the certificate atlas. The recorded runtimes are not portable benchmarks.
Figure 1: Empirical mean-squared error for the recorded single-seed sphere experiment. The observed plateau is consistent with the analytic population optimum 1/6 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
(n,d,r)
N
rn
K
Pred. dim.
Pred. codim.
Gens.
Deg.
Time (s)
(3,2,2)
6
6
12
5
1
1
3
0.50
(4,2,2)
10
8
18
7
3
10
3
0.52
(4,2,3)
10
12
22
9
1
1
4
0.74
(5,2,2)
15
10
25
9
6
50
3
1.03
(5,2,3)
15
15
30
12
3
15
4
11.95
(5,2,4)
15
20
35
14
1
1
5
202.66
Appendix
Table 3: Nonzero-ideal entries in the 3600-second certificate atlas.
We study the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. For a given architecture, we define a neuromanifold as the image of the map from all possible network weights into the product of polynomial rings. We quantify the expressivity by the cardinality of the neuromanifold, and derive a natural lower and upper bound. This leads to counting rational points over finite fields, a problem closely linked to the Weil conjectures. Finally, we present an architecture that exhibits a striking difference in the neuromanifolds when considered over a characteristic zero versus a finite-characteristic field, illustrating the critical role of field characteristic in the notion of expressivity.
Maksym Zubkov, Carol Wu, Shiwei Yang +2
Department of Mathematics, University of British Columbia, 1984 Mathematics Rd, Vancouver, V6T 1Z2, BC, Canada
We prove the weak composition-independence conjectures of Kohn, Marchetti, Massarenti and Mella. The proof uses deformations of rational curves and a single kernel of the linearized equation. Riemann-Roch gives a contradiction from tangent fields with controlled poles. By settling these conjectures, we prove that a neural network with a general polynomial activation function of sufficiently large degree is identifiable.
Motivated by theoretical problems in deep learning, we conjecture that post-composing a fixed number of pairwise distinct nonconstant polynomials with a generic polynomial of sufficiently large degree yields linearly independent polynomials. This generalizes Newman--Slater's theorem on powers of polynomials. We establish several cases of this conjecture and its origin-passing variant: We prove the result for two polynomials, and for an arbitrary number of polynomials when their degrees are bounded. Furthermore, we show how the conjecture implies a complete understanding of the identifiability (i.e., parameter symmetries) of deep fully connected neural network architectures with generic polynomial activation functions. In particular, for network architectures with layer-specific activations of increasing degree, our established versions of the conjecture fully characterize the set of parameters yielding the same end-to-end network function. As a special case, we fully resolve the identifiability of shallow polynomial networks.
Kathlén Kohn, Giovanni Luca Marchetti, Alex Massarenti +1
KTH Stockholm · AI4S AB Stockholm · University of Ferrara