Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

Xiuyi Yang

cs.LGOct 14, 2026

Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

Authors: Xiuyi Yang

Abstract

What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + λz 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly. The small scale explains the large one: it prescribes how to design, initialize and train the stack.

Figures & tables

Activation	Params	Train acc.	Test acc.	Test loss	s/epoch
GELU	233,866	99.88	98.20	0.0851	2.5
SiLU	233,866	99.70	98.13	0.0814	2.3
ReLU	233,866	99.90	98.07	0.0908	6.5
Tanh	233,866	99.74	97.79	0.0803	1.6
Poly + $\ell_{2}$ proj.	233,866	96.98	95.93	0.1374	4.7
Poly $z+\lambda z^{3}$	233,866	95.36	94.34	0.2042	2.7

Table 1: Activations on MNIST at depth 10 (128 hidden units, AdamW, 10 epochs, seed 42, CPU, no normalization; gated variants use hidden width 82 for parameter parity).

Activation	Norm	Pos	Test acc. (%)	Worst-layer grad. ratio
ReLU	None	post	98.07	3.5853
ReLU	BatchNorm	post	98.19	6.5187
ReLU	LayerNorm	post	98.11	3.5213
ReLU	RMSNorm	post	98.03	2.3955
ReLU	LayerNorm	pre	98.11	3.5213
ReLU	RMSNorm	pre	98.03	2.3955

Table 2: Normalization schemes on MNIST at depth 10, with ReLU and with the polynomial activation

f(z)=z+\lambda z^{3}

. Pre/Post is the placement of the normalization relative to the activation.

Configuration	2	3	5	10	15	20	30	50
ReLU	97.61	98.03	98.24	98.07	97.83	97.65	94.39	11.35
ReLU+BN	–	–	–	–	–	–	97.39	44.64
ReLU+LN	–	–	–	–	–	–	12.95	11.35
Poly	92.81	94.04	95.09	94.34	92.96	92.88	10.28	10.28
Poly+ $\ell_{2}$	92.68	94.45	95.21	95.93	95.62	95.43	94.10	11.35
Poly+LN	93.54	94.91	96.32	96.29	95.94	96.11	95.38	11.35

Table 3: Depth scaling: test accuracy (%) as a function of the number of linear layers

L

(MNIST, 128 hidden units, 10 epochs, seed 42).

L=30,50

come from a separate group of runs.

Configuration	Test acc. (%)	Pred. classes
None / xavier	10.99 $\pm$ 1.55	2
None / identity	31.38 $\pm$ 11.01	10
Residual / xavier	94.50 $\pm$ 0.86	10
Residual / identity	93.66 $\pm$ 1.10	10

Table 4: Identity residual connections without any normalization (MNIST, polynomial activation

z+\lambda z^{3}

L=100

, 128 hidden units, 10 epochs). Entry is test accuracy (%) as mean

\pm

standard deviation over 4 seeds; the last column is the number of predicted classes. The residual uses no projection and no normalization of any kind:

h^{i}=\varphi(W^{i}h^{i-1})+h^{i-1}

( norm=none ).

Configuration	10	50	200	300
Poly (no residual)	94.49	10.82	10.60	9.80
Poly + residual	97.11	93.97	94.81	93.80

Table 5: Depth scaling with the identity residual, no normalization (MNIST, polynomial activation, xavier init, 2 seeds). The plain polynomial column is at chance from

L=50

onward, either as a constant predictor or as an uninformative scatter over the classes; the residual column stays near its

L=10

value out to

L=300

. This collapsed column is not comparable across tables to the last digit (Section 6.5 ).

Configuration	depth	$m$ , $n$	vars	$\dim I$	$\|I\|$
Plain, $\varphi=z+\lambda z^{3}$	$1$	$1,2$	2	$0$	5
Plain, $\varphi=z+\lambda z^{3}$	$1$	$1,3$	2	$0$	5
Plain, $\varphi=z+\lambda z^{3}$	$1$	$2,2$	4	$2$	—
Plain, $\varphi=z+\lambda z^{3}$	$2$	$1,3$	3	$1$	—
Plain, $\varphi=z^{3}$	$1$	$1,2$	2	$1$	—
Plain, $\varphi=z^{3}$	$1$	$2,2$	4	$3$	—

Table 6: Critical-point ideal of the untruncated polynomial network under the square loss,

\varphi(z)=z+\lambda z^{3}

with

\lambda=0.01

m

is the number of hidden units,

n

the number of training samples, and

|I|

the degree of the ideal (stationary points counted over

\mathbb{C}

), reported only when

\dim I=0

. Every row is a small model — at most two hidden layers, at most three samples, at most seven variables — because the Grobner computation, not the architecture, sets the ceiling. A positive dimension means the stationary set is not a finite collection of points.

Configuration	depth	$m,n$	dim	deg	loss	const	zero eig.	n
Plain $z{+}\lambda z^{3}$	$1$	$1,2$	$0$	1	$1.16$	$\cdot$	$0$	2
			$0$	1	$0$	$\cdot$	$0$	1
			$0$	1	$1.16$	$\cdot$	$0$	1
			$0$	1	$0$	$\cdot$	$0$	1
Plain $z{+}\lambda z^{3}$	$1$	$1,3$	$0$	1	$1.25$	$\cdot$	$0$	3
			$0$	1	$0.45110171$	$\cdot$	$0$	2

Table 7: Numerical irreducible decomposition of the critical ideal by Bertini v1.6 for the small models of Table 6 . Each row is one irreducible component of the critical set of the loss (all rows grouped by identical invariants, the multiplicity in the ‘n’ column): its dimension, its degree (the number of witness points, i.e. the number of points a generic linear slice meets it in), the value of the loss on it, whether that loss is constant, and the number of zero eigenvalues of the loss Hessian at a generic point of it, taken over the trainable parameters only. ‘

\cdot

’ in the ‘const’ column means the loss is constant on the component: the spread over its witness points is at double-precision round-off (Hypothesis H2 of Yang (2021) ); the ‘zero eig.’ column then tests Hypothesis H3: it must reproduce the ‘dim’ column. Both hypotheses hold on every one of the 26 genuine components. Components lying in the cleared denominator

\lVert h\rVert^{2}=0

, where the loss is undefined, are artifacts of clearing the denominator and are excluded. The largest model here uses

4

trainable parameters and the slowest decomposition took

45

s, well inside the budget of Sec. 4 .

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 332 papers

Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks
Aug 27, 2026Kathlén Kohn, Giovanni Luca Marchetti, Alex Massarenti +1Parameter Identifiability Polynomial Neural Networks
KTH Stockholm · AI4S AB Stockholm · University of Ferrara
Beyond Overparameterization: Provable Learning of Input-Convex Multi-Layer Polynomial Networks with Active Queries
Oct 4, 2026Jinqi Tang, Qian Chen, Shihong Ding +1Polynomial Neural Networks Input Convex Neural Networks
Peking University
A Theory of Saddle Escape in Deep Nonlinear Networks
May 2, 2026Divit Rawal, Michael R. DeWeeseNeural Network Optimization Neural Network Training Dynamics
Department of Physics University of California, Berkeley Berkeley, CA 94720

Aug 27, 2026J/K move · Enter open · S save

Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks

Kathlén Kohn, Giovanni Luca Marchetti, Alex Massarenti, Massimiliano Mella

KTH Stockholm · AI4S AB Stockholm · University of Ferrara

Parameter Identifiability Polynomial Neural Networks Multilayer Perceptrons

Open paper AskPDF arXiv

Motivated by theoretical problems in deep learning, we conjecture that post-composing a fixed number of pairwise distinct nonconstant polynomials with a generic polynomial of sufficiently large degree yields linearly independent polynomials. This generalizes Newman--Slater's theorem on powers of polynomials. We establish several cases of this conjecture and its origin-passing variant: We prove the result for two polynomials, and for an arbitrary number of polynomials when their degrees are bounded. Furthermore, we show how the conjecture implies a complete understanding of the identifiability (i.e., parameter symmetries) of deep fully connected neural network architectures with generic polynomial activation functions. In particular, for network architectures with layer-specific activations of increasing degree, our established versions of the conjecture fully characterize the set of parameters yielding the same end-to-end network function. As a special case, we fully resolve the identifiability of shallow polynomial networks.

Abstract

Neural Network Optimization

Explore similar work

Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks

Beyond Overparameterization: Provable Learning of Input-Convex Multi-Layer Polynomial Networks with Active Queries

A Theory of Saddle Escape in Deep Nonlinear Networks