Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures

Maxim Bolshim; Alexander Kugaevskikh

cs.LGApr 13, 2026

Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures

Authors: Maxim Bolshim, Alexander Kugaevskikh

Organizations: ITMO University, Saint Petersburg, Russia

Abstract

Modern automatic differentiation frameworks (JAX, PyTorch) return the Hessian of the loss function as a monolithic tensor, without exposing the internal structure of inter-layer interactions. This paper presents an analytical formalism that explicitly decomposes the full Hessian into blocks indexed by the DAG of an arbitrary architecture. The canonical decomposition $H = H^{GN} + H^T$ separates the Gauss--Newton component (convex part) from the tensor component (residual curvature responsible for saddle points). For piecewise-linear activations (ReLU), the tensor component of the input Hessian vanishes ( $H^{T}_{v,w}\!\equiv\!0$ a.e., $H^f_{v,w}\!=\!H^{GN}_{v,w}\!\succeq\!0$ ); the full parametric Hessian contains residual terms that do not reduce to the GGN. Building on this decomposition, we introduce diagnostic metrics (inter-layer resonance~ $\mathcal{R}$ , geometric coupling~ $\mathcal{C}$ , stable rank~ $\mathcal{D}$ , GN-Gap) that are estimated stochastically in $O(P)$ time and reveal structural curvature interactions between layers. The theoretical analysis explains exponential decay of resonance in vanilla networks and its preservation under skip connections; empirical validation spans fully connected MLPs (Exp.,1--5) and convolutional architectures (ResNet-18, ${\sim}11$ M~parameters, Exp.,6). When the architecture reduces to a single node, all definitions collapse to the standard Hessian $\nabla^2_θ\mathcal{L}(θ)\in\mathbb{R}^{p\times p}$ .

Figures & tables

Graph and derivatives
Symbol	Definition
$G=(V,E)$	DAG of the neural network
$\mathrm{Pa}(v)$ , $\mathrm{Ch}(v)$	Parents / children of node $v$
$f_{v}\in\mathbb{R}^{d_{v}}$	Output of node $v$
$\theta_{v}\in\mathbb{R}^{p_{v}}$	Parameters of node $v$
$\delta_{v}$	Gradient $\nabla_{f_{v}}\mathcal{L}$

Table 1: Core notation.

Panel of Figure 1 — Figure 1: (a) DAG of a neural network with skip connection ( $v_{0}\!\to\!v_{2}$ ): forward edges (solid) and skip edge (dashed). The full Jacobian is a sum of path-wise contributions. (b) Backward pass of curvature routing: curvature matrices $H^{f}_{v,w}$ are propagated from $out$ over the graph via transposed Jacobians; with a skip connection, the block $H^{f}_{v_{0},out}$ receives contributions from both paths.

Panel of Figure 2 — Figure 2: Exp. 1: Geometric coupling $\bar{\mathcal{C}}(d)$ vs. distance $d$ (error bars: $\pm 1\sigma$ over 5 seeds). (a) $L\!=\!8$ , (b) $L\!=\!12$ . Plain MLP: $\mathcal{C}$ decays monotonically from 1 to 0.24 ( $L\!=\!12$ , init), reflecting loss of geometric coherence between distant layers. Residual MLP: $\mathcal{C}\!>\!0.93$ at all distances; skip connections preserve coupling. After training both architectures shift upward (curvature becomes more uniform).

$L$	Arch	$\rho_{\max}^{\mathrm{init}}$	$\rho_{\max}^{\mathrm{mid}}$	$\rho_{\max}^{\mathrm{final}}$	$\lambda_{1}^{\mathrm{init}}$	$\lambda_{1}^{\mathrm{mid}}$	$\lambda_{1}^{\mathrm{final}}$	Acc (%)
8	Plain	2.59	2.50	2.78	$-$ 0.02	0.05	0.08	49.8
8	Res.	1.25	1.31	1.33	0.02	0.04	0.04	51.3
10	Plain	2.69	2.47	2.59	$-$ 0.03	0.03	0.04	49.7
10	Res.	1.22	1.27	1.29	0.02	0.03	0.04	51.3
12	Plain	2.69	2.48	2.55	$-$ 0.03	0.03	0.04	49.7
12	Res.	1.20	1.23	1.25	0.01	0.03	0.03	51.3

Table 2: Spectral norms of Jacobians

\rho_{\max}

, Lyapunov exponent

\lambda_{1}

, and test accuracy for Exp. 1 configurations (CIFAR-10, mean over 5 seeds).

Panel of Figure 3 — Figure 3: Exp. 1: Mean resonance $\bar{\mathcal{R}}(d)$ vs. distance $d$ between layers (log scale on $y$ ; error bars $\pm 1\sigma$ over 5 seeds). (a) $L\!=\!8$ , (b) $L\!=\!12$ . Plain MLP exhibits exponential decay (straight lines on log scale, $R^{2}\!>\!0.91$ ); Residual MLP shows stabilization ( $b\!\approx\!0.02$ ). Scale differs: init $\sim 10^{-1}$ , final $\sim 10^{0}$ (reflects overall curvature growth during training).

$d_{u}$	$\mathcal{D}_{\mathrm{far}}^{\mathrm{init}}$	$\mathcal{D}_{\mathrm{far}}^{\mathrm{final}}$	$\mathcal{C}_{\mathrm{far}}^{\mathrm{init}}$	$\mathcal{C}_{\mathrm{far}}^{\mathrm{final}}$	Acc (%)
4	$1.65\pm 0.23$	$2.03\pm 0.20$	$0.21\pm 0.06$	$0.37\pm 0.02$	8.9 ‡
8	$2.58\pm 0.11$	$3.39\pm 0.36$	$0.35\pm 0.05$	$0.58\pm 0.03$	14.2
16	$4.04\pm 0.65$	$4.90\pm 0.24$	$0.43\pm 0.03$	$0.68\pm 0.01$	17.8
32	$5.67\pm 0.99$	$5.01\pm 0.41$	$0.52\pm 0.03$	$0.73\pm 0.01$	19.8
64	$6.95\pm 0.52$	$5.15\pm 0.34$	$0.61\pm 0.03$	$0.77\pm 0.02$	21.1
128	$9.16\pm 0.91$	$5.45\pm 0.42$	$0.66\pm 0.04$	$0.84\pm 0.01$	22.0

Table 3: Exp. 2: bottleneck ablation (

L\!=\!6

, CIFAR-100). Stable rank

\mathcal{D}_{\mathrm{far}}

and coupling

\mathcal{C}_{\mathrm{far}}

vs. narrow layer width

d_{u}

(mean

\pm 1\sigma

over 5 seeds). † Control:

d_{\mathrm{base}}\!=\!512

(stochastic estimate, 100 probes, subsample 64). ‡ Acc at uninformative model level (

d_{u}\!\ll\!K

); metrics for

d_{u}\!\geq\!8

are preferred for conclusions.

Panel of Figure 4 — Figure 4: Exp. 2: Stable rank $\mathcal{D}_{\mathrm{far}}$ vs. bottleneck width $d_{u}$ (CIFAR-100). Solid/dashed: init/final; bright: $L\!=\!6$ , faded: $L\!=\!8$ . $L\!=\!8$ : $\mathcal{D}_{\mathrm{far}}^{\mathrm{init}}$ is lower for $d_{u}\!\geq\!64$ ( ${\sim}8$ – $20\,\%$ ), consistent with additional rank compression at greater distance. Dotted: theoretical bound $\min(d_{u},K{-}1)$ . $d_{u}\!=\!512$ ( † , control: $d_{\mathrm{base}}\!=\!512$ , no bottleneck): Table 3 .

Activation	$\mathrm{Gap}^{\mathrm{init}}$	$\mathrm{Gap}^{\mathrm{final}}$	$\Delta$ (%)	$\mathbb{E}[\sigma^{\prime\prime\,2}]$	Acc (%)
ReLU	$0.618\pm 0.160$	$0.199\pm 0.030$	$-$ 67.8	0	$51.3\pm 0.3$
LeakyReLU	$0.626\pm 0.161$	$0.212\pm 0.013$	$-$ 66.1	0	$51.5\pm 0.3$
Softplus	$0.459\pm 0.098$	$0.272\pm 0.024$	$-$ 40.9	0.045	$52.7\pm 0.2$
SiLU	$1.967\pm 0.317$	$0.359\pm 0.051$	$-$ 81.7	0.142	$52.6\pm 0.3$
GELU	$2.296\pm 0.415$	$0.516\pm 0.117$	$-$ 77.5	0.271	$52.0\pm 0.4$

Table 4: Exp. 3b: GN-Gap with LayerNorm (

L\!=\!6

, width

\,=64

, CIFAR-10, mean

\,{\pm}\,1\sigma

over 5 seeds).

\Delta

: relative change init

\,{\to}\,

final.

Activation	$\mathrm{Gap}^{\mathrm{init}}$	$\mathrm{Gap}^{\mathrm{final}}$	$\Delta$ (%)	$\mathbb{E}[\sigma^{\prime\prime\,2}]$	Acc (%)
ReLU	${<}\,10^{-7}$	${<}\,10^{-7}$	—	0	$45.1\pm 0.9$
LeakyReLU	${\lesssim}\,10^{-7}$	${\lesssim}\,10^{-7}$	—	0	$45.0\pm 0.8$
Softplus	$0.122\pm 0.020$	$0.111\pm 0.003$	$-$ 8.9	0.058	$10.0\pm 0.0$
SiLU	$0.225\pm 0.005$	$0.217\pm 0.008$	$-$ 3.5	0.246	$11.7\pm 2.1$
GELU	$0.358\pm 0.008$	$0.345\pm 0.012$	$-$ 3.4	0.618	$11.9\pm 2.4$

Table 5: Exp. 3: GN-Gap, isolation protocol (no LayerNorm,

L\!=\!6

, width

\,=64

, CIFAR-10, mean

\,{\pm}\,1\sigma

over 5 seeds).

Panel of Figure 5 — Figure 5: Exp. 3: GN-Gap (init) vs. $\mathbb{E}[\sigma^{\prime\prime}(z)^{2}]$ for 5 activations (isolation protocol, LayerNorm off). Linear fit $R^{2}\!=\!0.908$ ; Spearman rank correlation $\rho_{s}\!=\!0.97$ ( $p\!<\!0.01$ ). With $n\!=\!5$ , $R^{2}$ has limited power, but monotonicity of ranking ( $\rho_{s}\!\approx\!1$ ) robustly confirms the scaling $H^{T}\!\sim\!\sigma^{\prime\prime\,2}$ .

Merge	$\sigma$	$\mathrm{Gap}^{\mathrm{init}}$	$\mathrm{Gap}^{\mathrm{final}}$	$\\|T\\|_{F}^{\mathrm{init}}$	$\\|T\\|_{F}^{\mathrm{final}}$
sum	ReLU	$6.7{\cdot}10^{-8}$	$9.3{\cdot}10^{-8}$	$4.1{\cdot}10^{-9}$	$2.0{\cdot}10^{-8}$
sum	SiLU	$7.0{\cdot}10^{-8}$	$9.6{\cdot}10^{-8}$	$4.1{\cdot}10^{-9}$	$2.0{\cdot}10^{-8}$
cat	ReLU	$8.4{\cdot}10^{-8}$	$1.2{\cdot}10^{-7}$	$6.6{\cdot}10^{-10}$	$3.3{\cdot}10^{-8}$
cat	SiLU	$1.33\pm 0.17$	$0.082\pm 0.007$	$6.8{\cdot}10^{-3}\pm 5.0{\cdot}10^{-4}$	$2.5{\cdot}10^{-2}\pm 1.1{\cdot}10^{-3}$

Table 6: Exp. 4: GN-Gap at the merge node of Diamond MLP (

k\!=\!2

, width

\,{=}\,32

, CIFAR-10,

\mathrm{Acc}\approx 50\,\%

; mean

\pm 1\sigma

over 5 seeds).

\|T_{\mathrm{merge}}\|_{F}

: Frobenius norm of the tensor component.

Panel of Figure 6 — Figure 6: Exp. 4: GN-Gap at the merge node of Diamond MLP ( $k\!=\!2$ ). Logarithmic scale highlights the ${\sim}7$ order-of-magnitude separation between cat $\,{+}\,$ SiLU ( $\sigma^{\prime\prime}\!\neq\!0$ , nonlinear merge) and the remaining configurations (linear merge or $\sigma^{\prime\prime}\!=\!0$ a.e.). Sum-merge yields $T\!=\!0$ by construction — $\mathrm{Gap}\!\approx\!0$ regardless of $\sigma$ .

Model	Checkpoint	$\\|H^{f}\\|_{F}$	$\\|H^{GN}\\|_{F}$	$\\|H^{T}\\|_{F}$	Gap
Attention	init	$9.2{\cdot}10^{-3}$	$1.7{\cdot}10^{-4}$	$9.2{\cdot}10^{-3}$	$57.5\pm 16.4$
Attention	mid	$5.2{\cdot}10^{-3}$	$7.4{\cdot}10^{-5}$	$5.2{\cdot}10^{-3}$	$79.9\pm 23.3$
Attention	final	$5.2{\cdot}10^{-3}$	$8.7{\cdot}10^{-5}$	$5.2{\cdot}10^{-3}$	$69.3\pm 20.1$
ReLU-MLP	init	$3.5{\cdot}10^{-2}$	$3.5{\cdot}10^{-2}$	$3.4{\cdot}10^{-9}$	$9.6{\cdot}10^{-8}$
ReLU-MLP	mid	$3.5{\cdot}10^{-1}$	$3.5{\cdot}10^{-1}$	$2.8{\cdot}10^{-8}$	$8.8{\cdot}10^{-8}$
ReLU-MLP	final	$3.6{\cdot}10^{-1}$	$3.6{\cdot}10^{-1}$	$2.6{\cdot}10^{-8}$	$7.4{\cdot}10^{-8}$

Table 7: Exp. 5: GN-Gap for Toy-Attention vs. per-position ReLU-MLP (

d\!=\!16

S\!=\!8

, synthetic regression, mean

\,{\pm}\,1\sigma

over 5 seeds).

\|H^{T}\|_{F}

: Frobenius norm of the tensor component; Gap

=\|H^{T}\|_{F}/\|H^{GN}\|_{F}

$d$	ReLU		SiLU
	ResNet	Plain	ResNet	Plain
$0$	${<}\,2{\cdot}10^{-6}$	${<}\,10^{-6}$	$0.43\pm 0.12$	$0.21\pm 0.11$
$1$	${<}\,2{\cdot}10^{-6}$	${<}\,10^{-6}$	$0.24\pm 0.07$	$0.12\pm 0.04$
$2$	${<}\,2{\cdot}10^{-6}$	${<}\,2{\cdot}10^{-6}$	$0.22\pm 0.06$	$0.06\pm 0.02$
$3$	${<}\,2{\cdot}10^{-6}$	${<}\,10^{-6}$	$0.15\pm 0.04$	$0.05\pm 0.02$
$4^{\dagger}$	${<}\,2{\cdot}10^{-6}$	${<}\,2{\cdot}10^{-6}$	${<}\,2{\cdot}10^{-6}$	${<}\,2{\cdot}10^{-6}$

Table 8: Exp. 6: GN-Gap by distance

d

for ResNet-18 (CIFAR-10, final checkpoint, mean

\,{\pm}\,1\sigma

over 5 seeds). Row

d\!=\!4

: linear classifier (Remark 39 ).

	$\bar{\mathcal{R}}(d)$				$\bar{\mathcal{C}}(d)$
$d$	ReLU		SiLU		ReLU		SiLU
	Res.	Plain	Res.	Plain	Res.	Plain	Res.	Plain
$0$	$4.5{\cdot}10^{-2}$	$2.1{\cdot}10^{-2}$	$3.0{\cdot}10^{-2}$	$2.1{\cdot}10^{-2}$	$1.00$	$1.00$	$1.00$	$1.00$
$1$	$3.5{\cdot}10^{-2}$	$3.5{\cdot}10^{-4}$	$1.0{\cdot}10^{-2}$	$1.1{\cdot}10^{-4}$	$0.95$	$0.93$	$0.56$	$0.24$
$2$	$3.3{\cdot}10^{-2}$	$6.0{\cdot}10^{-5}$	$4.2{\cdot}10^{-3}$	$6.9{\cdot}10^{-6}$	$0.92$	$0.83$	$0.31$	$0.06$
$3$	$3.6{\cdot}10^{-2}$	$2.9{\cdot}10^{-5}$	$2.1{\cdot}10^{-3}$	$1.2{\cdot}10^{-6}$	$0.89$	$0.71$	$0.17$	$0.01$

Table 9: Exp. 6: mean resonance

\bar{\mathcal{R}}(d)

and coupling

\bar{\mathcal{C}}(d)

at initialization (ResNet-18, CIFAR-10, mean over 5 seeds).

Panel of Figure 7 — Figure 7: Exp. 6: Mean resonance $\bar{\mathcal{R}}(d)$ for ResNet-18 (CIFAR-10, log-scale on $y$ ; $\pm 1\sigma$ over 5 seeds). (a) ReLU: ResNet preserves $\bar{\mathcal{R}}$ at init ( $R(0)/R(4)\!\approx\!1$ ); Plain decays by ${\sim}627\times$ . (b) SiLU: decay is stronger—Plain init $47\,000\times$ ; ResNet $25\times$ , slower than Plain (skip connections stabilize curvature).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

$d_{u}$	$\mathcal{D}_{\mathrm{far}}^{\mathrm{init}}$	$\mathcal{D}_{\mathrm{far}}^{\mathrm{final}}$	$\mathcal{C}_{\mathrm{far}}^{\mathrm{init}}$	$\mathcal{C}_{\mathrm{far}}^{\mathrm{final}}$	Acc (%)
4	$1.58\pm 0.43$	$2.44\pm 0.17$	$0.20\pm 0.08$	$0.33\pm 0.01$	9.6 ‡
8	$2.43\pm 0.30$	$3.92\pm 0.39$	$0.34\pm 0.08$	$0.45\pm 0.04$	14.1
16	$4.27\pm 0.57$	$5.01\pm 0.62$	$0.43\pm 0.03$	$0.56\pm 0.01$	17.7
32	$6.36\pm 0.69$	$5.13\pm 0.45$	$0.51\pm 0.03$	$0.62\pm 0.01$	19.1
64	$6.40\pm 0.60$	$5.41\pm 0.55$	$0.58\pm 0.02$	$0.68\pm 0.03$	20.5
128	$7.90\pm 0.28$	$4.89\pm 0.53$	$0.65\pm 0.01$	$0.76\pm 0.02$	21.4

Appendix

Table N.1: Exp. 2: bottleneck ablation (

L\!=\!8

, CIFAR-100). Same protocol as Table 3 (mean

\pm 1\sigma

over 5 seeds).

$L$	Arch.	$\rho_{\max}^{\mathrm{init}}$	$\rho_{\max}^{\mathrm{final}}$	$\lambda_{1}^{\mathrm{init}}$	$\lambda_{1}^{\mathrm{final}}$	Acc (%)
8	Plain SN	0.91	1.00	$-$ 1.01	$-$ 0.24	36.3
8	Res.	1.25	1.33	0.02	0.04	51.3
12	Plain SN	0.91	0.89	$-$ 1.01	$-$ 1.03	10.0
12	Res.	1.20	1.25	0.01	0.03	51.3
16	Plain SN	0.91	0.90	$-$ 1.01	$-$ 1.03	10.0
16	Res.	1.17	1.20	0.01	0.02	51.0

Appendix

Table N.2: Exp. 1b: spectral normalization. Same metrics as Table 2 . Plain SN guarantees

\rho\!\leq\!1

and

\lambda_{1}\!<\!0

; Residual MLP as control without SN (CIFAR-10, mean over 5 seeds).

Panel of Figure 19 — Figure N.1: Exp. 1b: resonance $\bar{\mathcal{R}}(d)$ under spectral normalization (Plain SN). Solid curves — initialization ( $\rho\!\approx\!0.91\!<\!1$ ): exponential decay over ${\sim}5.5$ orders of magnitude at $L\!=\!16$ , directly verifying inequality equation 10 . Dashed — after training: at $L\!=\!8$ ( $\rho\!\to\!1.0$ ) $R$ plateaus at ${\approx}0.94$ (boundary of the theorem condition); at $L\!=\!12$ decay persists ( $\lambda_{1}\!<\!0$ ). Error bands: $\pm 1\sigma$ over 5 seeds.

$L$	Arch.	Phase	$\rho_{\max}$	$\bar{\mathcal{R}}(0)$	$\bar{\mathcal{R}}(L{-}1)$	$\bar{\mathcal{C}}(L{-}1)$
16	Plain	init	$2.59\pm 0.19$	$0.072\pm 0.018$	$0.021\pm 0.001$	$0.263$
16	Plain	final	$2.56\pm 0.03$	$4.852\pm 1.219$	$0.80\pm 0.13$	$0.624$
16	Res.	init	$1.17\pm 0.00$	$0.121\pm 0.005$	$0.12\pm 0.00$	$0.983$
16	Res.	final	$1.21\pm 0.01$	$2.156\pm 0.111$	$1.44\pm 0.08$	$0.887$
32	Plain	init	$2.71\pm 0.13$	$0.067\pm 0.018$	$0.011\pm 0.003$	$0.145$
32	Plain	final	$2.62\pm 0.03$	$1.750\pm 0.530$	$0.52\pm 0.16$	$0.403$

Appendix

Table N.3: Exp. 1: metrics for depths

L\!\in\!\{16,32\}

(CIFAR-10, width

\,=128

, mean

\pm 1\sigma

over 5 seeds; stochastic estimation: 100 Rademacher probes, subsample 64).

\bar{\mathcal{R}}(0)

\bar{\mathcal{R}}(L{-}1)

— resonance of nearest/most distant pairs;

\bar{\mathcal{C}}(L{-}1)

— coupling at maximum distance.

$k$	Configuration	$\mathrm{Gap}^{\mathrm{init}}$	$\mathrm{Gap}^{\mathrm{final}}$	$\\|T\\|_{F}^{\mathrm{init}}$	$\\|T\\|_{F}^{\mathrm{final}}$
1	sum+ReLU	$6.5{\cdot}10^{-8}$	$8.7{\cdot}10^{-8}$	$3.6{\cdot}10^{-9}$	$2.5{\cdot}10^{-8}$
1	sum+SiLU	$6.5{\cdot}10^{-8}$	$9.2{\cdot}10^{-8}$	$3.5{\cdot}10^{-9}$	$2.6{\cdot}10^{-8}$
1	cat+ReLU	$8.1{\cdot}10^{-8}$	$1.2{\cdot}10^{-7}$	$6.0{\cdot}10^{-10}$	$5.6{\cdot}10^{-8}$
1	cat+SiLU	$1.35$	$0.078$	$6.6{\cdot}10^{-3}$	$3.3{\cdot}10^{-2}$
2	sum+ReLU	$6.7{\cdot}10^{-8}$	$9.3{\cdot}10^{-8}$	$4.1{\cdot}10^{-9}$	$2.0{\cdot}10^{-8}$
2	sum+SiLU	$7.0{\cdot}10^{-8}$	$9.6{\cdot}10^{-8}$	$4.1{\cdot}10^{-9}$	$2.0{\cdot}10^{-8}$

Appendix

Table N.4: Exp. 4: GN-Gap at the merging node of Diamond MLP for

k\!\in\!\{1,2,3\}

(width

\,=32

, CIFAR-10, mean over 5 seeds).

$d$	Activation	$\\|H^{T}\\|_{F}$	$\\|H^{GN}\\|_{F}$	$\mathrm{Gap}_{GN}$
0	Softplus	$1.50{\cdot}10^{-2}$	$1.22{\cdot}10^{-1}$	$0.123$
0	SiLU	$2.65{\cdot}10^{-2}$	$1.17{\cdot}10^{-1}$	$0.227$
0	GELU	$4.22{\cdot}10^{-2}$	$1.17{\cdot}10^{-1}$	$0.361$
2	Softplus	$1.24{\cdot}10^{-3}$	$1.05{\cdot}10^{-2}$	$0.119$
2	SiLU	$2.15{\cdot}10^{-3}$	$9.59{\cdot}10^{-3}$	$0.224$
2	GELU	$3.45{\cdot}10^{-3}$	$9.70{\cdot}10^{-3}$	$0.357$

Appendix

Table N.5: Exp. 3:

\|H^{T}\|_{F}

and

\|H^{GN}\|_{F}

by distance

d

(init,

L\!=\!6

, width

\,=64

, CIFAR-10, mean over 5 seeds). At

d\!=\!5

the tensor component vanishes (

H^{T}_{d=L-1}\!\equiv\!0

Panel of Figure 23 — Figure N.2: Exp. 4: cross-branch resonance $\bar{\mathcal{R}}_{AB}(d_{\mathrm{graph}})$ in Diamond MLP ( $k\!=\!2$ ). Solid curves — initialization (decay with distance, H4.2a); dashed — after training (growth, H4.2b). All 4 configurations show $R$ growth with $d_{\mathrm{graph}}$ at final ( $3.5$ – $5.7\times$ from $d\!=\!2$ to $d\!=\!4$ ). Error bands: $\pm 1\sigma$ over 5 seeds.

Activation	$\mathrm{Gap}_{\mathrm{exact}}^{\mathrm{init}}$	$\mathrm{Gap}_{\mathrm{stoch}}^{\mathrm{init}}$	$\Delta^{\mathrm{init}}$ (%)	$\Delta^{\mathrm{final}}$ (%)
Softplus	$0.122$	$0.123$	$0.82$	$0.25$
SiLU	$0.225$	$0.226$	$0.46$	$0.33$
GELU	$0.358$	$0.359$	$0.43$	$0.31$

Appendix

Table N.6: Ablation: GN-Gap discrepancy between exact and stochastic (30 probes) estimation (Exp. 3,

L\!=\!6

, width

\,=64

, mean over 5 seeds).

Condition	$d$	$\mathcal{R}(d)$	$\mathcal{C}(d)$	$\mathcal{D}(d)$
ReLU ResNet	0	$1.05$	$1.00$	$2.87$
	1	$0.87$	$0.93$	$2.73$
	2	$0.78$	$0.85$	$2.65$
	3	$0.72$	$0.79$	$2.85$
	4	$0.39$	$0.73$	$2.91$
ReLU Plain	0	$3.81$	$1.00$	$2.52$

Appendix

Table N.7: Exp. 6 (mid, epoch 50): metrics

\mathcal{R}

\mathcal{C}

\mathcal{D}

by distance

d

(mean over 5 seeds). Test accuracy: ReLU ResNet

80.8\,\%

, ReLU Plain

77.0\,\%

, SiLU ResNet

79.9\,\%

, SiLU Plain

78.4\,\%

Checkpoint	Condition	$d\!=\!0$	$d\!=\!1$	$d\!=\!2$	$d\!=\!3$	$d\!=\!4^{\dagger}$
init	ReLU ResNet	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$
	ReLU Plain	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$
	SiLU ResNet	$0.426$	$0.526$	$0.619$	$0.642$	${<}10^{-5}$
	SiLU Plain	$0.003$	$0.028$	$0.080$	$0.132$	${<}10^{-5}$
mid	ReLU ResNet	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$
	ReLU Plain	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$	${<}10^{-5}$

Appendix

Table N.8: Exp. 6: GN-Gap

(d)

across checkpoints (mean over 5 seeds). † At

d\!=\!4

the segment is purely linear (

H^{T}\!\equiv\!0

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 331 papers

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count
May 4, 2026Lurong PanNeural Network Optimization Feedforward Neural Networks
Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
May 22, 2026Anherutowa CalvoNeural Network Optimization Second-Order Optimization
19D Labs
Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training
May 25, 2026Maxim Bolshim, Alexander KugaevskikhNeural Network Memorization Stochastic Approximation
ITMO University, Saint Petersburg, Russia · ITMO University, St. Petersburg, Russia

May 4, 2026J/K move · Enter open · S save

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

Lurong Pan

Neural Network Optimization Feedforward Neural Networks Neural Network Compression

Open paper AskPDF arXiv

Communication Dynamics Neural Networks (CDNNs) apply the circulant-spectral machinery of the Communication Dynamics framework to neural-network layer design. We introduce CDLinear, a block-circulant linear layer with block size B = 2l + 1 that uses 1/B the parameters of a dense layer with the same input and output dimensions. The construction gives an explicit Fourier-domain diagnostic for optimization: for mean-squared loss, the weight Hessian is diagonalized by the discrete Fourier transform, with eigenvalues determined directly by the Fourier spectrum of the input blocks. Under input pre-whitening, the population Hessian condition number is exactly 1, and the empirical condition number is bounded by 1 + O(sqrt(B/N)) for N samples. We implement CDLinear in pure NumPy with hand-derived backward passes and verify gradients by finite differences. On the 8x8 MNIST digits benchmark, across three random seeds, a CDLinear MLP with B = 4 reaches 97.50% +/- 0.23% test accuracy using 2,380 parameters, compared with 98.15% +/- 0.47% for a dense baseline using 8,970 parameters. This gives a 3.8x parameter reduction at a 0.65% accuracy cost. The CD-MLP's mean Hessian condition number is 1.9e4, about 310x smaller than the dense baseline's 5.9e6. We position CDLinear as a special case of structured matrix neural-network layers, with the main contributions being a closed-form Hessian-spectrum diagnostic, a principled discrete sequence of block multiplicities, and an explicit conditioning analysis. We also release a reference PyTorch implementation integrating CDLinear into a DeepSeek-V3-style mixture-of-experts transformer for future large-scale benchmarks.

Abstract

Neural Network Optimization

Explore similar work

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training