JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

Biswa Sengupta; Jinhua Wang; Leo Brunswic

cs.LGFeb 20, 2026

JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

Authors: Biswa Sengupta, Jinhua Wang, Leo Brunswic

Organizations: LLM Suite Team, JP Morgan Chase & Co.

Abstract

Recent advances in deep learning, exemplified by Hyper-Connections (HC), have expanded the residual connection paradigm by introducing wider residual streams and diverse connectivity patterns. While these innovations yield significant performance gains, they compromise the identity mapping property of residual connections, leading to training instability, limited scalability, and increased memory overhead. To address these challenges, we propose JPmHC (Jacobian-spectrum Preserving manifold-constrained Hyper-Connections), a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams while explicitly controlling gradient conditioning. By constraining the mixer M on operator-norm-bounded manifolds (e.g., bistochastic, Stiefel, Grassmann), JPmHC prevents gradient pathologies and enhances stability. JPmHC introduces three key contributions: (i) a free-probability analysis that predicts Jacobian spectra for structured skips, providing actionable design rules for mixer selection; (ii) memory-efficient implicit differentiation for fixed-point projections, reducing activation memory and synchronization overhead; and (iii) a Stiefel-constrained mixer via Cayley transforms, ensuring orthogonality without post-hoc normalization. Empirical evaluations on ARC-AGI demonstrate that JPmHC achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines, with a rank- $p$ Grassmannian variant tracking between the two -- consistent with the spectral theory predictions. As a flexible and scalable extension of HC, JPmHC advances spectrum-aware, stable, and efficient deep learning, offering insights into topological architecture design and foundational model evolution. \newline \newline

Figures & tables

Figure 1: Scalar and OV theories vs. Monte Carlo singular value densities. Panels show four skip-connection types ( $n=4$ streams, $p=25$ per stream, $c_{2}L=0.05$ , $\eta=0.02$ , $500$ samples) at depths $L\in\{1,2,10\}$ (rows) for mixers $A_{n}\in\{\text{Identity},\,\text{Bistochastic},\,\text{Orthogonal},\}$ (columns). The scalar Dyson prediction (dotted orange curve) matches Monte Carlo histograms at $L=1$ for all cases. At $L\in{2,10}$ , bistochastic and Gaussian mixers develop mass near zero (spectral collapse), while orthogonal mixers preserve dynamical isometry. Scalar theory fails while Operator-value theory is able to fully catch the spectrum, regularization parameter $\eta$ used to reduce numerical instabilities smooth out the distribution: it pushes it away from zero and reduces the spikes thus increases the mass allocated to the 1.0 mode. The scaling $Lc_{2}=\text{const}$ ensures weights $W^{l}\sim\mathcal{N}(0,\sigma_{w}^{2}/L)$ maintain constant forward signal variance. This normalization is shown to be accurate as spectra have a main mode bounded away from 0 an infinity.

Panel of Figure 2 — Figure 2: Hyper-Connected Transformer Encoder Block. Block 1 (Multi-Head Attention, bottom) feeds into Block 2 (Feed-Forward, top). Within each block the input forks into a twisted skip path (thick arrows, $H_{\mathrm{res}}$ ) and a compute path (thin arrows) that projects $q{\to}1$ streams, applies the layer, expands $1{\to}q$ , and gates via $H_{\mathrm{out}}$ ; both paths merge at the $+$ node before Layer Norm.

Panel of Figure 3 — Figure 3: Two representative ARC-AGI tasks. Each task requires discovering a latent rule (here, pattern tiling and region filling) from a few demonstrations, then applying it to a novel test input.

Variant	Manifold	Key param	FLOPs/module	Norm	Constraint
Sinkhorn	Birkhoff polytope	$T{=}20$ , $k{=}16$	576	RMSNorm	sigmoid
Cayley	Stiefel $O(n)$	$s{=}2$ , $\alpha{=}0.1$	256	LayerNorm	softmax
Grassmann	$\mathrm{Gr}(n,p)$	$p{=}2$ , $\alpha{=}0.01$	—	LayerNorm	softmax

Table 1: JPmHC variant configurations and per-module compute cost.

Metric	Sinkhorn	Cayley	Grassmann ∗	Cayley/Sink
Training Steps	510,951	515,902	110,756	—
Eval Accuracy	86.5%	86.9%	82.2%	$1.00\times$
Exact Accuracy (greedy)	27.9%	31.4%	12.8%	$1.13\times$
Pass@1	36.5%	40.5%	27.5%	$1.11\times$
Pass@2	41.7%	45.4%	31.2%	$1.09\times$
Pass@5	45.5%	50.5%	35.9%	$1.11\times$

Table 2: ARC-AGI concept evaluation metrics (best observed per metric). Cayley and Sinkhorn have completed training (

{\sim}516

K and

{\sim}511

K steps respectively); Grassmann training is ongoing (

{\sim}111

K steps). ∗ Grassmann numbers are preliminary.

Step	Exact Acc.	Pass@1	Pass@2	Pass@10	Pass@100	Pass@1000
68,951	10.6%	—	—	—	—	—
81,903	12.7%	—	—	—	—	—
94,854	15.7%	—	—	—	—	—
107,805	17.6%	—	—	—	—	—
161,952	23.4%	35.4%	40.1%	47.4%	52.7%	54.1%
174,904	24.5%	35.6%	40.3%	47.7%	53.9%	57.1%

Table 3: Cayley evaluation metrics at each checkpoint. Bold indicates the best value for each metric. Training ran for

{\sim}516

K steps.

Step	Exact Acc.	Pass@1	Pass@2	Pass@10	Pass@100	Pass@1000
52,951	1.4%	5.2%	7.5%	12.1%	14.9%	16.1%
65,902	3.1%	9.0%	11.6%	16.2%	21.4%	22.6%
78,854	5.0%	11.6%	16.2%	21.1%	26.8%	29.5%
99,951	8.5%	19.1%	22.8%	29.4%	36.6%	39.6%
112,902	9.9%	22.2%	26.2%	33.1%	40.6%	43.9%
125,853	11.2%	23.7%	28.0%	35.5%	42.8%	45.5%

Table 4: Sinkhorn evaluation metrics at each checkpoint. Bold indicates the best value for each metric. Training ran for

{\sim}511

K steps; Sinkhorn continued to improve in exact-match accuracy through the final checkpoint.

Step	Exact Acc.	Pass@1	Pass@2	Pass@10	Pass@100	Pass@1000
12,951	0.2%	0.6%	1.3%	3.2%	3.9%	4.6%
28,951	0.8%	3.6%	5.5%	9.9%	12.4%	14.0%
41,902	2.6%	8.9%	12.1%	16.0%	18.6%	21.1%
58,951	5.5%	15.7%	18.6%	24.0%	29.4%	31.5%
71,902	7.4%	20.0%	24.1%	30.4%	35.7%	38.4%
84,854	9.6%	22.2%	27.3%	34.7%	40.4%	43.8%

Table 5: Grassmann evaluation metrics at each checkpoint (training ongoing at

{\sim}111

K steps). Bold indicates the best value for each metric. At matched step counts, Grassmann tracks ahead of Sinkhorn’s early trajectory (cf. Sinkhorn at 113K: 9.9% exact, 22.2% pass@1).

Panel of Figure 4 — Figure 4: Pass@ $k$ scaling comparison. Cayley consistently outperforms Sinkhorn across all sampling budgets $k$ . Grassmann results are preliminary ( ${\sim}111$ K steps vs. ${\sim}500$ K+ for others); at matched step counts, Grassmann tracks ahead of Sinkhorn’s early trajectory. The gap between Cayley and Sinkhorn narrows at higher $k$ , indicating Sinkhorn has higher prediction variance.

Panel of Figure 5 — Figure 5: Evaluation accuracy curves. (a) Per-token accuracy shows Cayley and Sinkhorn both exceeding 86% at convergence, with Grassmann tracking a steep early trajectory. (b) Exact-match accuracy reveals a persistent gap: Cayley plateaus at ${\sim}31$ % while Sinkhorn saturates at ${\sim}28$ %. Grassmann at 111K steps (12.8%) tracks ahead of Sinkhorn’s comparable point (9.9% at 113K). ∗ Grassmann training is ongoing.

Panel of Figure 6 — Figure 6: Evaluation LM loss (per-token cross-entropy, lower is better). The Cayley variant achieves the lowest loss ( $0.643$ at 419K steps) with a $1.27\times$ advantage over Sinkhorn’s best ( $0.817$ ). Grassmann’s steep descent suggests it may approach Sinkhorn-level loss with continued training. Both Cayley and Sinkhorn show mild loss increase after their respective optima, suggesting slight overfitting at late training stages.

Variant	Forward FLOPs	Backward FLOPs	Total per module
Sinkhorn (implicit)	$\mathcal{O}(Tn^{2})=320$	$\mathcal{O}(kn^{2})=256$	576
Cayley	$\mathcal{O}(sn^{3})=128$	$\mathcal{O}(sn^{3})=128$	256
Grassmann	$\mathcal{O}(np)=8$	$\mathcal{O}(n^{3})^{\dagger}=64$	72

Table 6: Forward and backward pass compute estimates per JPmHC module.

Feature	HC / mHC	JPmHC (Ours)
Mixing parameterization	Learned $n\times n$ / Sinkhorn	Stiefel, Grassmann, Birkhoff
Manifold constraint	Birkhoff polytope only	Stiefel, Grassmann, Birkhoff
Implicit differentiation	—	Sinkhorn backward
Riemannian optimization	—	Grassmann (Cayley ADAM)
Spectral analysis	—	Generating set selection
CUDA graph compatible	Not addressed	All variants

Table 7: Feature comparison with HC and mHC.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$n$	$n!$	$\binom{n!}{4}$	Matrix size	Feasibility
3	6	15	$6\times 6$	$<0.01$ s
4	24	10,626	$24\times 24$	$\sim 0.3$ s
5	120	8,214,570	$120\times 120$	$\sim 1$ hour
6	720	$\sim 10^{10}$	$720\times 720$	Intractable

Appendix

Table 8: Tractability of exhaustive spectral gap search.

Variant	$D_{\text{res}}$	Total params	Notes
Sinkhorn	$n^{2}=16$	$(n+n+16)\times nd+36=49{,}188$	Fused $\phi$
Cayley	$3n^{2}=48$	$48\times nd+\text{LayerNorm}=102{,}400$	Fused $\phi$ + LN
Grassmann	$np=8$	$(n+n+8)\times nd+28=32{,}796$	$p=2$
Perm Mix	$K=6$	$(n+n+6)\times nd+22=28{,}694$	+ perm indices

Appendix

Table 9: Parameter count per JPmHC module (

n=4

d=512

nd=2048

Iterations $k$	Relative residual $\\|\mathbf{r}\\|/\\|\mathbf{r}_{0}\\|$	Gradient error (%)
1	$7.5\times 10^{-1}$	43.2
2	$5.6\times 10^{-1}$	28.1
4	$3.2\times 10^{-1}$	11.5
8	$1.0\times 10^{-1}$	2.8
16	$1.0\times 10^{-2}$	0.08
32	$1.0\times 10^{-4}$	$<0.001$

Appendix

Table 10: Gauss-Seidel convergence for

n=4

with typical

\mathbf{P}

matrices.

Parameter	Value
Total parameters	$\sim$ 7M
Hidden dimension $d$	512
Number of streams $n$	4
Effective hidden dim $nd$	2048
Unique transformer layers (weight-tied)	2
Recursive cycles per layer	6

Appendix

Table 11: Full TRM architecture configuration.

Hyperparameter	Value
Optimizer	AdamAtan2 [ Kunstner et al., 2023 ]
Learning rate	$1\times 10^{-4}$
Global batch size	768
Weight decay	0.1
Gradient clipping	1.0
LR schedule	Step decay at 80% and 90% of training

Appendix

Table 12: Training hyperparameters (shared across all variants).

$n$	2	3	4	5	6	8
$\rho=1-1/n$	0.500	0.667	0.750	0.800	0.833	0.875
Iterations $k$	7	12	16	21	26	37

Appendix

Table 13: Required Gauss-Seidel iterations for

\epsilon=0.01

convergence.

	Standard Sinkhorn	Implicit Sinkhorn (Ours)
Autograd nodes	$\mathcal{O}(T\cdot n^{2})$ ( $\sim\!128$ K)	$\mathcal{O}(1)$ (constant)
Backward kernels	$\sim\!128$ K microsecond-scale	$\sim\!20$ millisecond-scale
DDP overlap	Poor (55% stalls)	Excellent
Memory (saved tensors)	$\mathcal{O}(T\cdot n^{2})$	$\mathcal{O}(n^{2})$ (just $\mathbf{P}$ )
Gradient accuracy	Exact	$\epsilon\leq 0.01$ (controlled)

Appendix

Table 14: Backward pass complexity: standard vs. implicit Sinkhorn.

Property	Doubly-Stochastic ( $\mathcal{B}_{n}$ )	Orthonormal ( $O(n)$ )
Norm behavior	Contractive ( $\\|\mathbf{H}\mathbf{x}\\|\leq\\|\mathbf{x}\\|$ )	Preserving ( $\\|\mathbf{H}\mathbf{x}\\|=\\|\mathbf{x}\\|$ )
Entries	Non-negative	Unconstrained sign
Row/col sums	Both equal $\mathbf{1}$	Not constrained
$\det(\mathbf{H})$	$\in[0,1]$	$\pm 1$
Convex hull	Permutation matrices	Not a convex set
Gradient flow	May attenuate	Preserved

Appendix

Table 15: Properties of doubly-stochastic (Sinkhorn) vs. orthonormal (Cayley) mixing.

Variant	Parameters	$n=4$
Sinkhorn (full $n\times n$ )	$n^{2}$	16
Cayley (full $n\times n$ )	$n^{2}$	16
Grassmannian ( $n\times p$ , $p=n/2$ )	$np$	8

Appendix

Table 16: Parameter count for residual mixing matrix (

n=4

Hyper-Connections

Hyper-connections expand neural-network residual paths into multiple streams with learned routing and mixing to improve capacity, stability, and efficiency.

See all 18 papers

Spectral-Sphere-Constrained Hyper-Connections
Mar 21, 2026Zhaoyi Liu, Haichuan Zhang, Ang LiHyper-Connections Residual Learning
University of Maryland, College Park · University of Utah
go- $m$ HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices
Apr 2, 2026Torque Dandachi, Sophia Diggs-Galligan
Independent, San Francisco, United States of America.
Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning
Aug 6, 2026Zezhong Jin, Xiaoyu Wang, Zhe Li +4Audio Representation Learning Residual Learning
Dept. of EEE, The Hong Kong Polytechnic University · Baidu Inc. · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR

Mar 21, 2026J/K move · Enter open · S save

Spectral-Sphere-Constrained Hyper-Connections

Zhaoyi Liu, Haichuan Zhang, Ang Li

University of Maryland, College Park · University of Utah

Hyper-Connections Residual Learning Constrained Optimization

Open paper AskPDF arXiv

Hyper-Connections (HC) extend residual connections into multiple streams, employing residual matrices for cross-stream mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variants restrict these matrices to be doubly stochastic via Sinkhorn-Knopp (SK) algorithm or permutation-based parameterizations. We reveal three limitations of this doubly stochastic constraint: (1) identity degeneration, where learned matrices collapse around the identity initialization and diminish cross-stream interactions, (2) a expressivity bottleneck, where the doubly stochastic constraint restricts the freedom of the subdominant spectrum of the residual matrices, preventing the model from selectively preserving or attenuating cross-stream variations, and (3) parameterization inefficiencies, manifesting as unstable SK iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections (

\mathrm{s}^{2}

HC). By confining residual matrices to a spectral norm sphere,

\mathrm{s}^{2}

HC restores free control over the subdominant spectrum, enabling the model to selectively preserve or attenuate cross-stream variations. This shift eliminates unstable SK iterations and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.

Abstract

Hyper-Connections

Explore similar work

Spectral-Sphere-Constrained Hyper-Connections

go-mmmHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices

Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning

go- $m$ HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices