go-$m$HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices

Torque Dandachi; Sophia Diggs-Galligan

cs.LGApr 2, 2026

go- $m$ HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices

Authors: Torque Dandachi, Sophia Diggs-Galligan

Organizations: Independent, San Francisco, United States of America.

Abstract

Doubly stochastic matrices enable learned mixing across residual streams, but parameterizing the set of doubly stochastic matrices (the Birkhoff polytope) exactly and efficiently remains an open challenge. Existing exact methods scale factorially with the number of streams ( $d$ ), while Kronecker-factorized approaches are efficient but expressivity-limited. We introduce a novel exact parameterization grounded in the theory of generalized orthostochastic matrices, which scales as $\mathcal{O}(d^3)$ and exposes a single hyperparameter $s$ which continuously interpolates between a computationally efficient boundary and the fully expressive Birkhoff polytope. Building on Manifold-Constrained Hyper-Connections ( $m$ HC), a framework for learned dynamic layer connectivity, we instantiate this parameterization in go- $m$ HC. Our method composes naturally with Kronecker-factorized methods, substantially recovering expressivity at similar FLOP costs. Spectral analysis indicates that go- $m$ HC fills the Birkhoff polytope far more completely than Kronecker-factorized baselines. On synthetic stream-mixing tasks, go- $m$ HC achieves the minimum theoretical loss while converging up to $10\times$ faster. We validate our approach in a 30M parameter GPT-style language model. The expressivity, efficiency, and exactness of go- $m$ HC offer a practical avenue for scaling $d$ as a new dimension of model capacity.

Figures & tables

Figure 1: Spectral analysis and architectural integration of go- m HC. (Left) Spectral Comparison: Illustration of spectral reach and approximation gaps across several manifold parameterization methods. m HC (and the Sinkhorn-Knopp algorithm) is not guaranteed to always remain within the manifold after finite iterations iterations (red points). go- m HC (Ours) closely approximates the Birkhoff polytope $\mathsf{B}_{d}$ within an $\epsilon(s)$ error margin using only $\frac{d(d-1)}{2}$ parameters. m HC-lite provides full coverage but requires $d!$ parameters, while KromHC is restricted to tensors of $\mathsf{B}_{2}$ , disallowing non-adjacent cycles of length $>2$ and severely restricting the space of learnable matrices. (Middle) Parameterization Pipeline: Our method maps learned skew-symmetric parameters $\mathbf{X}=-\mathbf{X}^{\top}\in\mathbb{R}^{ds\times ds}$ to an orthogonal matrix $\mathbf{Q}\in\mathcal{Q}(ds)$ via the Cayley Transform $(\mathbf{I}-\mathbf{X})(\mathbf{I}+\mathbf{X})^{-1}$ . The final doubly stochastic matrix $\mathbf{B}\in\mathbb{R}^{d\times d}$ is obtained by a block-wise Frobenius norm projection $\frac{1}{s}\|\mathbf{Q}_{ij}\|_{F}^{2}$ . (Right) Hyper-Connections: The transformation $\mathcal{H}^{\text{res}}$ is integrated into the residual stream $\mathbf{x}_{l+1}=\mathcal{H}_{l}^{\text{res}}\mathbf{x}_{l}+\text{Layer }\mathcal{F}(\mathbf{x}_{l})$ , where the doubly stochastic constraint ensures training stability and improved gradient flow.

Figure 2: Illustration of the Karpelevič region – the spectrum of stochastic matrices – for

\mathsf{B}_{d}

with

d=2\text{ to }d=5

. The black circle corresponds to the unit disc and the red points are the eigenvalues

\omega

of the

d\times d

permutation matrices contained in the polytope. The affine space defined by doubly stochastic matrices is strictly contained within the volume of the blue region, except for in

\mathsf{B}_{2}

where the space is a line. The spectrum of

\mathsf{B}_{d}

is the union of the spectrum of

\mathsf{B}_{d-1}

and the polygon with

d

sides inscribed on the unit disc.

Figure 3: Histogram of the spectral reach of a toy model implementing the map $\mathcal{P}_{\text{go}}$ to $\mathsf{B}_{3}$ ( $d=3$ ) for different values of $s\in\{1,2\}$ on random targets. (a) When $s=1$ , the $d=3$ model is able to learn maps with full spectral coverage within the hypocycloid–covering the triangular region (Karpelevič region of $\mathsf{B}_{3}\setminus\mathsf{B}_{2}$ ) up to error $\epsilon(s=1)$ , as well all real eigenvalues on the interval $[-1,1]$ (Karpelevič region of $\mathsf{B}_{2}$ ). (b) $s=2$ covers a larger region, with a smaller gap in the spectra $\epsilon(s=2)<\epsilon(s=1)$ .

Figure 4: Histogram of the spectral reach of a toy model implementing the maps $\mathcal{P}_{\text{SK}}$ , $\mathcal{P}_{\text{lite}}$ , $\mathcal{P}_{\text{krom}}$ , and $\mathcal{P}_{\text{go}}$ on random targets in $\mathsf{B}_{4}$ . The blue and green lines in (a-e) are the boundary of the spectrum of the $\mathsf{B}_{3}\setminus\mathsf{B}_{2}$ and $\mathsf{B}_{4}\setminus\mathsf{B}_{3}$ polytopes respectively, as illustrated in figure 2 . (a) m HC(implemented via $\mathcal{P}_{\text{SK}}$ ) contains points outside of $\mathsf{B}_{4}$ (this effect is exaggerated for illustration via initial conditions with large variance). (b) m HC-lite shows good expressivity, filling the region near the boundary almost completely, without extending past it. (c) KromHC fails to represent the complex region inside of $\mathsf{B}_{4}$ – since the “shadow” is $1$ -dimensional, it fills an infinitesimal slice of the full polytope. (d) For $s=1$ , we see go- m HCrepresent a sizeable portion of the region, although it cannot represent all matrices near the boundary of the polytope. (e) With $s=2$ , go- m HCbegins to represent larger regions of $\mathsf{B}_{4}$ , filling the region more densely.

Figure 5: Comparing the total number of learnable parameters in m HC with its exact variants. go- m HC (green), same scaling as m HC regardless of

s

, with

s=1

requiring less parameters due to the skew-symmetry. m HC-lite (orange) scales with

\mathcal{O}(d!)

despite its exactness and completeness over

\mathsf{B}_{d}

, blowing up beyond

d=8

and becoming intractable. KromHC (red) scales the fastest

\mathcal{O}(d\log d)

faster than m HC by trading off expressivity and imposing a strong symmetry on residual stream interactions. KromHC relies on the factorization of

d

and we therefore only analyze the best case parameter count when

d

is a power of 2.

Method	Parameter Count
m HC	$\color[rgb]{0,0,1}(dC+1)d^{2}+2d^{2}C+2d+3$
go- m HC ( $s$ )	$\color[rgb]{0,0,1}(dC+1)\frac{ds(ds-1)}{2}+2d^{2}C+2d+3$
m HC-lite	$\color[rgb]{0,0,1}(dC+1)d!+2d^{2}C+2d+3$
KromHC	$\color[rgb]{0,0,1}(dC+1)\sum_{k=1}^{K}(i_{k}!)+2d^{2}C+2d+3$

Table 1: Total Parameter Counts

Method	FLOP complexity (per layer)
m HC	$\mathcal{O}\left(C\cdot S\cdot d^{2}\right)$
go- m HC ( $s$ )	$\mathcal{O}\left(C\cdot s^{3}\cdot d^{3}\right)$
m HC-lite	$\mathcal{O}\left(C\cdot d^{2}\cdot d!\right)$
KromHC	$\mathcal{O}\left(C\cdot\log_{i_{k}}(d)\cdot i_{k}^{2}\cdot(i_{k}!)+C\cdot d^{2}\right)$

Table 2: Theoretical FLOP complexity per layer. FLOP counts demonstrate that go- m HC provides a practical middle ground for scaling residual streams, bypassing the superexponential cost of m HC-lite and the iterative overhead of SK iterations in m HC, with

S

being the number of SK iterations. Nominal values for the constants below are:

S=20,i_{k}=2

, and

s\in\{1,2\}

. See appendix B.6 for our full FLOP estimates.

Figure 6: A representative sample of a training curve on a synthetic $4$ -stream mixing task with a fixed learning rate $\eta=10^{-3}$ , a batch size of $C=64$ , dataset size $|\mathcal{X}|=100$ and noise mangitude $\epsilon=10^{-1}$ . We see that both m HC-lite (orange) and KromHC (red) take longer to converge than go- m HC (greens). We also note that go- m HC with $s\geq 2$ saturates at the same loss as that of m HC-lite, which happens to be the theoretical minimum achievable loss due to the noise term in equation 15 . KromHC has the worst performance due to its expressivity-speed trade-off and go- m HC with $s=1$ offers a loss in-between that of KromHC and m HC-lite or its $s\geq 2$ counterparts.

Figure 7: Phase-space comparison of training trajectories under varying noise regimes using the toy model in appendix A.5.2 . We plot the MSE of m HC-lite against go- m HC across four orders of magnitude of noise magnitude (color-coded). Each curve represents a training trajectory starting from the top-right (high loss) and progressing toward the bottom-left (convergence). The dashed diagonal ( $y=x$ ) separates regions of relative performance; trajectories residing in the upper-left green shaded region indicate that go- m HC achieves a lower loss than m HC-lite at equivalent training stages. As noise magnitude decreases, go- m HC maintains a consistent performance advantage, converging to the optimal loss floor significantly faster than the lite variant. See figure A6 for the individual loss curves.

Figure 8: Comparison of convergence speed and final loss across different m HC architectures as a function of the number of residual streams

d

. Top Panel : Convergence efficiency measured by Epochs to Convergence across varying

d

. While KromHC (red, dashed) show relatively stable or slightly increasing epoch counts for small

d

, go- m HC (green) exhibits a steady increase in training time as the number of residual streams grows. An inset displays the loss trajectory for

d=32

, highlighting that even though go- m HC takes longer, it achieves a significantly lower error floor than KromHC despite the longer convergence time. We did not generate m HC-lite data points beyond

d=7

due to its factorial scaling, but suspect it follows a similar trend to that of go- m HC. Bottom Panel : Scalability of the Final MSE Loss. Both go- m HC and m HC-lite consistently reach the theoretical minimum (dotted line) across the tested range. In contrast, KromHC suffers from severe performance degradation as

d

increases, with the final loss diverging significantly from the optimal baseline, we find that it scales as

\mathcal{O}(d)

regardless of

i_{k}

in appendix B.3.2 .

Figure 9: Gradient norm of different Hyper-Connection (HC) variants. We show the global gradient norm for a 30M parameter GPT-style transformer, trained on TinyStories. Our training runs indicate that all m HC variants exhibit similarly convergent behavior with well-defined gradients, with a substantial improvement over HC, which suffers from unstable gradients. The training loss values remain within a negligible margin of one another (appendix G.2 ).

Model	Grammar	Creativity	Consistency
Residual	$6.20\pm 0.11$	$\underline{5.87\pm 0.09}$	$4.83\pm 0.09$
m HC-lite	$\underline{6.58\pm 0.12}$	$5.80\pm 0.07$	$\underline{5.13\pm 0.11}$
KromHC	$\underline{6.56\pm 0.13}$	$5.79\pm 0.07$	$5.07\pm 0.11$
go- m HC ( $s=1$ )	$\underline{6.57\pm 0.11}$	$5.74\pm 0.07$	$\underline{5.12\pm 0.09}$
go- m HC ( $s=2$ )	$\mathbf{6.63\pm 0.12}$	$\mathbf{5.88\pm 0.06}$	$\mathbf{5.22\pm 0.11}$
ICC(3,1)	0.890	0.671	0.825

Table 3: Evaluation with GPT-4.1. Results reflect the mean and standard error across

N=70

samples, where each sample score is the consensus of 3 independent trials. Bold indicates the maximum value; underline indicates scores within one standard error of the maximum. ICC(3,1) denotes intra-judge reliability across the 3 trials per sample.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Ablation of figure 7 with SGD instead of Adam. We used the same hyperparameters and only changed $\eta=0.1$ . Our method remains faster at converging with a different non-momentum based optimizer.

Figure A2 : Loss curves for our methods and baselines for a target matrices $\{\mathbf{T_{i}}\}$ generated using 6 different parameterizations. We see that regardless of the distribution, our method maintains it’s advantage – in convergence and expressivity.

Figure A4 : Convergence dynamics across the number of residual streams ( $d$ ) and residual stream configurations ( $s$ ). Each panel illustrates the training trajectory (MSE vs. Iterations) for dimensions $d\in[2,10]$ . We compare the baseline $m$ HC-lite and KromHC against three configurations of go- $m$ HC. While KromHC (red) and $m$ HC-lite (orange) exhibit slower convergence and higher error floors as $d$ increases, go- $m$ HC demonstrates robust scalability. Notably, increasing the number of residual streams ( $s=2,3$ ) in go- $m$ HC accelerates convergence speed and improves final accuracy. We ran KromHC for powers of $2$ only, as this constitutes the regime where KromHC is fastest, if $i_{k}\neq 2$ , the performance interpolates to that of m HC-lite in the limit. We don’t run m HC-lite for $d\geq 7$ due to the factorial scaling of the model.

Figure A5 : Similar to figure A4 , sweeping sparsity $S\in\{0,0.5,0.9\}$ . We see that sparsity doesn’t change the overall trend, and only rescales loss due to the sparse vectors having smaller contributions.

Figure A6 : Averaged loss trajectories for m HC-lite and go- m HC for our toy model with different noise levels. They both achieve the same final loss, with go- m HCconsistently reaching that value $10\times$ faster for a fixed learning rate $\eta=0.001$ across noise levels.

Figure A7 :

Figure A8 : The minimum achievable error and convergence properties of KromHC as a function of $d$ and $i_{k}$ . We find that the dynamics and results are mostly independent of choice of $i_{k}$ . Namely, the approximation error of a matrix in $\mathsf{B}_{d}$ from a (product of) matrices in $\mathsf{B}_{q}$ with $q<d$ will have an MSE that scales with $d$ , with little-to-no dependence on $q$ ( $i_{k}$ ).

Figure A9 : Random doubly stochastic matrice exhibit depth decoupling. Tracking the second eigenvalue, we see that the residual stream product tends towards the barycenter. Increasing $s$ allows for more flexibility of having less overlap of matrices. This is not surprising since we are sampling a sum of permutation matrices that are decoupled from each other.

Method	Approximate FLOPs
m HC	$4CSd^{2}$
go- m HC ( $s$ )	$\frac{8}{3}C(ds)^{3}+4C(ds)^{2}$
m HC-lite	$3C(d!)+2Cd^{2}(d!)$
KromHC	$3CK(i_{k}!)+2CKi_{k}^{2}(i_{k}!)+C\left(\frac{i_{k}^{2K+2}-i_{k}^{4}}{i_{k}^{2}-1}\right)$

Appendix

Table 4:

Figure A10 : We additionally show loss curves that include m HCand another variant of m HCthat transforms matrices linearly. We argue that the linear transformation is preferable in the context of our toy model due to faster convergence and the absence of vanishing gradients. We note that the speed up we see is similar to that of go- m HCover the other methods.

Figure A11 : Sensitivity of convergence trajectories to the inclusion of a stream read term $\mathcal{H}^{\text{pre}}$ which can break KromHC’s symmetry. We compare the impact of the learning signal being generated after an $\mathcal{H}^{\text{pre}}$ projection (dashed lines) against the signal originating from within the residual stream only (solid lines). While the inclusion of a learned $\mathcal{H}^{\text{pre}}$ allows for symmetry breaking in KromHC, it lowers the overall MSE floor (shaded regions) for all methods due to it’s ability to sample outside the set of stochastic matrices – which seems to be the dominant effect here. The relative performance hierarchy across all models remains invariant to the choice of toy model. In both configurations, go- m HC(green) demonstrates the fastest convergence and lowest final loss, whereas KromHC (red) consistently exhibits the highest error and, along with m HC-lite, the slowest training dynamics. This suggests that the advantages of the our proposed method are independent of the read/write symmetry-breaking in Hyper Connections.

Figure A12 : Numerically calculated projection on the 3-cycle

I,\sigma,\sigma^{2}

with

\sigma

being a cyclic permutation matrix (

\sigma^{3}=1

Figure A13 : Visualizations of the Birkhoff Polytope $\mathsf{B}_{d}$ . The green region denotes the support of the $1$ -orthostochastic matrices and the pink region is the more general coverage of the Birkhoff polytope that isn’t covered when $s=1$ .

Figure A15 :

Figure A16 :

Figure A17 :

Category	Hyperparameter	Value
Model	Layers	6
	Attention Heads	6
	Embedding Dimension	384
	Context Window	512
	Dropout	0.1
Optimization	Peak Learning Rate	$1\times 10^{-3}$

Appendix

Table 5: Hyperparameter configurations for training on the TinyStories dataset.

Figure A19 :

Figure A20 : A version of figures A18 & A19 with

d=8

for KromHC and go- m HC(

s=1

Model	Grammar	Creativity	Consistency
Residual	$5.93\pm 0.12$	$\underline{5.84\pm 0.06}$	$4.58\pm 0.09$
m HC-lite	$\mathbf{6.37\pm 0.13}$	$\mathbf{5.90\pm 0.06}$	$4.73\pm 0.08$
KromHC	$6.04\pm 0.13$	$5.82\pm 0.06$	$4.57\pm 0.09$
go- m HC ( $s=1$ )	$\underline{6.26\pm 0.13}$	$\underline{5.88\pm 0.07}$	$4.64\pm 0.08$
go- m HC ( $s=2$ )	$\underline{6.34\pm 0.14}$	$\underline{5.90\pm 0.06}$	$\mathbf{4.90\pm 0.10}$
ICC(3,1)	0.893	0.555	0.710

Appendix

Table 6: Evaluation Performance: GPT-4.1 Mini. Scores represent mean

\pm

SEM (

N=70

). Bold indicates the maximum; underline indicates scores within one standard error of the maximum.

Model	Grammar	Creativity	Consistency
Residual	$6.20\pm 0.10$	$\underline{6.29\pm 0.09}$	$5.47\pm 0.10$
m HC-lite	$\mathbf{6.47\pm 0.10}$	$\mathbf{6.33\pm 0.07}$	$\mathbf{5.82\pm 0.09}$
KromHC	$6.26\pm 0.11$	$6.25\pm 0.08$	$5.63\pm 0.10$
go- m HC ( $s=1$ )	$\underline{6.45\pm 0.10}$	$\underline{6.30\pm 0.10}$	$5.70\pm 0.09$
go- m HC ( $s=2$ )	$\underline{6.43\pm 0.11}$	$6.25\pm 0.08$	$\underline{5.80\pm 0.11}$
ICC(3,1)	0.834	0.605	0.685

Appendix

Table 7: Evaluation Performance: GPT-4.1 Nano. Scores represent mean

\pm

SEM (

N=70

). Bold indicates the maximum; underline indicates scores within one standard error of the maximum.

Judge Pair	Metric	Grammar	Creativity	Consistency
4.1 vs. 4.1 mini	Spearman $\rho$	0.732	0.345	0.519
	Cross-Model ICC	0.817	0.535	0.639
4.1 vs. 4.1 nano	Spearman $\rho$	0.603	0.302	0.467
	Cross-Model ICC	0.748	0.405	0.567
4.1 mini vs. 4.1 nano	Spearman $\rho$	0.682	0.201	0.434
	Cross-Model ICC	0.786	0.318	0.393

Appendix

Table 8: Inter-Judge Alignment. Pairwise comparison of rating consistency across

N=70

samples. Spearman

\rho

measures rank-order correlation, while Cross-Model ICC(3,1) assesses the absolute agreement between different model tiers acting as judges.

Model Variant	Config	$\Delta$ Parameters (M)	Total Parameters (M)
Residual Baseline	—	0.00	30.14
m HC-lite	$d=4$	0.60	30.74
KromHC	$d=4$	0.23	30.37
go- m HC (s=1)	$d=4$	0.46	30.60
go- m HC (s=2)	$d=4$	1.34	31.48
KromHC	$d=8$	0.84	30.98

Appendix

Table 9: Model Parameter Overhead (

\Delta

) Grouped by Dimension

d

TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes
May 20, 2026Anton LyubininHyper-Connections
Spectral-Sphere-Constrained Hyper-Connections
Mar 21, 2026Zhaoyi Liu, Haichuan Zhang, Ang LiHyper-Connections Residual Learning
University of Maryland, College Park · University of Utah
JPmHC Dynamical Isometry via Orthogonal Hyper-Connections
Feb 20, 2026Biswa Sengupta, Jinhua Wang, Leo BrunswicHyper-Connections Residual Learning
LLM Suite Team, JP Morgan Chase & Co.

May 20, 2026J/K move · Enter open · S save

TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

Anton Lyubinin

Hyper-Connections

Open paper AskPDF arXiv

Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams, but unconstrained mixing leads to training instability. Manifold-Constrained Hyper-Connections (mHC) address this by enforcing approximate double stochasticity via Sinkhorn normalization, while mHC-lite ensures exact constraints through convex combinations of permutation matrices at the cost of factorial complexity. KromHC reduces this cost using Kronecker-product parameterizations, but restricts the mixing matrices to a structured submanifold of the Birkhoff polytope . We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP), which construct exactly doubly stochastic mixing matrices with

(n-1)^2

degrees of freedom. Our approach avoids iterative normalization and combinatorial explosion while preserving full expressivity of the Birkhoff polytope. Empirical results on language model pre-training' demonstrate competitive performance with improved stability and scalability.

Abstract

Explore similar work

TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

Spectral-Sphere-Constrained Hyper-Connections

JPmHC Dynamical Isometry via Orthogonal Hyper-Connections