Spectral-Sphere-Constrained Hyper-Connections

Zhaoyi Liu; Haichuan Zhang; Ang Li

cs.LGMar 21, 2026

Spectral-Sphere-Constrained Hyper-Connections

Authors: Zhaoyi Liu, Haichuan Zhang, Ang Li

Organizations: University of Maryland, College Park · University of Utah

Abstract

Hyper-Connections (HC) extend residual connections into multiple streams, employing residual matrices for cross-stream mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variants restrict these matrices to be doubly stochastic via Sinkhorn-Knopp (SK) algorithm or permutation-based parameterizations. We reveal three limitations of this doubly stochastic constraint: (1) identity degeneration, where learned matrices collapse around the identity initialization and diminish cross-stream interactions, (2) a expressivity bottleneck, where the doubly stochastic constraint restricts the freedom of the subdominant spectrum of the residual matrices, preventing the model from selectively preserving or attenuating cross-stream variations, and (3) parameterization inefficiencies, manifesting as unstable SK iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections ( $\mathrm{s}^{2}$ HC). By confining residual matrices to a spectral norm sphere, $\mathrm{s}^{2}$ HC restores free control over the subdominant spectrum, enabling the model to selectively preserve or attenuate cross-stream variations. This shift eliminates unstable SK iterations and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.

Figures & tables

Figure 1:

\mathrm{s}^{2}

HC addresses the identity degeneration, expressivity bottleneck, and parameterization inefficiencies in existing Manifold-Constrained Hyper-Connections (mHC Xie et al. (2025) ) and its permutation-based variants (mHC-lite Yang and Gao (2026) and KromHC Zhou et al. (2026) ). Left : Learned residual matrices (4 streams). mHC, mHC-lite, and KromHC degenerate into the identity mapping, whereas

\mathrm{s}^{2}

HC leverages expressive, non-degenerate residual matrices. Middle : Model performance, with each axis scaled to its own range and absolute scores annotated.

\mathrm{s}^{2}

HC improves over baselines across benchmarks. Right : Parameterization overhead, measured by the parameter complexity of the residual parameterization as the number of streams grows.

\mathrm{s}^{2}

HC scales compactly, avoiding both the factorial explosion of mHC-lite and the restriction of KromHC to specific stream counts.

Figure 2: Depth-wise analysis of

\mathcal{H}_{l}^{\mathrm{res}}

for mHC, mHC-lite and KromHC after training. Left : the median row-wise maximum entries of

\mathcal{H}_{l}^{\mathrm{res}}

. Right : the proportion of

\mathcal{H}_{l}^{\mathrm{res}}

where all row maximums are on the diagonal.

Figure 3: Mean pairwise cosine similarity among residual streams after being mixed by

\mathcal{H}_{l}^{\mathrm{res}}

across model layers. Identity denotes the baseline where

\mathcal{H}_{l}^{\mathrm{res}}

is fixed as an identity matrix while keeping all other settings identical to mHC, mHC-lite, and KromHC.

Figure 4: Overview of Spectral-Sphere-Constrained Hyper-Connections ( $\mathrm{s}^{2}$ HC). The right plane depicts the zero-marginal subspace $\mathcal{Z}_{n}$ , where the SVD parameterization generates $\mathcal{H}_{l}^{\mathrm{disp}}$ under the bounded spectral norm $\|\mathcal{H}_{l}^{\mathrm{disp}}\|_{2}\leq 1$ , centered at the origin $O$ . The left plane depicts the target affine subspace $\mathcal{A}_{n}$ , where the Birkhoff polytope $\mathcal{B}_{n}$ (inner polygon) is contained in the affine-constrained spectral norm sphere $\mathcal{S}_{n}$ . The affine shift $+J$ maps $O$ to $J$ , and $\mathcal{H}_{l}^{\mathrm{disp}}$ to the residual matrix $\mathcal{H}_{l}^{\mathrm{res}}$ .

Method	WinoGrande	BoolQ	PIQA	HellaSwag	ARC-E	ARC-C	MMLU	TruthfulQA	Avg.
Qwen3
RC	50.6	58.4	67.1	42.0	58.8	29.7	25.3	36.7	46.1
mHC	53.5	59.8	67.7	43.6	56.6	29.3	25.4	32.9	46.1
mHC-lite	53.4	60.1	67.1	43.1	56.4	28.7	24.3	33.1	45.8
KromHC	49.5	60.4	67.7	43.2	56.3	28.3	24.7	32.4	45.3
$\mathrm{s}^{2}$ HC (Ours)	53.6	60.3	68.3	44.0	59.8	30.9	25.6	34.4	47.1

Table 1: Evaluation results (accuracy

\%

) on benchmarks. Higher values indicate better performance.

Figure 5: Expressivity analysis of

\mathrm{s}^{2}

HC on Qwen3. Left: Mean pairwise cosine similarity among residual streams after being mixed by

\mathcal{H}_{l}^{\mathrm{res}}

across model layers. Identity denotes the baseline where

\mathcal{H}_{l}^{\mathrm{res}}

is fixed as an identity matrix. Right: the spectrum of

\mathcal{H}_{l}^{\mathrm{res}}

within the subspace orthogonal to the mean component. More results on Gemma3 and Qwen2.5 can be found in Appendix F .

Figure 6: Gradient norm dynamics during training for the Qwen3 model on FineWebEdu. HC denotes the unconstrained Hyper-Connections Zhu et al. (2024) . The unconstrained HC exhibits exploding gradients, clamped at 5 for visualization. Constrained hyper-connection paradigms show stable gradient trajectories.

Panel of Figure 7 — Figure 7: Empirical validation of the stability of constrained residual matrices for mHC, mHC-lite, and $\mathrm{s}^{2}$ HC. Statistics are computed from the trained Qwen3 model with 4 residual streams on the 1024 samples. Left: Column-sum distribution of layer-wise residual matrices $\mathcal{H}_{l}^{\mathrm{res}}$ for each layer. Middle: Column-sum distribution of the full-model composite mapping $\prod_{l=0}^{27}\mathcal{H}_{27-l}^{\mathrm{res}}$ . Right: Spectral norm of the cumulative composite mapping $\|\prod_{l=0}^{L-1}\mathcal{H}_{L-1-l}^{\mathrm{res}}\|_{2}$ across the first $L$ layers. Lines denote means across samples while shaded regions indicate standard deviations. More results on Gemma3 and Qwen2.5 are provided in Appendix F .

Figure 8: Training throughput and parameterization overhead across varying numbers of streams

n

. The mHC result is based on our PyTorch re-implementation and may underestimate the throughput of the specialized-kernel implementation in Xie et al. (2025) . KromHC is plotted as discrete points at

n=4

due to its restriction to powers of two under the default setting. Left: Training throughput evaluated on a 4

\times

H200 setup with a fixed global batch size. The data point for mHC-lite at

n=7

is omitted, as its factorial parameter explosion exceeds the GPU memory budget, resulting in an Out-Of-Memory failure. Right: Parameterization overhead (params) of the hyper-connections, excluding base model weights.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Visualization of layer-wise residual matrices and composite mappings on Qwen3. The figure displays single-layer residual matrices

\mathcal{H}_{l}^{\mathrm{res}}

at selected depths (

l\in\{2,16,20,26\}

), alongside their end-to-end composite mappings (

\prod_{l=0}^{27}\mathcal{H}_{27-l}^{\mathrm{res}}

) across mHC, mHC-lite, KromHC, and

\mathrm{s}^{2}

HC (Ours). Each matrix is computed by averaging over all tokens within a sequence. The numerical annotations along the vertical and horizontal axes denote the row sums and column sums, respectively.

Figure 10: Distribution of layer-wise residual matrix entries in Qwen3 across mHC, mHC-lite, KromHC and our proposed

\mathrm{s}^{2}

HC.

$\mathcal{H}_{l}^{\mathrm{res}}$	WinoGrande	BoolQ	PIQA	HellaSwag	ARC-E	ARC-C	MMLU	TruthfulQA
Qwen3
✗	53.3	60.5	67.4	43.2	58.9	31.1	24.0	34.2
✓	53.6	60.3	68.3	44.0	59.8	30.9	25.6	34.4
Gemma3
✗	52.9	59.6	70.3	48.0	67.0	35.4	24.3	36.8
✓	55.3	60.6	70.7	48.2	67.5	36.0	25.3	37.9

Appendix

Table 2: Comparison with Identity Mapping Baseline.

	WinoGrande	BoolQ	PIQA	HellaSwag	ARC-E	ARC-C	MMLU	TruthfulQA
Qwen3
Orthogonal	52.8	60.5	67.9	43.9	59.8	30.4	24.3	34.7
$\mathrm{s}^{2}$ HC (Ours)	53.6	60.3	68.3	44.0	59.8	30.9	25.6	34.4
Gemma3
Orthogonal	53.1	59.5	71.5	48.5	65.7	33.8	25.2	38.6
$\mathrm{s}^{2}$ HC (Ours)	55.3	60.6	70.7	48.2	67.5	36.0	25.3	37.9

Appendix

Table 3: Comparison with Orthogonal-Constrained Baseline.

Panel of Figure 11 — Figure 11: Gradient norm dynamics during training on three models. Orth. denotes the orthogonal-constrained baseline. The results are clamped at 2 for visualization.

Setting	Qwen3	Gemma3	Qwen2.5
Micro-batch size per GPU	8	8	4
Global batch size	512	512	1024
Sequence length	2048	2048	2048
Training tokens	15B	20B	30B
Warmup iterations	1000	1000	1000
Weight decay	0.1	0.1	0.1

Appendix

Table 4: Training hyperparameters.

Benchmarks	PIQA	BoolQ	MMLU	TruthfulQA	WinoGrande	HellaSwag	ARC-E	ARC-C
# Shots	0	0	0	0	5	10	25	25

Appendix

Table 5: Configurations of number of shots for each evaluation benchmark.

Panel of Figure 12 — Figure 12: Visualization of layer-wise residual matrices and composite mappings. The figure displays single-layer residual matrices $\mathcal{H}_{l}^{\mathrm{res}}$ at selected depths ( $l\in\{6,15,20,25\}$ for Gemma3 and $l\in\{1,8,16,23\}$ for Qwen2.5), alongside their end-to-end composite mappings ( $\prod_{l=0}^{25}\mathcal{H}_{25-l}^{\mathrm{res}}$ for Gemma3 and $\prod_{l=0}^{27}\mathcal{H}_{27-l}^{\mathrm{res}}$ for Qwen2.5) across mHC, mHC-lite, KromHC, and $\mathrm{s}^{2}$ HC (Ours). Each matrix is computed by averaging over all tokens within a sequence. The numerical annotations along the vertical and horizontal axes denote the row sums and column sums, respectively.

Panel of Figure 13 — Figure 13: More expressivity analysis of $\mathrm{s}^{2}$ HC on Gemma3 and Qwen2.5. Left: Mean pairwise cosine similarity among residual streams after being mixed by $\mathcal{H}_{l}^{\mathrm{res}}$ across model layers. Identity denotes the baseline where $\mathcal{H}_{l}^{\mathrm{res}}$ is fixed as an identity matrix. Right: the spectrum of $\mathcal{H}_{l}^{\mathrm{res}}$ within the subspace orthogonal to the mean component.

Panel of Figure 14 — Figure 14: More stability analysis of constrained residual matrices for mHC, mHC-lite, KromHC and $\mathrm{s}^{2}$ HC. Statistics are computed from the trained Gemma3 and Qwen2.5 models with 4 residual streams. Left: Column-sum distribution of layer-wise residual matrices $\mathcal{H}_{l}^{\mathrm{res}}$ for each layer. Middle: Column-sum distribution of the full-model composite mapping ( $\prod_{l=0}^{25}\mathcal{H}_{25-l}^{\mathrm{res}}$ for Gemma3 and $\prod_{l=0}^{27}\mathcal{H}_{27-l}^{\mathrm{res}}$ for Qwen2.5). Right: Spectral norm of the cumulative composite mapping ( $\|\prod_{l=0}^{L-1}\mathcal{H}_{L-1-l}^{\mathrm{res}}\|_{2}$ ) across the first $L$ layers. Lines denote means across samples while shaded regions indicate standard deviations.

Hyper-Connections

Hyper-connections expand neural-network residual paths into multiple streams with learned routing and mixing to improve capacity, stability, and efficiency.

See all 18 papers

Aug 6, 2026

Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning

Residual connections are fundamental to deep speaker recogni- tion models, such as ECAPA-TDNN and ResNet. However, standard identity mapping limits information flow to a sin- gle path, constraining representation capacity. We introduce Manifold-Constrained Hyper-Connections (mHC), reformulat- ing residual paths as a multi-stream evolution where informa- tion is mixed through a doubly stochastic matrix. By employing Sinkhorn-Knopp iterations, mHC ensures energy conservation by preserving signal intensity and feature mean, which stabi- lizes gradients and mitigates signal degradation in complex net- works. We evaluate mHC by replacing standard residual con- nections in backbones including ECAPA-TDNN, ResNet-34, Res2Net, and E-Res2Net. Extensive experiments on VoxCeleb1 demonstrate that mHC connections consistently enhance per- formance across all architectures, highlighting its effectiveness for robust speaker representation learning.

Zezhong Jin, Xiaoyu Wang, Zhe Li +4

Dept. of EEE, The Hong Kong Polytechnic University · Baidu Inc. · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR

Audio Representation Learning Residual Learning Hyper-Connections Speech Processing

Ask PDF

Feb 20, 2026

JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

Recent advances in deep learning, exemplified by Hyper-Connections (HC), have expanded the residual connection paradigm by introducing wider residual streams and diverse connectivity patterns. While these innovations yield significant performance gains, they compromise the identity mapping property of residual connections, leading to training instability, limited scalability, and increased memory overhead. To address these challenges, we propose JPmHC (Jacobian-spectrum Preserving manifold-constrained Hyper-Connections), a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams while explicitly controlling gradient conditioning. By constraining the mixer M on operator-norm-bounded manifolds (e.g., bistochastic, Stiefel, Grassmann), JPmHC prevents gradient pathologies and enhances stability. JPmHC introduces three key contributions: (i) a free-probability analysis that predicts Jacobian spectra for structured skips, providing actionable design rules for mixer selection; (ii) memory-efficient implicit differentiation for fixed-point projections, reducing activation memory and synchronization overhead; and (iii) a Stiefel-constrained mixer via Cayley transforms, ensuring orthogonality without post-hoc normalization. Empirical evaluations on ARC-AGI demonstrate that JPmHC achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines, with a rank-

p

Grassmannian variant tracking between the two -- consistent with the spectral theory predictions. As a flexible and scalable extension of HC, JPmHC advances spectrum-aware, stable, and efficient deep learning, offering insights into topological architecture design and foundational model evolution. \newline \newline

Biswa Sengupta, Jinhua Wang, Leo Brunswic

LLM Suite Team, JP Morgan Chase & Co.

Hyper-Connections Residual Learning

Ask PDF

May 20, 2026

TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams, but unconstrained mixing leads to training instability. Manifold-Constrained Hyper-Connections (mHC) address this by enforcing approximate double stochasticity via Sinkhorn normalization, while mHC-lite ensures exact constraints through convex combinations of permutation matrices at the cost of factorial complexity. KromHC reduces this cost using Kronecker-product parameterizations, but restricts the mixing matrices to a structured submanifold of the Birkhoff polytope . We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP), which construct exactly doubly stochastic mixing matrices with

(n-1)^2

degrees of freedom. Our approach avoids iterative normalization and combinatorial explosion while preserving full expressivity of the Birkhoff polytope. Empirical results on language model pre-training' demonstrate competitive performance with improved stability and scalability.

Anton Lyubinin

Hyper-Connections

Ask PDF

Abstract

Hyper-Connections

Explore similar work

Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning

JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes