Spectral-Sphere-Constrained Hyper-Connections

Zhaoyi Liu; Haichuan Zhang; Ang Li

cs.LGMar 21, 2026

Spectral-Sphere-Constrained Hyper-Connections

Authors: Zhaoyi Liu, Haichuan Zhang, Ang Li

Organizations: University of Maryland, College Park · University of Utah

Abstract

Hyper-Connections (HC) extend residual connections into multiple streams, employing residual matrices for cross-stream mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variants restrict these matrices to be doubly stochastic via Sinkhorn-Knopp (SK) algorithm or permutation-based parameterizations. We reveal three limitations of this doubly stochastic constraint: (1) identity degeneration, where learned matrices collapse around the identity initialization and diminish cross-stream interactions, (2) a expressivity bottleneck, where the doubly stochastic constraint restricts the freedom of the subdominant spectrum of the residual matrices, preventing the model from selectively preserving or attenuating cross-stream variations, and (3) parameterization inefficiencies, manifesting as unstable SK iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections ( $\mathrm{s}^{2}$ HC). By confining residual matrices to a spectral norm sphere, $\mathrm{s}^{2}$ HC restores free control over the subdominant spectrum, enabling the model to selectively preserve or attenuate cross-stream variations. This shift eliminates unstable SK iterations and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.

Figures & tables

Figure 1:

\mathrm{s}^{2}

HC addresses the identity degeneration, expressivity bottleneck, and parameterization inefficiencies in existing Manifold-Constrained Hyper-Connections (mHC Xie et al. (2025) ) and its permutation-based variants (mHC-lite Yang and Gao (2026) and KromHC Zhou et al. (2026) ). Left : Learned residual matrices (4 streams). mHC, mHC-lite, and KromHC degenerate into the identity mapping, whereas

\mathrm{s}^{2}

HC leverages expressive, non-degenerate residual matrices. Middle : Model performance, with each axis scaled to its own range and absolute scores annotated.

\mathrm{s}^{2}

HC improves over baselines across benchmarks. Right : Parameterization overhead, measured by the parameter complexity of the residual parameterization as the number of streams grows.

\mathrm{s}^{2}

HC scales compactly, avoiding both the factorial explosion of mHC-lite and the restriction of KromHC to specific stream counts.

Figure 2: Depth-wise analysis of

\mathcal{H}_{l}^{\mathrm{res}}

for mHC, mHC-lite and KromHC after training. Left : the median row-wise maximum entries of

\mathcal{H}_{l}^{\mathrm{res}}

. Right : the proportion of

\mathcal{H}_{l}^{\mathrm{res}}

where all row maximums are on the diagonal.

Figure 3: Mean pairwise cosine similarity among residual streams after being mixed by

\mathcal{H}_{l}^{\mathrm{res}}

across model layers. Identity denotes the baseline where

\mathcal{H}_{l}^{\mathrm{res}}

is fixed as an identity matrix while keeping all other settings identical to mHC, mHC-lite, and KromHC.

Figure 4: Overview of Spectral-Sphere-Constrained Hyper-Connections ( $\mathrm{s}^{2}$ HC). The right plane depicts the zero-marginal subspace $\mathcal{Z}_{n}$ , where the SVD parameterization generates $\mathcal{H}_{l}^{\mathrm{disp}}$ under the bounded spectral norm $\|\mathcal{H}_{l}^{\mathrm{disp}}\|_{2}\leq 1$ , centered at the origin $O$ . The left plane depicts the target affine subspace $\mathcal{A}_{n}$ , where the Birkhoff polytope $\mathcal{B}_{n}$ (inner polygon) is contained in the affine-constrained spectral norm sphere $\mathcal{S}_{n}$ . The affine shift $+J$ maps $O$ to $J$ , and $\mathcal{H}_{l}^{\mathrm{disp}}$ to the residual matrix $\mathcal{H}_{l}^{\mathrm{res}}$ .

Method	WinoGrande	BoolQ	PIQA	HellaSwag	ARC-E	ARC-C	MMLU	TruthfulQA	Avg.
Qwen3
RC	50.6	58.4	67.1	42.0	58.8	29.7	25.3	36.7	46.1
mHC	53.5	59.8	67.7	43.6	56.6	29.3	25.4	32.9	46.1
mHC-lite	53.4	60.1	67.1	43.1	56.4	28.7	24.3	33.1	45.8
KromHC	49.5	60.4	67.7	43.2	56.3	28.3	24.7	32.4	45.3
$\mathrm{s}^{2}$ HC (Ours)	53.6	60.3	68.3	44.0	59.8	30.9	25.6	34.4	47.1

Table 1: Evaluation results (accuracy

\%

) on benchmarks. Higher values indicate better performance.

Figure 5: Expressivity analysis of

\mathrm{s}^{2}

HC on Qwen3. Left: Mean pairwise cosine similarity among residual streams after being mixed by

\mathcal{H}_{l}^{\mathrm{res}}

across model layers. Identity denotes the baseline where

\mathcal{H}_{l}^{\mathrm{res}}

is fixed as an identity matrix. Right: the spectrum of

\mathcal{H}_{l}^{\mathrm{res}}

within the subspace orthogonal to the mean component. More results on Gemma3 and Qwen2.5 can be found in Appendix F .

Figure 6: Gradient norm dynamics during training for the Qwen3 model on FineWebEdu. HC denotes the unconstrained Hyper-Connections Zhu et al. (2024) . The unconstrained HC exhibits exploding gradients, clamped at 5 for visualization. Constrained hyper-connection paradigms show stable gradient trajectories.

Panel of Figure 7 — Figure 7: Empirical validation of the stability of constrained residual matrices for mHC, mHC-lite, and $\mathrm{s}^{2}$ HC. Statistics are computed from the trained Qwen3 model with 4 residual streams on the 1024 samples. Left: Column-sum distribution of layer-wise residual matrices $\mathcal{H}_{l}^{\mathrm{res}}$ for each layer. Middle: Column-sum distribution of the full-model composite mapping $\prod_{l=0}^{27}\mathcal{H}_{27-l}^{\mathrm{res}}$ . Right: Spectral norm of the cumulative composite mapping $\|\prod_{l=0}^{L-1}\mathcal{H}_{L-1-l}^{\mathrm{res}}\|_{2}$ across the first $L$ layers. Lines denote means across samples while shaded regions indicate standard deviations. More results on Gemma3 and Qwen2.5 are provided in Appendix F .

Figure 8: Training throughput and parameterization overhead across varying numbers of streams

n

. The mHC result is based on our PyTorch re-implementation and may underestimate the throughput of the specialized-kernel implementation in Xie et al. (2025) . KromHC is plotted as discrete points at

n=4

due to its restriction to powers of two under the default setting. Left: Training throughput evaluated on a 4

\times

H200 setup with a fixed global batch size. The data point for mHC-lite at

n=7

is omitted, as its factorial parameter explosion exceeds the GPU memory budget, resulting in an Out-Of-Memory failure. Right: Parameterization overhead (params) of the hyper-connections, excluding base model weights.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Visualization of layer-wise residual matrices and composite mappings on Qwen3. The figure displays single-layer residual matrices

\mathcal{H}_{l}^{\mathrm{res}}

at selected depths (

l\in\{2,16,20,26\}

), alongside their end-to-end composite mappings (

\prod_{l=0}^{27}\mathcal{H}_{27-l}^{\mathrm{res}}

) across mHC, mHC-lite, KromHC, and

\mathrm{s}^{2}

HC (Ours). Each matrix is computed by averaging over all tokens within a sequence. The numerical annotations along the vertical and horizontal axes denote the row sums and column sums, respectively.

Figure 10: Distribution of layer-wise residual matrix entries in Qwen3 across mHC, mHC-lite, KromHC and our proposed

\mathrm{s}^{2}

HC.

$\mathcal{H}_{l}^{\mathrm{res}}$	WinoGrande	BoolQ	PIQA	HellaSwag	ARC-E	ARC-C	MMLU	TruthfulQA
Qwen3
✗	53.3	60.5	67.4	43.2	58.9	31.1	24.0	34.2
✓	53.6	60.3	68.3	44.0	59.8	30.9	25.6	34.4
Gemma3
✗	52.9	59.6	70.3	48.0	67.0	35.4	24.3	36.8
✓	55.3	60.6	70.7	48.2	67.5	36.0	25.3	37.9

Appendix

Table 2: Comparison with Identity Mapping Baseline.

	WinoGrande	BoolQ	PIQA	HellaSwag	ARC-E	ARC-C	MMLU	TruthfulQA
Qwen3
Orthogonal	52.8	60.5	67.9	43.9	59.8	30.4	24.3	34.7
$\mathrm{s}^{2}$ HC (Ours)	53.6	60.3	68.3	44.0	59.8	30.9	25.6	34.4
Gemma3
Orthogonal	53.1	59.5	71.5	48.5	65.7	33.8	25.2	38.6
$\mathrm{s}^{2}$ HC (Ours)	55.3	60.6	70.7	48.2	67.5	36.0	25.3	37.9

Appendix

Table 3: Comparison with Orthogonal-Constrained Baseline.

Panel of Figure 11 — Figure 11: Gradient norm dynamics during training on three models. Orth. denotes the orthogonal-constrained baseline. The results are clamped at 2 for visualization.

Setting	Qwen3	Gemma3	Qwen2.5
Micro-batch size per GPU	8	8	4
Global batch size	512	512	1024
Sequence length	2048	2048	2048
Training tokens	15B	20B	30B
Warmup iterations	1000	1000	1000
Weight decay	0.1	0.1	0.1

Appendix

Table 4: Training hyperparameters.

Benchmarks	PIQA	BoolQ	MMLU	TruthfulQA	WinoGrande	HellaSwag	ARC-E	ARC-C
# Shots	0	0	0	0	5	10	25	25

Appendix

Table 5: Configurations of number of shots for each evaluation benchmark.

Panel of Figure 12 — Figure 12: Visualization of layer-wise residual matrices and composite mappings. The figure displays single-layer residual matrices $\mathcal{H}_{l}^{\mathrm{res}}$ at selected depths ( $l\in\{6,15,20,25\}$ for Gemma3 and $l\in\{1,8,16,23\}$ for Qwen2.5), alongside their end-to-end composite mappings ( $\prod_{l=0}^{25}\mathcal{H}_{25-l}^{\mathrm{res}}$ for Gemma3 and $\prod_{l=0}^{27}\mathcal{H}_{27-l}^{\mathrm{res}}$ for Qwen2.5) across mHC, mHC-lite, KromHC, and $\mathrm{s}^{2}$ HC (Ours). Each matrix is computed by averaging over all tokens within a sequence. The numerical annotations along the vertical and horizontal axes denote the row sums and column sums, respectively.

Panel of Figure 13 — Figure 13: More expressivity analysis of $\mathrm{s}^{2}$ HC on Gemma3 and Qwen2.5. Left: Mean pairwise cosine similarity among residual streams after being mixed by $\mathcal{H}_{l}^{\mathrm{res}}$ across model layers. Identity denotes the baseline where $\mathcal{H}_{l}^{\mathrm{res}}$ is fixed as an identity matrix. Right: the spectrum of $\mathcal{H}_{l}^{\mathrm{res}}$ within the subspace orthogonal to the mean component.

Panel of Figure 14 — Figure 14: More stability analysis of constrained residual matrices for mHC, mHC-lite, KromHC and $\mathrm{s}^{2}$ HC. Statistics are computed from the trained Gemma3 and Qwen2.5 models with 4 residual streams. Left: Column-sum distribution of layer-wise residual matrices $\mathcal{H}_{l}^{\mathrm{res}}$ for each layer. Middle: Column-sum distribution of the full-model composite mapping ( $\prod_{l=0}^{25}\mathcal{H}_{25-l}^{\mathrm{res}}$ for Gemma3 and $\prod_{l=0}^{27}\mathcal{H}_{27-l}^{\mathrm{res}}$ for Qwen2.5). Right: Spectral norm of the cumulative composite mapping ( $\|\prod_{l=0}^{L-1}\mathcal{H}_{L-1-l}^{\mathrm{res}}\|_{2}$ ) across the first $L$ layers. Lines denote means across samples while shaded regions indicate standard deviations.

Hyper-Connections

Hyper-connections expand neural-network residual paths into multiple streams with learned routing and mixing to improve capacity, stability, and efficiency.

See all 18 papers

Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning
Aug 6, 2026Zezhong Jin, Xiaoyu Wang, Zhe Li +4Audio Representation Learning Residual Learning
Dept. of EEE, The Hong Kong Polytechnic University · Baidu Inc. · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR
JPmHC Dynamical Isometry via Orthogonal Hyper-Connections
Feb 20, 2026Biswa Sengupta, Jinhua Wang, Leo BrunswicHyper-Connections Residual Learning
LLM Suite Team, JP Morgan Chase & Co.
TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes
May 20, 2026Anton LyubininHyper-Connections

Aug 6, 2026J/K move · Enter open · S save

Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning

Zezhong Jin, Xiaoyu Wang, Zhe Li, Chong-Xin Gan+3

Dept. of EEE, The Hong Kong Polytechnic University · Baidu Inc. · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR

Audio Representation Learning Residual Learning Hyper-Connections Speech Processing

Open paper AskPDF arXiv

Residual connections are fundamental to deep speaker recogni- tion models, such as ECAPA-TDNN and ResNet. However, standard identity mapping limits information flow to a sin- gle path, constraining representation capacity. We introduce Manifold-Constrained Hyper-Connections (mHC), reformulat- ing residual paths as a multi-stream evolution where informa- tion is mixed through a doubly stochastic matrix. By employing Sinkhorn-Knopp iterations, mHC ensures energy conservation by preserving signal intensity and feature mean, which stabi- lizes gradients and mitigates signal degradation in complex net- works. We evaluate mHC by replacing standard residual con- nections in backbones including ECAPA-TDNN, ResNet-34, Res2Net, and E-Res2Net. Extensive experiments on VoxCeleb1 demonstrate that mHC connections consistently enhance per- formance across all architectures, highlighting its effectiveness for robust speaker representation learning.

Abstract

Hyper-Connections

Explore similar work

Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning

JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes