Hyper-Connections

Ask about this topic

Overview

Hyper-connections generalize a standard residual skip by maintaining several parallel residual streams and learning how layers read from and mix them, adding a route to capacity beyond width or depth. Common designs include token-dependent, manifold-constrained mixers—often doubly stochastic—as well as orthogonal or spectrally constrained transforms, sparse updates, and tensorized low-rank parameterizations. They are studied in language-model pretraining and finetuning, as well as vision, diffusion, and speech tasks; researchers also use stream ablations and probes to understand how the connections affect model behavior. The central tension is that richer routing can improve performance and parameter efficiency, yet streams may collapse or remain near identity, while constraints that stabilize gradients can limit expressivity or add computational cost.

Latest papers

CardsList

18 papers1 paper in the last four weeks, down 67% on the four weeks before. 0.0% of all new papers.

  1. The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction

    Oct 5, 2026Xiaoyu Li, Zhizhou Sha, Chiwun YangHyper-ConnectionsConstrained Optimization

    University of New South Wales · University of Texas at Austin · City University of Hong Kong

  2. Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers

    Sep 27, 2026Tongtong Liang, Siqi Kou, Ziqiao Xi +6Diffusion TransformerResidual Learning

    UC San Diego · Shanghai Jiao Tong University · Aether AI

  3. How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing

    Sep 4, 2026Pengxiang Zhao, Xing Li, Xianzhi Yu +2Hyper-ConnectionsResidual Learning

    Huawei Technologies Co., Ltd.

  4. oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions

    Sep 2, 2026Haoqiang Guo, Xuyi Chen, Bo Ke +5Hyper-ConnectionsTransformer

    The Hong Kong University of Science and Technology · Baidu Inc.

  5. TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing

    Aug 8, 2026Yuxuan Gu, Wuyang Zhou, Huijun Xing +1Residual LearningHyper-Connections

    Department of Electrical and Electronic Engineering, Imperial College London

  6. Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning

    Aug 6, 2026Zezhong Jin, Xiaoyu Wang, Zhe Li +4Audio Representation LearningResidual Learning

    Dept. of EEE, The Hong Kong Polytechnic University · Baidu Inc. · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR

  7. Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

    Jul 20, 2026Valentijn Oldenburg, Floris de Kam, Bente Zuijdam +5TransformerHyper-Connections

    University of Amsterdam, Netherlands

  8. xHC: Expanded Hyper-Connections

    Jul 16, 2026Xiangdong Zhang, Xiaohan Qin, Sunan Zou +10Language Model PretrainingTransformer

    School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · School of CS, Peking University +3

  9. HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction

    Jun 25, 2026Luxi Lin, Shuang Peng, Rui Ma +7Hyper-ConnectionsSpeculative Decoding

    ByteDance

  10. Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation

    Jun 2, 2026Ekaterina Alimaskina, Gleb Molodtsov, Aleksandr BeznosikovTransformerHyper-Connections

    1MIRAI, Moscow, Russia · 2BRAIn Lab, Moscow, Russia · 3Yandex Research, Moscow, Russia +1

  11. Accelerating Birkhoff Projection for Manifold-Constrained Hyper-Connections

    May 26, 2026Chenrui Wang, Yixuan QiuHyper-Connections

    RenminSchoolUniversityof Statisticsof China · School of Statistics and Data Science & Institute of Big Data Research · Shanghai University of Finance and Economics

  12. TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

    May 20, 2026Anton LyubininHyper-Connections

  13. The EΔΔ-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality

    May 7, 2026Arash ShahmansooriTransformerResidual Learning

    Independent Researcher

  14. Hyperloop Transformers

    Apr 23, 2026Abbas Zeitoun, Lucas Torroba-Hennigen, Yoon KimLLM CompressionHyper-Connections

    Massachusetts Institute of Technology

  15. Spectral-Sphere-Constrained Hyper-Connections

    Mar 21, 2026Zhaoyi Liu, Haichuan Zhang, Ang LiHyper-ConnectionsResidual Learning

    University of Maryland, College Park · University of Utah

  16. Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation

    Mar 20, 2026Lokendra Kumar, Shubham Aggarwal3D Medical Image SegmentationHyper-Connections

  17. Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

    Mar 16, 2026William Peng, Josheev Rai, Kevin Tseng +2TransformerHyper-Connections

    Stanford University · Georgia Institute of Technology · University of California, Berkeley +2

  18. JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

    Feb 20, 2026Biswa Sengupta, Jinhua Wang, Leo BrunswicHyper-ConnectionsResidual Learning

    LLM Suite Team, JP Morgan Chase & Co.