cs.CLNov 17, 2026

TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates

Authors: Xinying Hu, You Wu, Haoyi Wu, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Kewei Tu

Organizations: School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging · Meituan

Abstract

Hyper-Connections (HC) expand the residual stream of a Transformer into NN parallel streams, and manifold-constrained HC (mHC) stabilizes them at scale. However, mHC benefits little from more than four streams, partly because each sublayer writes a single output to all streams, limiting how differently they can be updated. In this work, we propose TPHC, which integrates tensor-parallel (TP) shards with HC streams to promote greater independence across streams. Specifically, we route a separate mixture of the residual streams to each TP shard and keep the individual shard outputs separate during write-back, so each stream can receive a different mixture of those outputs. This design makes the residual-stream updates more expressive without adding parameters or FLOPs to the attention or FFN sublayers. With native support for stream sharding under TP, TPHC requires only 9% more training time than mHC on an 8B model when both use four streams. Increasing the number of streams from four to sixteen raises its training time by only 19%. On a 1.1B model trained on 100B tokens, TPHC reduces validation loss by 0.7-1.0% and raises average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3-0.7 percentage points relative to mHC at matched stream counts N∈{4,8,16}N\in\{4,8,16\}. Our code is available at https://github.com/wuyou2002/TPHC.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. xHC: Expanded Hyper-Connections

    Jul 16, 2026Xiangdong Zhang, Xiaohan Qin, Sunan Zou +10Language Model PretrainingTransformer

  2. Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation

    Jun 2, 2026Ekaterina Alimaskina, Gleb Molodtsov, Aleksandr BeznosikovTransformerHyper-Connections

  3. Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

    Mar 16, 2026William Peng, Josheev Rai, Kevin Tseng +2TransformerHyper-Connections