cs.CLNov 18, 2026

Objective-Specialized Residual Streams in Manifold-Constrained Hyper-Connections

Authors: Shantanu Agarwal, Ruchita V. Maitri, Zhiqi Huang, Vivek Datla, Alfy Samuel

Organizations: Capital One

Abstract

Manifold-constrained Hyper-Connections (mHC) extend a Transformer's residual stream into multiple parallel streams with learned cross-stream connections. Prior work trains all streams toward a shared objective. We instead study whether assigning distinct learning objectives to different streams can improve task performance and induce objective-specific specialization. Using complementary lexical (sparse) and semantic (dense) retrieval objectives as a test case, we find that objective-specialized mHC consistently outperforms a non-specialized mHC model across the full sparse--dense fusion range, for both two- and four-stream models and at two model scales. We then examine how specialization changes the model internally using probes of stream representations, residual mixing, objective-specific gradient flow, and read/write gates. The specialized and non-specialized models behave similarly through most of the network but diverge in the final layers: the specialized model develops increasingly objective-specific gradient flow, distinct read/write routing patterns, and reduced cross-stream mixing, and streams assigned to different objectives become increasingly dissimilar while those trained toward the same objective remain highly similar. Fixing the residual mixing to the identity leaves the specialization benefit intact, indicating that the benefit stems from objective-specific read/write routing rather than learned cross-stream mixing. These results show that assigning distinct objectives to mHC streams can improve multi-objective retrieval while inducing structured, depth-dependent specialization in the residual computation.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

    Mar 16, 2026William Peng, Josheev Rai, Kevin Tseng +2TransformerHyper-Connections

  2. Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation

    Jun 2, 2026Ekaterina Alimaskina, Gleb Molodtsov, Aleksandr BeznosikovTransformerHyper-Connections

  3. How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing

    Sep 4, 2026Pengxiang Zhao, Xing Li, Xianzhi Yu +2Hyper-ConnectionsResidual Learning