Manifold-constrained Hyper-Connections (mHC) extend a Transformer's residual stream into multiple parallel streams with learned cross-stream connections. Prior work trains all streams toward a shared objective. We instead study whether assigning distinct learning objectives to different streams can improve task performance and induce objective-specific specialization. Using complementary lexical (sparse) and semantic (dense) retrieval objectives as a test case, we find that objective-specialized mHC consistently outperforms a non-specialized mHC model across the full sparse--dense fusion range, for both two- and four-stream models and at two model scales. We then examine how specialization changes the model internally using probes of stream representations, residual mixing, objective-specific gradient flow, and read/write gates. The specialized and non-specialized models behave similarly through most of the network but diverge in the final layers: the specialized model develops increasingly objective-specific gradient flow, distinct read/write routing patterns, and reduced cross-stream mixing, and streams assigned to different objectives become increasingly dissimilar while those trained toward the same objective remain highly similar. Fixing the residual mixing to the identity leaves the specialization benefit intact, indicating that the benefit stems from objective-specific read/write routing rather than learned cross-stream mixing. These results show that assigning distinct objectives to mHC streams can improve multi-objective retrieval while inducing structured, depth-dependent specialization in the residual computation.
Figures & tables
Figure 1: Schematic comparison of the three encoder configurations studied in this work. In all three, streams propagate through the full encoder stack and the retrieval heads read only the final-layer ( l=L ) stream representations. (left) Single-stream : the standard ModernBERT encoder, with a single residual stream read by both the sparse and dense retrieval heads. (middle) Non-specialized mHC : the residual stream is expanded into n parallel streams with learned cross-stream routing. Both retrieval heads operate on the mean-pooled representation obtained from all n streams, so the sparse and dense objectives jointly optimize the same stream representation. (right) Objective-specialized mHC : the sparse and dense heads instead read disjoint sets of final-layer streams, so each objective selectively trains its assigned stream(s). For clarity the schematic depicts n=2 streams; we study both n=2 and n=4 in our experiments.
Model
ArguAna
C-FEVER
DBPedia
FEVER
FiQA
HotpotQA
NFC
NQ
Quora
SciDocs
SciF
TREC-C
Touché
Avg
n=1
Sparse-only
0.3943
0.1052
0.1484
0.6384
0.1953
0.5895
0.2219
0.1600
0.6722
0.1119
0.6492
0.5381
0.2372
0.3585
Dense-only
0.3974
0.2116
0.2702
0.5049
0.2373
0.3797
0.2460
0.4569
0.8479
0.1328
0.5513
0.6784
0.2201
0.3950
Sparse+Dense
0.3833
0.2217
0.2552
0.6525
0.2649
0.5678
0.2611
0.4110
0.8274
0.1310
0.6707
0.6990
0.2643
0.4315
n=2
mHC no-spec
0.3827
0.2132
0.2579
0.6529
0.2718
0.5696
0.2708
0.4079
0.8229
0.1346
0.6709
0.6843
0.2634
0.4310
Table 1: BEIR nDCG@10 across all 13 datasets using ModernBERT-base. Sparse-only and Dense-only are evaluated in their native retrieval mode; Sparse+Dense and the mHC models are hybrid retrievers fused at α=0.9 , the shared empirical optimum across all multi-objective models. Underline marks the best score within each stream count n ; bold marks the best score for that dataset across all configurations. The advantage of objective specialization is not confined to the aggregate score; it extends to a majority of individual datasets and holds at both n=2 and n=4 . Excluding the in-domain NQ dataset, mHC+spec still improves over mHC no-spec on the remaining 12 datasets ( 0.4329→0.4470 at n=2 ; 0.4431→0.4510 at n=4 ). Results reported here are mean over two seeds. Per seed variation is provided in Appendix 5 . Results with ModernBERT-large (Table 4 ) show the same trend: mHC+spec improves over mHC no-spec at both n=2 and n=4 , indicating that the benefit of objective specialization is robust to model scale rather than specific to a single model size.
Figure 2: Average BEIR nDCG@10 as a function of the fusion weight α , for ModernBERT-base (left) and ModernBERT-large (right); both mean over two seeds. α=0 and α=1 correspond to pure sparse and pure dense retrieval, respectively. Color encodes objective specialization (red: specialized, blue: non-specialized); Sparse+Dense ( n=1 , grey) has no stream routing. Specialized models (red) dominate at every α for both stream counts and at both base/large model scales.
Figure 3: Four depth-resolved specialization probes at the final checkpoint, for n=2 (top) and n=4 (bottom) streams (lines are the mean and shaded bands ±1 standard deviation over 4 seeds; flat layers 0 – 9 omitted). From left: stream cosine similarity, residual-mixing identity retention diag(A) , per-objective gradient share on the assigned stream(s), and read/write gate shift. Red: specialized; blue: non-specialized control. Solid vs. dashed distinguishes the two objectives / gates / stream-pair types (legends). Metrics are defined and discussed in Section 5 .
Model
Mixing A
nDCG@10 ( α=0.9 )
mHC no-spec
learned
0.4410±0.0023
mHC spec
learned
0.4529±0.0019
mHC no-spec
identity
0.4415±0.0025
mHC spec
identity
0.4546±0.0001
Table 2: Fixing the residual-mixing matrix to the identity ( Aℓ=I at every layer). ModernBERT-base, n=4 , average BEIR nDCG@10 over 13 datasets at α=0.9 ; mean ± standard deviation over two seeds.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Shape
Initialization
Wpre
n×nd
0
Wpost
n×nd
0
Wres
n2×nd
0
bpre
n
logit(1/n)+ε , renorm.
bpost
n
0
bres
n2
diag 1 , off-diag 0
Appendix
Table 3: Initialization of each mHC block. Projection weights are zero-initialized so routing starts static; the biases place the block near an identity residual layer. The read-gate bias uses the uniform-noisy scheme (target σ(bkpre)=1/n plus noise ε∼N(0,σnoise2) , σnoise=10−2 , renormalized to sum to one) for both specialized and non-specialized models. At initialization this yields read gate p≈[1/n,…,1/n] (summing to 1 ), write gate q=[1,…,1] , and a doubly-stochastic mixing matrix A with its largest mass on the diagonal.
Figure 4: Attention-sublayer routing probes at the final checkpoint (lines are the mean and shaded bands ±1 standard deviation over 4 seeds; flat layers 0 – 9 omitted). Left pair: n=2 ; right pair: n=4 . Within each pair: mean diagonal of the residual-mixing matrix diag(A) , and the shift in each gate’s sparse-stream share Δ (read gate p solid, write gate q dashed). Red: specialized; blue: non-specialized control. Like the MLP sublayer, attention drives A toward the identity under specialization; unlike the MLP, its read gate does not shift ( Δs(pℓ)≈0 ) while its write gate moves strongly toward the dense-assigned stream(s) ( Δs(qL)≈−0.44 ), i.e. attention specializes its write but not its read. Metrics defined in Section 5 .
Model
ArguAna
C-FEVER
DBPedia
FEVER
FiQA
HotpotQA
NFC
NQ
Quora
SciDocs
SciF
TREC-C
Touché
Avg
n=1
Sparse-only
0.3967
0.0954
0.1348
0.5931
0.2133
0.5568
0.2251
0.1654
0.6761
0.1118
0.6507
0.5178
0.2307
0.3514
Dense-only
0.4110
0.2157
0.3072
0.5351
0.3103
0.4061
0.2800
0.5063
0.8562
0.1611
0.5972
0.6996
0.2205
0.4236
Sparse+Dense
0.4268
0.2204
0.2711
0.6586
0.3095
0.5838
0.2828
0.4488
0.8443
0.1398
0.6970
0.7614
0.2939
0.4568
n=2
mHC no-spec
0.4103
0.2108
0.2838
0.6698
0.3076
0.5864
0.2983
0.4546
0.8453
0.1421
0.6911
0.7621
0.2836
0.4574
Appendix
Table 4: BEIR nDCG@10 across all 13 datasets using ModernBERT-large, mean over two seeds (42 and 1234). Sparse-only and Dense-only are evaluated in their native retrieval mode; Sparse+Dense and the mHC models are hybrid retrievers fused at α=0.9 . Underline marks the best score within each stream count n ; bold marks the best score for that dataset across all configurations.
Model
Seed
ArguAna
C-FEVER
DBPedia
FEVER
FiQA
HotpotQA
NFC
NQ
Quora
SciDocs
SciF
TREC-C
Touché
Avg
n=1
Sparse-only
42
0.4038
0.1040
0.1504
0.6151
0.1998
0.5857
0.2238
0.1637
0.6823
0.1134
0.6555
0.5704
0.2269
0.3611
1234
0.3849
0.1064
0.1463
0.6617
0.1908
0.5932
0.2200
0.1562
0.6620
0.1105
0.6428
0.5057
0.2476
0.3560
Dense-only
42
0.3946
0.2119
0.2690
0.5025
0.2419
0.3785
0.2529
0.4606
0.8488
0.1315
0.5600
0.6725
0.2214
0.3959
1001
0.4003
0.2112
0.2715
0.5072
0.2326
0.3808
0.2392
0.4532
0.8471
0.1341
0.5426
0.6843
0.2187
0.3941
Sparse+Dense
42
0.3944
0.2022
0.2581
0.6382
0.2676
0.5657
0.2622
0.4047
0.8289
0.1309
0.6622
0.6797
0.2615
0.4274
Appendix
Table 5: Per-seed BEIR nDCG@10 across all 13 datasets (hybrid fusion at α=0.9 ) for ModernBERT-base.
Model
Seed
ArguAna
C-FEVER
DBPedia
FEVER
FiQA
HotpotQA
NFC
NQ
Quora
SciDocs
SciF
TREC-C
Touché
Avg
n=1
Sparse-only
42
0.4111
0.0981
0.1326
0.5833
0.2124
0.5491
0.2278
0.1701
0.6799
0.1143
0.6582
0.5424
0.2262
0.3543
1234
0.3822
0.0926
0.1371
0.6028
0.2142
0.5644
0.2223
0.1608
0.6722
0.1092
0.6433
0.4932
0.2351
0.3484
Dense-only
42
0.4107
0.2158
0.3080
0.5403
0.3078
0.4052
0.2777
0.5047
0.8563
0.1606
0.5944
0.7053
0.2233
0.4239
1234
0.4112
0.2156
0.3063
0.5299
0.3127
0.4070
0.2823
0.5079
0.8562
0.1616
0.5999
0.6938
0.2178
0.4232
Sparse+Dense
42
0.4291
0.2133
0.2670
0.6538
0.3027
0.5829
0.2872
0.4477
0.8459
0.1391
0.7031
0.7472
0.2915
0.4547
Appendix
Table 6: Per-seed BEIR nDCG@10 across all 13 datasets using ModernBERT-large, for each model and training seed. Sparse-only and Dense-only are evaluated in their native retrieval mode; the remaining models are hybrid retrievers fused at α=0.9 . The two-seed means are the values reported in Table 4 . As at the base scale, the specialization gain holds within each seed: at both n=2 and n=4 , mHC+spec exceeds mHC no-spec in every seed.
Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual connections, yet their internal mechanisms remain unexplored. In particular, the recently introduced Manifold-Constrained Hyper-Connections (mHC) architecture posits multiple residual streams with constrained interaction, but lacks in-depth mechanistic analysis. We present the first open-source mHC language model (https://huggingface.co/wgpeng/mhc-780m) and analyze the multiple-stream architecture with a suite of representation-level metrics and causal interventions to probe how parallel streams encode and utilize information. Specifically, we introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference. Through targeted pairwise interventions and controlled recovery experiments, we distinguish functional redundancy from asymmetric utilization and reveal how information is distributed across streams beyond what is observable from representational similarity alone.
William Peng, Josheev Rai, Kevin Tseng +2
Stanford University · Georgia Institute of Technology · University of California, Berkeley +2
Hyper-Connections (HC) replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices. We study how this symmetry is resolved in practice: whether streams specialize in a balanced way or exhibit dominant-stream usage. Using fine-grained diagnostics for HC-based language models, we trace how multi-stream representations are actually used. We find that after an early seeding stage, residual mixing often remains close to identity, limiting a core HC mechanism for exchanging information between streams. Moreover, both signal and interpretable features concentrate in a dominant stream, and the nominally multi-stream residual connection can underutilize its capacity, behaving closer to a single-stream residual pathway. Finally, we show that breaking symmetry at stream initialization reduces dominant behavior and improves performance across mHC variants. Our code is publicly available.
Ekaterina Alimaskina, Gleb Molodtsov, Aleksandr Beznosikov
1MIRAI, Moscow, Russia · 2BRAIn Lab, Moscow, Russia · 3Yandex Research, Moscow, Russia +1
Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Targeted interventions establish the functional significance of these patterns. Replacing the late mixers by identity increases C4 perplexity by only 1.9% and preserves the six-task average score, whereas replacing the early mixers increases perplexity by 41%. Fixing each early mixer to its C4 diagnostic mean increases perplexity by only 0.2% and reduces the average score by 0.25 percentage points, showing that its site-specific structure matters more than its token-wise variation on the evaluated metrics. Likewise, retaining the three largest routing weights per token at every site increases perplexity by at most 2.7% and changes the average score by at most 0.4 points. Thus, the studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.