Manifold-constrained Hyper-Connections (mHC) extend a Transformer's residual stream into multiple parallel streams with learned cross-stream connections. Prior work trains all streams toward a shared objective. We instead study whether assigning distinct learning objectives to different streams can improve task performance and induce objective-specific specialization. Using complementary lexical (sparse) and semantic (dense) retrieval objectives as a test case, we find that objective-specialized mHC consistently outperforms a non-specialized mHC model across the full sparse--dense fusion range, for both two- and four-stream models and at two model scales. We then examine how specialization changes the model internally using probes of stream representations, residual mixing, objective-specific gradient flow, and read/write gates. The specialized and non-specialized models behave similarly through most of the network but diverge in the final layers: the specialized model develops increasingly objective-specific gradient flow, distinct read/write routing patterns, and reduced cross-stream mixing, and streams assigned to different objectives become increasingly dissimilar while those trained toward the same objective remain highly similar. Fixing the residual mixing to the identity leaves the specialization benefit intact, indicating that the benefit stems from objective-specific read/write routing rather than learned cross-stream mixing. These results show that assigning distinct objectives to mHC streams can improve multi-objective retrieval while inducing structured, depth-dependent specialization in the residual computation.
Figures & tables
Figure 1: Schematic comparison of the three encoder configurations studied in this work. In all three, streams propagate through the full encoder stack and the retrieval heads read only the final-layer ( l=L ) stream representations. (left) Single-stream : the standard ModernBERT encoder, with a single residual stream read by both the sparse and dense retrieval heads. (middle) Non-specialized mHC : the residual stream is expanded into n parallel streams with learned cross-stream routing. Both retrieval heads operate on the mean-pooled representation obtained from all n streams, so the sparse and dense objectives jointly optimize the same stream representation. (right) Objective-specialized mHC : the sparse and dense heads instead read disjoint sets of final-layer streams, so each objective selectively trains its assigned stream(s). For clarity the schematic depicts n=2 streams; we study both n=2 and n=4 in our experiments.
Model
ArguAna
C-FEVER
DBPedia
FEVER
FiQA
HotpotQA
NFC
NQ
Quora
SciDocs
SciF
TREC-C
Touché
Avg
n=1
Sparse-only
0.3943
0.1052
0.1484
0.6384
0.1953
0.5895
0.2219
0.1600
0.6722
0.1119
0.6492
0.5381
0.2372
0.3585
Dense-only
0.3974
0.2116
0.2702
0.5049
0.2373
0.3797
0.2460
0.4569
0.8479
0.1328
0.5513
0.6784
0.2201
0.3950
Sparse+Dense
0.3833
0.2217
0.2552
0.6525
0.2649
0.5678
0.2611
0.4110
0.8274
0.1310
0.6707
0.6990
0.2643
0.4315
n=2
mHC no-spec
0.3827
0.2132
0.2579
0.6529
0.2718
0.5696
0.2708
0.4079
0.8229
0.1346
0.6709
0.6843
0.2634
0.4310
Table 1: BEIR nDCG@10 across all 13 datasets using ModernBERT-base. Sparse-only and Dense-only are evaluated in their native retrieval mode; Sparse+Dense and the mHC models are hybrid retrievers fused at α=0.9 , the shared empirical optimum across all multi-objective models. Underline marks the best score within each stream count n ; bold marks the best score for that dataset across all configurations. The advantage of objective specialization is not confined to the aggregate score; it extends to a majority of individual datasets and holds at both n=2 and n=4 . Excluding the in-domain NQ dataset, mHC+spec still improves over mHC no-spec on the remaining 12 datasets ( 0.4329→0.4470 at n=2 ; 0.4431→0.4510 at n=4 ). Results reported here are mean over two seeds. Per seed variation is provided in Appendix 5 . Results with ModernBERT-large (Table 4 ) show the same trend: mHC+spec improves over mHC no-spec at both n=2 and n=4 , indicating that the benefit of objective specialization is robust to model scale rather than specific to a single model size.
Figure 2: Average BEIR nDCG@10 as a function of the fusion weight α , for ModernBERT-base (left) and ModernBERT-large (right); both mean over two seeds. α=0 and α=1 correspond to pure sparse and pure dense retrieval, respectively. Color encodes objective specialization (red: specialized, blue: non-specialized); Sparse+Dense ( n=1 , grey) has no stream routing. Specialized models (red) dominate at every α for both stream counts and at both base/large model scales.
Figure 3: Four depth-resolved specialization probes at the final checkpoint, for n=2 (top) and n=4 (bottom) streams (lines are the mean and shaded bands ±1 standard deviation over 4 seeds; flat layers 0 – 9 omitted). From left: stream cosine similarity, residual-mixing identity retention diag(A) , per-objective gradient share on the assigned stream(s), and read/write gate shift. Red: specialized; blue: non-specialized control. Solid vs. dashed distinguishes the two objectives / gates / stream-pair types (legends). Metrics are defined and discussed in Section 5 .
Model
Mixing A
nDCG@10 ( α=0.9 )
mHC no-spec
learned
0.4410±0.0023
mHC spec
learned
0.4529±0.0019
mHC no-spec
identity
0.4415±0.0025
mHC spec
identity
0.4546±0.0001
Table 2: Fixing the residual-mixing matrix to the identity ( Aℓ=I at every layer). ModernBERT-base, n=4 , average BEIR nDCG@10 over 13 datasets at α=0.9 ; mean ± standard deviation over two seeds.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Shape
Initialization
Wpre
n×nd
0
Wpost
n×nd
0
Wres
n2×nd
0
bpre
n
logit(1/n)+ε , renorm.
bpost
n
0
bres
n2
diag 1 , off-diag 0
Appendix
Table 3: Initialization of each mHC block. Projection weights are zero-initialized so routing starts static; the biases place the block near an identity residual layer. The read-gate bias uses the uniform-noisy scheme (target σ(bkpre)=1/n plus noise ε∼N(0,σnoise2) , σnoise=10−2 , renormalized to sum to one) for both specialized and non-specialized models. At initialization this yields read gate p≈[1/n,…,1/n] (summing to 1 ), write gate q=[1,…,1] , and a doubly-stochastic mixing matrix A with its largest mass on the diagonal.
Figure 4: Attention-sublayer routing probes at the final checkpoint (lines are the mean and shaded bands ±1 standard deviation over 4 seeds; flat layers 0 – 9 omitted). Left pair: n=2 ; right pair: n=4 . Within each pair: mean diagonal of the residual-mixing matrix diag(A) , and the shift in each gate’s sparse-stream share Δ (read gate p solid, write gate q dashed). Red: specialized; blue: non-specialized control. Like the MLP sublayer, attention drives A toward the identity under specialization; unlike the MLP, its read gate does not shift ( Δs(pℓ)≈0 ) while its write gate moves strongly toward the dense-assigned stream(s) ( Δs(qL)≈−0.44 ), i.e. attention specializes its write but not its read. Metrics defined in Section 5 .
Model
ArguAna
C-FEVER
DBPedia
FEVER
FiQA
HotpotQA
NFC
NQ
Quora
SciDocs
SciF
TREC-C
Touché
Avg
n=1
Sparse-only
0.3967
0.0954
0.1348
0.5931
0.2133
0.5568
0.2251
0.1654
0.6761
0.1118
0.6507
0.5178
0.2307
0.3514
Dense-only
0.4110
0.2157
0.3072
0.5351
0.3103
0.4061
0.2800
0.5063
0.8562
0.1611
0.5972
0.6996
0.2205
0.4236
Sparse+Dense
0.4268
0.2204
0.2711
0.6586
0.3095
0.5838
0.2828
0.4488
0.8443
0.1398
0.6970
0.7614
0.2939
0.4568
n=2
mHC no-spec
0.4103
0.2108
0.2838
0.6698
0.3076
0.5864
0.2983
0.4546
0.8453
0.1421
0.6911
0.7621
0.2836
0.4574
Appendix
Table 4: BEIR nDCG@10 across all 13 datasets using ModernBERT-large, mean over two seeds (42 and 1234). Sparse-only and Dense-only are evaluated in their native retrieval mode; Sparse+Dense and the mHC models are hybrid retrievers fused at α=0.9 . Underline marks the best score within each stream count n ; bold marks the best score for that dataset across all configurations.
Model
Seed
ArguAna
C-FEVER
DBPedia
FEVER
FiQA
HotpotQA
NFC
NQ
Quora
SciDocs
SciF
TREC-C
Touché
Avg
n=1
Sparse-only
42
0.4038
0.1040
0.1504
0.6151
0.1998
0.5857
0.2238
0.1637
0.6823
0.1134
0.6555
0.5704
0.2269
0.3611
1234
0.3849
0.1064
0.1463
0.6617
0.1908
0.5932
0.2200
0.1562
0.6620
0.1105
0.6428
0.5057
0.2476
0.3560
Dense-only
42
0.3946
0.2119
0.2690
0.5025
0.2419
0.3785
0.2529
0.4606
0.8488
0.1315
0.5600
0.6725
0.2214
0.3959
1001
0.4003
0.2112
0.2715
0.5072
0.2326
0.3808
0.2392
0.4532
0.8471
0.1341
0.5426
0.6843
0.2187
0.3941
Sparse+Dense
42
0.3944
0.2022
0.2581
0.6382
0.2676
0.5657
0.2622
0.4047
0.8289
0.1309
0.6622
0.6797
0.2615
0.4274
Appendix
Table 5: Per-seed BEIR nDCG@10 across all 13 datasets (hybrid fusion at α=0.9 ) for ModernBERT-base.
Model
Seed
ArguAna
C-FEVER
DBPedia
FEVER
FiQA
HotpotQA
NFC
NQ
Quora
SciDocs
SciF
TREC-C
Touché
Avg
n=1
Sparse-only
42
0.4111
0.0981
0.1326
0.5833
0.2124
0.5491
0.2278
0.1701
0.6799
0.1143
0.6582
0.5424
0.2262
0.3543
1234
0.3822
0.0926
0.1371
0.6028
0.2142
0.5644
0.2223
0.1608
0.6722
0.1092
0.6433
0.4932
0.2351
0.3484
Dense-only
42
0.4107
0.2158
0.3080
0.5403
0.3078
0.4052
0.2777
0.5047
0.8563
0.1606
0.5944
0.7053
0.2233
0.4239
1234
0.4112
0.2156
0.3063
0.5299
0.3127
0.4070
0.2823
0.5079
0.8562
0.1616
0.5999
0.6938
0.2178
0.4232
Sparse+Dense
42
0.4291
0.2133
0.2670
0.6538
0.3027
0.5829
0.2872
0.4477
0.8459
0.1391
0.7031
0.7472
0.2915
0.4547
Appendix
Table 6: Per-seed BEIR nDCG@10 across all 13 datasets using ModernBERT-large, for each model and training seed. Sparse-only and Dense-only are evaluated in their native retrieval mode; the remaining models are hybrid retrievers fused at α=0.9 . The two-seed means are the values reported in Table 4 . As at the base scale, the specialization gain holds within each seed: at both n=2 and n=4 , mHC+spec exceeds mHC no-spec in every seed.