Multi-Stream Attention Residuals

Xiaohan Qin; Xiangdong Zhang; Tuo Dai; Sunan Zou; Huaijin Wu; Yebin Yang; Zhuo Xia; Lin Yao

cs.CLOct 9, 2026

Multi-Stream Attention Residuals

Authors: Xiaohan Qin, Xiangdong Zhang, Tuo Dai, Sunan Zou, Huaijin Wu, Yebin Yang, Zhuo Xia, Lin Yao, +2 more

Organizations: School of AI, Shanghai Jiao Tong University · Shanghai Innovation Institute · Dots Studio, Xiaohongshu Inc. · School of CS, Peking University · Independent Researcher

Abstract

Attention Residuals (AttnRes) enables content-dependent retrieval over network depth, which can be viewed as single-stream softmax attention along the depth axis. Inspired by the success of multi-head attention along the sequence axis, we introduce Multi-Stream Attention Residuals (MSAR), an extension of AttnRes that generalizes this mechanism to multi-head depth attention, yielding substantial performance gains with minimal additional overhead. MSAR draws stream diversity from sequence context through multi-scale causal convolutions, allowing different heads to retrieve historical representations through distinct contextual views while preserving the integrity of the hidden representation. The retrieved streams are combined through lightweight adaptive mixing to generate the single-stream layer input. By retaining single-stream sources and expanding them on demand, MSAR supports multi-stream computation without requiring persistent multi-stream storage or communication. Across all 12 downstream benchmarks at the 18B and 28B MoE scales, MSAR consistently outperforms prior residual variants, including standard residual connections, manifold-constrained Hyper-Connections (mHC), and AttnRes. Scaling-law experiments yield an iso-loss compute speedup of $1.58\times$ over the standard residual baseline, compared with $1.35\times$ for AttnRes and $1.30\times$ for mHC. These results make MSAR a promising residual architecture for large-scale pre-training.

Figures & tables

Panel of Figure 1 — Figure 1: MSAR improves optimization and downstream performance at 18B scale. MSAR achieves lower training loss and higher average downstream score than baseline residual variants.

Figure 2: Comparison of residual architectures. (a) Standard Residuals; (b) Block AttnRes; (c) Block MSAR. Block MSAR expands single-stream sources into contextual streams for per-stream depth attention, then fuses the retrieved streams into a single input for each residual branch.

	18B				28B
Benchmark	Vanilla	mHC	AttnRes	MSAR	Vanilla	mHC	AttnRes	MSAR
General
MMLU	48.9	54.7	53.0	56.6	54.6	56.8	59.9	62.5
MMLU-Pro	21.1	27.4	23.9	27.9	30.1	34.9	36.0	38.7
MMLU-Redux	46.4	49.9	49.4	53.7	50.6	53.9	55.7	58.7
ARC-C	55.7	66.3	68.7	74.9	70.8	74.9	75.6	78.0

Table 1: Downstream performance comparison of MSAR with Vanilla, mHC, and AttnRes on 18B and 28B MoE models. Scores are reported in percent (higher is better), with the best result for each benchmark at each scale in bold .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Attribute	10B	18B	28B
Total Parameters	10B	18B	28B
Activated Parameters	1.1B	1.7B	2.7B
Transformer Layers	15	28	32
Leading Dense Layers	1	1	1
MoE Layers	14	27	31
Hidden Dimension	2080	2112	2560

Appendix

Table 3: Backbone configurations and training hyperparameters. The 10B model is used for ablation studies, and the 18B and 28B models are used for the main results. Activated parameter counts are approximate and exclude input embeddings.

Total	Activated	Layers	Hidden	Dense	Expert	Q/KV	Training	Global	Learning	Layers per Block
Parameters †	Parameters †	Layers	Dimension	FFN	FFN	Heads	Budget	Batch Size	Rate	(AttnRes/MSAR)
2.21B	0.34B	15	1024	3104	320	8/4	-	160	$6.95\times 10^{-4}$	2
3.75B	0.46B	23	1040	3072	352	8/4	-	208	$5.74\times 10^{-4}$	3
9.47B	1.10B	15	2080	6144	704	16/8	-	288	$4.82\times 10^{-4}$	2
17.40B	1.77B	28	2112	6144	672	16/8	-	384	$3.97\times 10^{-4}$	4

Appendix

Table 4: Scaling-law configurations. All methods use matched backbones and training-token budgets at each scale. The block size is shared by AttnRes and MSAR.

	18B		28B
Method	$\Delta P/P_{\mathrm{act}}$	$\Delta F/F_{\mathrm{Vanilla}}$	$\Delta P/P_{\mathrm{act}}$	$\Delta F/F_{\mathrm{Vanilla}}$
mHC	$0.67\%$	$0.66\%$	$0.58\%$	$0.60\%$
AttnRes	$0.01\%$	$0.10\%$	$0.01\%$	$0.10\%$
MSAR	$0.31\%$	$0.51\%$	$0.27\%$	$0.48\%$

Appendix

Table 5: Residual-module parameter and training FLOPs overheads.

\Delta P/P_{\mathrm{act}}

is relative to Vanilla’s activated parameters (approximately 1.7B and 2.7B at the two scales);

\Delta F/F_{\mathrm{Vanilla}}

is relative to its total training FLOPs, including the LM head.

Method	Operation	Read	Write	Total I/O
Method	Operation	Read	Write	Symbolic	Typical
Vanilla	Residual merge	$2d$	$d$	$3d$	$3d$
mHC	Compute mixing coefficients	$md$	$-$	$(8m+2)d$	$34d$
	Apply $\mathcal{H}^{\mathrm{pre}}$	$md$	$d$
	Apply $\mathcal{H}^{\mathrm{post}}$	$d$	$md$
	Apply $\mathcal{H}^{\mathrm{res}}$	$md$	$md$

Appendix

Table 6: Memory access cost per token per residual sublayer incurred by each residual mechanism during the forward pass. We report leading terms proportional to

d

, excluding parameter traffic and the internal I/O of

f_{l}

. Phase 1 costs are averaged across depth and amortized over the

S

targets in each block. The Typical column uses

N=8

blocks,

S=16

residual sublayers per block (128 residual sublayers in total, corresponding to 64 Transformer layers), and

m=4

streams.

Figure 4: Token-averaged depth-attention weights of the 10B Block MSAR model. Rows show Pre-Attn (top) and Pre-MLP (bottom); columns show the four streams, numbered 1–4. Source 0 is the embedding, and sources 1–8 are blocks, with the current block represented by its nonempty partial sum. Hatched regions mark unavailable sources.

Method	Update rule	Weight	Source
Single-state recurrence
Vanilla	$\bm{h}_{l}=\bm{h}_{l-1}+f_{l-1}(\bm{h}_{l-1})$	Fixed	$\bm{h}_{l-1}$
ReZero [ Bachlechner et al., 2020 ]	$\bm{h}_{l}=\bm{h}_{l-1}+\alpha_{l}f_{l-1}(\bm{h}_{l-1})$	Static	$\bm{h}_{l-1}$
Highway [ Srivastava et al., 2015 ]	$\bm{h}_{l}=(1-\bm{g}_{l})\odot\bm{h}_{l-1}+\bm{g}_{l}\odot f_{l-1}(\bm{h}_{l-1})$	Dynamic	$\bm{h}_{l-1}$
Multi-state recurrence
SiameseNorm [ Li et al., 2026 ]	$\bm{h}_{l}^{1}=\operatorname{Norm}(\bm{h}_{l-1}^{1}+\bm{y}_{l-1});\hskip 8.19447pt\bm{h}_{l}^{2}=\bm{h}_{l-1}^{2}+\bm{y}_{l-1}$	Fixed	$[\bm{h}_{l-1}^{1},\bm{h}_{l-1}^{2}]$

Appendix

Table 7: Comparison of residual update mechanisms. Weight distinguishes architecture-fixed, learned-static (input-independent after training), and input-dependent (dynamic) mixing coefficients. Source indicates the representations directly accessible to the residual mechanism. Branch normalization is absorbed into

f_{l}

; depth-attention weights are normalized over available sources.

Explore similar work

Dual Attention Residuals
Jul 21, 2026Xingda Yu, Yining Li, Xinzhang Liu +5Transformer Cross-Attention
TeleAI
Multi-Head Attention Residuals
Jul 22, 2026Cheng Luo, Zefan Cai, Junjie HuTransformer Attention Residual Connections
Independent Researcher · University of Wisconsin–Madison
Multi-Gate Residuals
May 22, 2026Zhizhan Zheng, Feiyun Zhang, Shuchun Liu +4Residual Learning
Shanghai Yichuang Information Technology Co.,Ltd. · Independent Researcher · Fudan University

Jul 21, 2026J/K move · Enter open · S save

Dual Attention Residuals

Xingda Yu, Yining Li, Xinzhang Liu, Zhihao Yang+4

TeleAI

Transformer Cross-Attention Attention Residual Connections Residual Learning

Open paper AskPDF arXiv

Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain multiple residual trajectories. These capabilities have largely been studied in isolation, and assigning an independent retriever to each stream still prevents one trajectory from influencing depth selection in another. We propose Dual Attention Residuals (DAR), which brings multi-stream interaction into historical retrieval through reciprocal cross-stream addressing. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history. The retrieved states are combined for an unchanged Transformer branch and updated through constrained gated writes; a block-form variant operates on block-level histories to control overhead. Across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model, DAR consistently improves validation loss over standard residual Transformers and Attention Residuals. Routing ablations show that the gain cannot be explained by an additional stream or value projection alone. Representation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.

Abstract

Explore similar work

Dual Attention Residuals

Multi-Head Attention Residuals

Multi-Gate Residuals