Competing Learning in Hybrid Architectures

Xingyu Shen; Yingfa Chen; Ziqing Qiao; Chaojun Xiao; Xu Han; Zhiyuan Liu

cs.CLOct 19, 2026

Competing Learning in Hybrid Architectures

Authors: Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu

Organizations: Department of Science and Technology, Tsinghua University, Beijing, China

Abstract

Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.

Figures & tables

Figure 1: Illustration of our layer retrieval ability estimation method (described in Section 3.2 ). As an example, the sequence to copy is “123”, and the layer under inspection is layer $k$ . The difference in probability change tells us the contribution of layer $k$ in retrieving the target token “9”.

Figure 4: Demonstration of Competing Learning. The curves indicate the retrieval ability of attention layers in a hybrid model consisting of attention and recurrent layers (KDA), trained on our retrieval task, Spaced Retrieval (Section 3.1 ).

Figure 5: Relative gradient magnitude changes in the Attention layers compared to the baseline without KDA freezing. KDA is frozen at steps 100, 150, or 175, with an unfreezing branch introduced 50 steps later. Across these timings, keeping KDA frozen progressively amplifies attention gradient norms relative to the baseline, whereas unfreezing KDA substantially attenuates this amplification.

Size	Model	LM Loss $\downarrow$	CSR $\uparrow$	Recall $\uparrow$	RULER $\uparrow$	WikiText $\downarrow$	LongPPL $\downarrow$
180M	Vanilla	3.5903	37.76	10.39	2.84	54.00	258.47
180M	BLD	3.5702	37.49	14.12	8.82	50.38	108.64
250M	Vanilla	3.2844	38.31	21.54	17.32	36.52	57.22
250M	BLD	3.2642	41.49	25.33	20.98	34.27	17.96
390M	Vanilla	3.1611	41.91	26.73	22.32	29.10	17.09
390M	BLD	3.1441	44.25	31.27	26.89	28.11	11.16

Table 1: Zero-shot evaluation results of BLD (ours) versus vanilla pre-training. Size denotes total parameters, including embeddings, rounded to the nearest 10M. CSR and Recall are macro averages over seven and six tasks, respectively. RULER averages 57 task–length entries at 1K–16K. All three scores are percentages. Bold indicates the better result at each model size.

Figure 8: Left: Loss as a function of non-embedding model size, fitted with power-plus-constant scaling laws. Right: The training-loss curve of the 1B vanilla and BLD models.

Architecture	Size	Model	LM Loss $\downarrow$	CSR $\uparrow$	Recall $\uparrow$	RULER $\uparrow$	WikiText $\downarrow$	LongPPL $\downarrow$
GDN	400M	Vanilla	3.1947	43.23	17.33	12.33	31.25	152.84
GDN	400M	BLD	3.1624	43.90	32.05	25.42	28.69	11.77
GLA	400M	Vanilla	3.2911	41.19	24.58	23.98	34.57	32.69
GLA	400M	BLD	3.2113	43.20	30.12	28.54	30.65	13.26

Table 2: Zero-shot evaluation results of BLD (ours) versus vanilla pre-training with GDN and GLA hybrid models with 3:1 recurrence-to-attention ratio.

Figure 9: The $\alpha$ values of attention heads. With GQA, query heads 0–3 and 4–7 share a KV head within each group and therefore share the same KV-head-level $\alpha$ value. Greater $\alpha$ values indicate heads that are important for long-distance retrieval.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Key length / $g$	Five digits; $g\in\{64,8192\}$
Leading spaces	0–20
Vocabulary	12 symbols: ten digits, delimiter, and space/padding
Training / evaluation	50,000 distinct keys / 2,000 held-out pairs per $g$
Decoder blocks	KDA–attention–KDA–attention
Hidden / FFN width	256 / 512

Appendix

Table 3: The configuration of the dataset (Spaced Retrieval) and the models in Section 3.3 .

Setting	Value
Optimizer / updates	AdamW ( Loshchilov and Hutter, 2017 ) / 1,500
Batch	32 examples per update
Learning rate	Peak $3\times 10^{-4}$ ; final $10^{-5}$
Schedule	100 warmup updates, followed by cosine decay
Adam coefficients	$(\beta_{1},\beta_{2})=(0.9,0.95)$
Weight decay / gradient clip	0.1 / global norm 1.0

Appendix

Table 4: Synthetic training and attribution evaluation settings.

Setting	Value
Frozen parameters	Entire KDA blocks 0 and 2, including mixer, MLP, and normalization
Trainable components	Attention blocks and shared parameters
Updates completed before freezing ( $s$ )	0, 1, 10, 25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 600
Permanent freeze	Keep KDA blocks frozen through update 1,500

Appendix

Table 5: Synthetic freezing and unfreezing protocol for the experiment in Section 3.3.1 .

Figure 10: Attention retrieval ability under KDA freezing and unfreezing at $g=64$ , using the same setup as in Figure 4 .

Setting	Value
Short-distance fraction $p$	0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1
Training distances	$g=64$ and $g=8192$
Optimizer updates	1,500
Examples per update	32
Evaluation distances	$g=64$ and $g=8192$

Appendix

Table 6: Training-data proportions and settings for the data-distribution experiment in Section 3.3.3 .

Model size	Total params (M)	Non-emb. params (M)	Decoder blocks	Hidden dim.	FFN dim.	KDA heads	Attn. heads (Q/KV)
180M	181	63	8	768	1792	6	12/3
250M	254	136	16	768	2048	6	6/2
390M	391	234	16	1024	2688	8	8/2
510M	508	351	24	1024	2688	8	8/2
750M	755	520	16	1536	4096	12	12/3
1B	1015	780	24	1536	4096	12	12/3

Appendix

Table 7: KDA hybrid model configurations. Parameter counts are rounded to the nearest million; tied embeddings are counted once.

Setting	Value
Tokenizer / vocabulary	Qwen3 ( Yang et al., 2025a ) / 153,000 entries
Repeating block pattern	Attention–KDA–KDA–KDA; KDA-to-attention ratio 3:1
Embeddings / FFN	Tied input/output embeddings / SwiGLU ( Shazeer, 2020 )
Normalization	RMSNorm ( Zhang and Sennrich, 2019 ) , $\epsilon=10^{-6}$
KDA head dimension	128
KDA value expansion / convolution width	2 / 4

Appendix

Table 8: Architecture settings shared across KDA language models, in Section 4.2 .

Setting	Early pretraining	Remaining pretraining
DCLM data view	Global shard 01	Global shards 02–10
Token fraction	Approximately $1/11$ (9.09%)	Approximately $10/11$ (90.91%)
Vanilla parameters	All trainable	All trainable
BLD: frozen components	Entire KDA blocks: mixer, MLP, and normalization	None
BLD: trainable components	Attention blocks, tied embeddings, and final normalization	All components
Weight / optimizer initialization	Train from scratch	Continue from current weights; reset optimizer and schedule

Appendix

Table 9: Freezing configuration of BLD in pretraining experiments in Section 4.2 .

Model size	Early pretraining tokens	Remaining pretraining tokens
180M	120M	1.20B
250M	280M	2.80B
390M	480M	4.80B
510M	720M	7.20B
750M	1.07B	10.65B
1B	1.60B	16.00B

Appendix

Table 10: Training token budgets for the LLM pretraining experiments in Section 4.2 , rounded for readability.

Setting	Early pretraining	Remaining pretraining
Optimizer	AdamW ( Loshchilov and Hutter, 2017 )	Muon ( Jordan et al., 2024 ) with auxiliary AdamW
Peak learning rate	$10^{-4}$	Muon: 0.02; AdamW: 0.001
Minimum learning rate	$10^{-4}$ (no terminal decay)	Muon: 0.001; AdamW: $5\times 10^{-5}$
Schedule	Linear warmup, then constant	Linear warmup, constant interval, then cosine decay
Adam coefficients	$(0.9,0.95)$	$(0.9,0.95)$
Muon momentum / iterations	Not applicable	0.95 / 5 Newton–Schulz iterations

Appendix

Table 11: Optimization settings shared by Vanilla and BLD.

Size	Early warmup	Remaining warmup	Final decay	GPUs	Accum.
180M	23	229	229	1	16
250M	53	534	534	1	16
390M	92	916	916	2	8
510M	137	1,373	1,373	2	8
750M	198	2,032	2,032	2	8
1B	305	3,052	3,052	8	2

Appendix

Table 12: Size-dependent schedules and batching. Warmup and decay durations are in optimizer updates; accumulation is unchanged throughout pretraining.

Setting	Value
Dataset / sample	GovReport ( Huang et al., 2021 ) / 50 documents
Sampling seed / token limit	42 / 32,768 per document
Input formatting	Raw text; no chat template or additional BOS
Reference model	Qwen3-8B ( Yang et al., 2025a ) ; fixed annotations across compared models
Short context / stride	$K=4096$ / $d=1024$
Selected targets	3,920 tokens across 49 documents

Appendix

Table 13: Configuration for calculating LongPPL in Section 4.2 .

Figure 11: RULER accuracy by context length for six KDA model sizes. Blue dashed lines denote Vanilla; orange solid lines denote BLD. Darker shades indicate larger models. Each point averages the tasks evaluated at that length.

Size / mixer	Method	ARC-E	ARC-C	Hella.	PIQA	BoolQ	Wino.	LAMBADA	Mean
180M	Vanilla	36.36	22.44	27.38	58.27	58.87	51.07	9.94	37.76
180M	BLD	35.65	22.44	27.64	59.30	55.84	49.96	11.59	37.49
250M	Vanilla	38.43	23.98	29.80	58.98	45.05	51.38	20.57	38.31
250M	BLD	41.16	23.72	30.33	62.35	60.58	48.70	23.56	41.49
390M	Vanilla	43.73	24.74	34.31	64.20	47.68	49.25	29.46	41.91
390M	BLD	43.43	24.57	35.35	65.40	58.75	50.75	31.48	44.25

Appendix

Table 14: The concrete results of the hybrid models on CSR (Section 4.2 ). LAMBADA reports accuracy. Means use unrounded scores from all seven tasks. GDN and GLA denote the transfer configurations in Appendix H . Unless otherwise specified, all models are KDA-based hybrid models.

Size / mixer	Method	SWDE	FDA	SQuAD	TriviaQA	DROP	NQ	Mean
180M	Vanilla	9.09	0.54	13.40	21.74	13.08	4.50	10.39
180M	BLD	13.95	13.43	14.68	23.52	14.23	4.91	14.12
250M	Vanilla	27.90	17.15	21.88	33.12	19.45	9.72	21.54
250M	BLD	29.79	30.85	25.00	35.25	18.93	12.13	25.33
390M	Vanilla	34.74	26.04	27.38	38.68	20.17	13.34	26.73
390M	BLD	38.34	40.83	30.09	41.35	19.84	17.14	31.27

Appendix

Table 15: Recall results using the contains metric (percent). Means use unrounded scores. Unless otherwise specified, all models are KDA-based hybrid models.

Size / mixer	Method	1K (10)	2K (11)	4K (12)	8K (12)	16K (12)	All (57)
180M	Vanilla	8.60	2.91	1.50	1.08	1.08	2.84
180M	BLD	21.20	13.36	5.67	4.25	2.08	8.82
250M	Vanilla	27.40	23.91	14.50	13.50	9.50	17.32
250M	BLD	29.00	22.91	19.17	19.17	16.17	20.98
390M	Vanilla	32.00	25.27	21.25	18.08	16.83	22.32
390M	BLD	36.20	30.45	26.50	25.08	18.08	26.89

Appendix

Table 16: RULER scores by context length (percent). Parentheses give task counts; All averages the 57 task–length entries. Unless otherwise specified, all models are KDA-based hybrid models.

Task	180M V	180M B	250M V	250M B	390M V	390M B
Single-1	8.20	34.80	96.20	100.00	100.00	98.40
Single-2	8.60	40.40	41.00	31.40	62.80	72.80
Single-3	0.00	5.40	28.60	69.00	49.00	78.60
Multi-key-1	6.60	11.20	15.80	20.80	26.40	32.40
Multi-key-2	0.00	0.00	0.20	0.40	0.80	3.20
Multi-key-3	0.00	0.00	0.00	1.60	0.80	2.00

Appendix

Table 17: Task-wise RULER scores averaged over evaluated lengths (percent). V: Vanilla; B: BLD. QA-1 averages four lengths, QA-2 three, and other tasks five.

Task	510M V	510M B	750M V	750M B	1B V	1B B
Single-1	100.00	98.80	99.60	100.00	99.80	100.00
Single-2	67.80	99.00	79.60	90.40	83.00	92.80
Single-3	73.80	60.20	60.60	81.40	71.60	77.00
Multi-key-1	26.60	48.40	39.20	61.00	61.80	58.40
Multi-key-2	1.00	2.80	2.80	5.40	26.60	30.80
Multi-key-3	1.60	3.80	2.20	6.60	2.60	6.60

Appendix

Table 18: Task-wise RULER scores averaged over evaluated lengths (percent). V: Vanilla; B: BLD. QA-1 averages four lengths, QA-2 three, and other tasks five.

Task	GDN V	GDN B	GLA V	GLA B
Single-1	99.60	99.60	94.80	99.80
Single-2	18.20	86.00	84.80	93.40
Single-3	1.00	44.20	55.00	79.20
Multi-key-1	13.20	38.00	27.20	33.40
Multi-key-2	0.00	1.20	0.60	2.60
Multi-key-3	0.00	2.60	1.00	1.60

Appendix

Table 19: Task-wise RULER scores averaged over evaluated lengths (percent). V: Vanilla; B: BLD. QA-1 averages four lengths, QA-2 three, and other tasks five.

Loss window	$E$	$A_{V}$	$\alpha_{V}$	$A_{B}$	$\alpha_{B}$	RMSE (V)	RMSE (B)
Final 100	2.38867	0.45960	0.34649	0.45266	0.34502	0.01321	0.01424

Appendix

Table 20: Shared-floor fits for the original-checkpoint six-size series.

A_{V},\alpha_{V}

are scaling law fitted coefficients for the vanilla model while

A_{B},\alpha_{B}

are those for the BLD models. RMSE columns are Vanilla/BLD.

Setting	Value
Dataset	First 50 nonempty documents from each local Books3 and BookCorpus2 cache from The Pile ( Gao et al., 2020 ) (100 total)
Input processing	Independent documents; no added special tokens; cached token IDs and checked tokenization
Input / prediction limit	16,384 input tokens / 16,384 predictions per document
Execution	Full decoder prefix; chunked output projection; original position settings
Precision	BF16 model execution; FP32 logits

Appendix

Table 21: Position-wise language-modeling diagnostic settings.

Figure 12: Average loss of KDA hybrid models of different scales, as functions of token positions.

Setting	GDN	GLA
Blocks / hidden / FFN width	16 / 1,024 / 2,688	16 / 1,024 / 2,688
Attention block indices	0, 4, 8, 12	0, 4, 8, 12
Attention Q / KV heads	8 / 2	8 / 2
Recurrent blocks / heads	12 / 8	12 / 8
Recurrent dimensions	Head dimension 128	Key expansion 1
Value expansion	2	2

Appendix

Table 22: Configurations for the GDN and GLA models in Section 4.4 . Parameter and token counts are rounded; M and B denote million and billion.

Setting	Value
Optimization steps / batch size	1,000 / 1 sequence
Optimizer	AdamW, no weight decay
Peak / minimum learning rate	0.02 / 0.002
Schedule	200-step warmup and final 200-step decay
Sequence length	16,384 tokens
Passkeys per sequence	10, each containing 32 words

Appendix

Table 23: Settings for retrieval head identification experiments in Section 4.5 .

Setting	Value
Mixer order	KDA–Attention–KDA–Attention
Total model parameters	3,555,588
Independent seeds	10
Frozen modules	Both KDA mixers
Frozen parameter count	1,453,060
Measured parameters	262,144 per attention mixer

Appendix

Table 24: Experimental settings for Figure 5 .

Hybrid Language Models

See all 14 papers

Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs
Oct 5, 2026Hyunji Lee, Joykirat Singh, Zaid Khan +5Long-Context Language Modeling Recurrent Transformers
UNC Chapel Hill · The University of Texas at Austin · Mila +1
What Attention Recalls and Recurrence Controls in Hybrid Language Models
Sep 3, 2026Kirill Afendulev, Alexey Dontsov, Elena Tutubalina +1Hybrid Attention Recurrent Transformers
AIRI, Moscow, Russia · YSDA, Moscow, Russia · HSE University, Moscow, Russia
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
May 27, 2026Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1Parameter Sharing Transformer Attention
Carnegie Mellon University · Google Research

Oct 5, 2026J/K move · Enter open · S save

Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs

Hyunji Lee, Joykirat Singh, Zaid Khan, Justin Chih-Yao Chen+4

UNC Chapel Hill · The University of Texas at Austin · Mila+1

Long-Context Language Modeling Recurrent Transformers Hybrid Attention

Open paper AskPDF arXiv

Recurrent-attention hybrid language models (LMs), which interleave attention and recurrent layers, are increasingly used to combine the efficiency of the recurrent layers with the strong performance of attention layers. Prior work suggests that attention and recurrent layers offer complementary pathways to use past information: attention supports precise memory recall from earlier tokens, while recurrent layers support consolidation of disparate information over long contexts. However, we observe that simply having access to both pathways does not mean that hybrid LMs are effectively using them. We find that they rely substantially more on attention than on the recurrent state. Standard supervised fine-tuning improves overall performance but does not improve how the two memory pathways are coordinated: the model becomes more reliant on information propagated by attention layers, while its use of information propagated by recurrent layers remains limited. To encourage better coordination between the two memory pathways, we add an auxiliary loss that limits attention's access to earlier context while the recurrent state propagates through the full sequence. This objective encourages the model to retain and use information through the recurrent pathway alongside attention. It improves overall performance, with particularly strong gains on tasks involving longer contexts or requiring information aggregation, consistent with the strengths of recurrent layers observed in analysis. Crucially, this imbalance and the benefit of our auxiliary loss generalize: they apply to multiple recurrent-attention LMs in question-answering and agentic tasks, as well as to attention-based LMs that combine different forms of memory. Together, our findings show that simply providing multiple memory pathways does not ensure their effective use, and that targeted supervision is needed to better coordinate them.

Abstract

Hybrid Language Models

Explore similar work

Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs

What Attention Recalls and Recurrence Controls in Hybrid Language Models

Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations