Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.
Figures & tables
Figure 1: Illustration of our layer retrieval ability estimation method (described in Section 3.2 ). As an example, the sequence to copy is “123”, and the layer under inspection is layer k . The difference in probability change tells us the contribution of layer k in retrieving the target token “9”.
Figure 2
Figure 4: Demonstration of Competing Learning. The curves indicate the retrieval ability of attention layers in a hybrid model consisting of attention and recurrent layers (KDA), trained on our retrieval task, Spaced Retrieval (Section 3.1 ).
Figure 5: Relative gradient magnitude changes in the Attention layers compared to the baseline without KDA freezing. KDA is frozen at steps 100, 150, or 175, with an unfreezing branch introduced 50 steps later. Across these timings, keeping KDA frozen progressively amplifies attention gradient norms relative to the baseline, whereas unfreezing KDA substantially attenuates this amplification.
Figure 5
Size
Model
LM Loss ↓
CSR ↑
Recall ↑
RULER ↑
WikiText ↓
LongPPL ↓
180M
Vanilla
3.5903
37.76
10.39
2.84
54.00
258.47
180M
BLD
3.5702
37.49
14.12
8.82
50.38
108.64
250M
Vanilla
3.2844
38.31
21.54
17.32
36.52
57.22
250M
BLD
3.2642
41.49
25.33
20.98
34.27
17.96
390M
Vanilla
3.1611
41.91
26.73
22.32
29.10
17.09
390M
BLD
3.1441
44.25
31.27
26.89
28.11
11.16
Table 1: Zero-shot evaluation results of BLD (ours) versus vanilla pre-training. Size denotes total parameters, including embeddings, rounded to the nearest 10M. CSR and Recall are macro averages over seven and six tasks, respectively. RULER averages 57 task–length entries at 1K–16K. All three scores are percentages. Bold indicates the better result at each model size.
Figure 8: Left: Loss as a function of non-embedding model size, fitted with power-plus-constant scaling laws. Right: The training-loss curve of the 1B vanilla and BLD models.
Architecture
Size
Model
LM Loss ↓
CSR ↑
Recall ↑
RULER ↑
WikiText ↓
LongPPL ↓
GDN
400M
Vanilla
3.1947
43.23
17.33
12.33
31.25
152.84
GDN
400M
BLD
3.1624
43.90
32.05
25.42
28.69
11.77
GLA
400M
Vanilla
3.2911
41.19
24.58
23.98
34.57
32.69
GLA
400M
BLD
3.2113
43.20
30.12
28.54
30.65
13.26
Table 2: Zero-shot evaluation results of BLD (ours) versus vanilla pre-training with GDN and GLA hybrid models with 3:1 recurrence-to-attention ratio.
Figure 9: The α values of attention heads. With GQA, query heads 0–3 and 4–7 share a KV head within each group and therefore share the same KV-head-level α value. Greater α values indicate heads that are important for long-distance retrieval.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Key length / g
Five digits; g∈{64,8192}
Leading spaces
0–20
Vocabulary
12 symbols: ten digits, delimiter, and space/padding
Training / evaluation
50,000 distinct keys / 2,000 held-out pairs per g
Decoder blocks
KDA–attention–KDA–attention
Hidden / FFN width
256 / 512
Appendix
Table 3: The configuration of the dataset (Spaced Retrieval) and the models in Section 3.3 .
Setting
Value
Optimizer / updates
AdamW ( Loshchilov and Hutter, 2017 ) / 1,500
Batch
32 examples per update
Learning rate
Peak 3×10−4 ; final 10−5
Schedule
100 warmup updates, followed by cosine decay
Adam coefficients
(β1,β2)=(0.9,0.95)
Weight decay / gradient clip
0.1 / global norm 1.0
Appendix
Table 4: Synthetic training and attribution evaluation settings.
Setting
Value
Frozen parameters
Entire KDA blocks 0 and 2, including mixer, MLP, and normalization
Table 8: Architecture settings shared across KDA language models, in Section 4.2 .
Setting
Early pretraining
Remaining pretraining
DCLM data view
Global shard 01
Global shards 02–10
Token fraction
Approximately 1/11 (9.09%)
Approximately 10/11 (90.91%)
Vanilla parameters
All trainable
All trainable
BLD: frozen components
Entire KDA blocks: mixer, MLP, and normalization
None
BLD: trainable components
Attention blocks, tied embeddings, and final normalization
All components
Weight / optimizer initialization
Train from scratch
Continue from current weights; reset optimizer and schedule
Appendix
Table 9: Freezing configuration of BLD in pretraining experiments in Section 4.2 .
Model size
Early pretraining tokens
Remaining pretraining tokens
180M
120M
1.20B
250M
280M
2.80B
390M
480M
4.80B
510M
720M
7.20B
750M
1.07B
10.65B
1B
1.60B
16.00B
Appendix
Table 10: Training token budgets for the LLM pretraining experiments in Section 4.2 , rounded for readability.
Setting
Early pretraining
Remaining pretraining
Optimizer
AdamW ( Loshchilov and Hutter, 2017 )
Muon ( Jordan et al., 2024 ) with auxiliary AdamW
Peak learning rate
10−4
Muon: 0.02; AdamW: 0.001
Minimum learning rate
10−4 (no terminal decay)
Muon: 0.001; AdamW: 5×10−5
Schedule
Linear warmup, then constant
Linear warmup, constant interval, then cosine decay
Adam coefficients
(0.9,0.95)
(0.9,0.95)
Muon momentum / iterations
Not applicable
0.95 / 5 Newton–Schulz iterations
Appendix
Table 11: Optimization settings shared by Vanilla and BLD.
Size
Early warmup
Remaining warmup
Final decay
GPUs
Accum.
180M
23
229
229
1
16
250M
53
534
534
1
16
390M
92
916
916
2
8
510M
137
1,373
1,373
2
8
750M
198
2,032
2,032
2
8
1B
305
3,052
3,052
8
2
Appendix
Table 12: Size-dependent schedules and batching. Warmup and decay durations are in optimizer updates; accumulation is unchanged throughout pretraining.
Setting
Value
Dataset / sample
GovReport ( Huang et al., 2021 ) / 50 documents
Sampling seed / token limit
42 / 32,768 per document
Input formatting
Raw text; no chat template or additional BOS
Reference model
Qwen3-8B ( Yang et al., 2025a ) ; fixed annotations across compared models
Short context / stride
K=4096 / d=1024
Selected targets
3,920 tokens across 49 documents
Appendix
Table 13: Configuration for calculating LongPPL in Section 4.2 .
Figure 11: RULER accuracy by context length for six KDA model sizes. Blue dashed lines denote Vanilla; orange solid lines denote BLD. Darker shades indicate larger models. Each point averages the tasks evaluated at that length.
Size / mixer
Method
ARC-E
ARC-C
Hella.
PIQA
BoolQ
Wino.
LAMBADA
Mean
180M
Vanilla
36.36
22.44
27.38
58.27
58.87
51.07
9.94
37.76
180M
BLD
35.65
22.44
27.64
59.30
55.84
49.96
11.59
37.49
250M
Vanilla
38.43
23.98
29.80
58.98
45.05
51.38
20.57
38.31
250M
BLD
41.16
23.72
30.33
62.35
60.58
48.70
23.56
41.49
390M
Vanilla
43.73
24.74
34.31
64.20
47.68
49.25
29.46
41.91
390M
BLD
43.43
24.57
35.35
65.40
58.75
50.75
31.48
44.25
Appendix
Table 14: The concrete results of the hybrid models on CSR (Section 4.2 ). LAMBADA reports accuracy. Means use unrounded scores from all seven tasks. GDN and GLA denote the transfer configurations in Appendix H . Unless otherwise specified, all models are KDA-based hybrid models.
Size / mixer
Method
SWDE
FDA
SQuAD
TriviaQA
DROP
NQ
Mean
180M
Vanilla
9.09
0.54
13.40
21.74
13.08
4.50
10.39
180M
BLD
13.95
13.43
14.68
23.52
14.23
4.91
14.12
250M
Vanilla
27.90
17.15
21.88
33.12
19.45
9.72
21.54
250M
BLD
29.79
30.85
25.00
35.25
18.93
12.13
25.33
390M
Vanilla
34.74
26.04
27.38
38.68
20.17
13.34
26.73
390M
BLD
38.34
40.83
30.09
41.35
19.84
17.14
31.27
Appendix
Table 15: Recall results using the contains metric (percent). Means use unrounded scores. Unless otherwise specified, all models are KDA-based hybrid models.
Size / mixer
Method
1K (10)
2K (11)
4K (12)
8K (12)
16K (12)
All (57)
180M
Vanilla
8.60
2.91
1.50
1.08
1.08
2.84
180M
BLD
21.20
13.36
5.67
4.25
2.08
8.82
250M
Vanilla
27.40
23.91
14.50
13.50
9.50
17.32
250M
BLD
29.00
22.91
19.17
19.17
16.17
20.98
390M
Vanilla
32.00
25.27
21.25
18.08
16.83
22.32
390M
BLD
36.20
30.45
26.50
25.08
18.08
26.89
Appendix
Table 16: RULER scores by context length (percent). Parentheses give task counts; All averages the 57 task–length entries. Unless otherwise specified, all models are KDA-based hybrid models.
Task
180M V
180M B
250M V
250M B
390M V
390M B
Single-1
8.20
34.80
96.20
100.00
100.00
98.40
Single-2
8.60
40.40
41.00
31.40
62.80
72.80
Single-3
0.00
5.40
28.60
69.00
49.00
78.60
Multi-key-1
6.60
11.20
15.80
20.80
26.40
32.40
Multi-key-2
0.00
0.00
0.20
0.40
0.80
3.20
Multi-key-3
0.00
0.00
0.00
1.60
0.80
2.00
Appendix
Table 17: Task-wise RULER scores averaged over evaluated lengths (percent). V: Vanilla; B: BLD. QA-1 averages four lengths, QA-2 three, and other tasks five.
Task
510M V
510M B
750M V
750M B
1B V
1B B
Single-1
100.00
98.80
99.60
100.00
99.80
100.00
Single-2
67.80
99.00
79.60
90.40
83.00
92.80
Single-3
73.80
60.20
60.60
81.40
71.60
77.00
Multi-key-1
26.60
48.40
39.20
61.00
61.80
58.40
Multi-key-2
1.00
2.80
2.80
5.40
26.60
30.80
Multi-key-3
1.60
3.80
2.20
6.60
2.60
6.60
Appendix
Table 18: Task-wise RULER scores averaged over evaluated lengths (percent). V: Vanilla; B: BLD. QA-1 averages four lengths, QA-2 three, and other tasks five.
Task
GDN V
GDN B
GLA V
GLA B
Single-1
99.60
99.60
94.80
99.80
Single-2
18.20
86.00
84.80
93.40
Single-3
1.00
44.20
55.00
79.20
Multi-key-1
13.20
38.00
27.20
33.40
Multi-key-2
0.00
1.20
0.60
2.60
Multi-key-3
0.00
2.60
1.00
1.60
Appendix
Table 19: Task-wise RULER scores averaged over evaluated lengths (percent). V: Vanilla; B: BLD. QA-1 averages four lengths, QA-2 three, and other tasks five.
Loss window
E
AV
αV
AB
αB
RMSE (V)
RMSE (B)
Final 100
2.38867
0.45960
0.34649
0.45266
0.34502
0.01321
0.01424
Appendix
Table 20: Shared-floor fits for the original-checkpoint six-size series. AV,αV are scaling law fitted coefficients for the vanilla model while AB,αB are those for the BLD models. RMSE columns are Vanilla/BLD.
Setting
Value
Dataset
First 50 nonempty documents from each local Books3 and BookCorpus2 cache from The Pile ( Gao et al., 2020 ) (100 total)
Input processing
Independent documents; no added special tokens; cached token IDs and checked tokenization
Input / prediction limit
16,384 input tokens / 16,384 predictions per document
Execution
Full decoder prefix; chunked output projection; original position settings
Recurrent-attention hybrid language models (LMs), which interleave attention and recurrent layers, are increasingly used to combine the efficiency of the recurrent layers with the strong performance of attention layers. Prior work suggests that attention and recurrent layers offer complementary pathways to use past information: attention supports precise memory recall from earlier tokens, while recurrent layers support consolidation of disparate information over long contexts. However, we observe that simply having access to both pathways does not mean that hybrid LMs are effectively using them. We find that they rely substantially more on attention than on the recurrent state. Standard supervised fine-tuning improves overall performance but does not improve how the two memory pathways are coordinated: the model becomes more reliant on information propagated by attention layers, while its use of information propagated by recurrent layers remains limited. To encourage better coordination between the two memory pathways, we add an auxiliary loss that limits attention's access to earlier context while the recurrent state propagates through the full sequence. This objective encourages the model to retain and use information through the recurrent pathway alongside attention. It improves overall performance, with particularly strong gains on tasks involving longer contexts or requiring information aggregation, consistent with the strengths of recurrent layers observed in analysis. Crucially, this imbalance and the benefit of our auxiliary loss generalize: they apply to multiple recurrent-attention LMs in question-answering and agentic tasks, as well as to attention-based LMs that combine different forms of memory. Together, our findings show that simply providing multiple memory pathways does not ensure their effective use, and that targeted supervision is needed to better coordinate them.
Hyunji Lee, Joykirat Singh, Zaid Khan +5
UNC Chapel Hill · The University of Texas at Austin · Mila +1
Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from one context with the recurrent state from another in a single forward pass. On Qwen3.5 and Falcon-H1, the two channels split sharply by function. Exact retrieval survives only through attention (64-98% of full accuracy) and collapses to zero through recurrence. Output language and persona reverse the pattern: both survive recurrence (70-80% and 3-5x) while KV-only drops to ~1% language accuracy. State-swap confirms this causally: the answer takes its value from the KV side and its language from the recurrent side. Recurrent-only generation also accepts words that were never in the context but share meaning or parts with seen items. Attention provides a lookup over what was said; the recurrent state shapes how the model says it next.
Kirill Afendulev, Alexey Dontsov, Elena Tutubalina +1
AIRI, Moscow, Russia · YSDA, Moscow, Russia · HSE University, Moscow, Russia
Softmax attention is the cornerstone of modern large language models, but its memory scales linearly and compute quadratically with sequence length. Linear recurrent models, such as linear attention and state space models, have become widely studied as alternatives to attention due to their linear compute and constant memory. While these sub-quadratic token mixing methods, or mixers, achieve promising efficiency gains and competitive results on a wide range of benchmarks, current linear recurrent models still lag behind on tasks that require long-context retrieval or in-context learning. A growing body of work studies hybrid architectures that attempt to mitigate these trade-offs by statically interleaving or merging attention and recurrent blocks. In this work, we explore a new axis of developing hybrid models: across the token sequence. We propose Oryx, a hybrid model that can, throughout a sequence, flexibly switch between different mixers, for example quadratic attention for rich context utilization and linear recurrences for efficient generation. Oryx ties at least 90% of its parameters across mixers, enabling attention and recurrent modes to operate over shared internal representations. We validate our design with Mamba-2 and Gated DeltaNet variants, up to 1.4B models. Under fixed token budgets and a mixed-training strategy, Oryx achieves comparable or better performance than its single-mixer baselines. At the 1.4B scale, all instances of Oryx outperform their respective baselines by at least 0.7 percentage points on averaged language modeling tasks. On retrieval tasks, Oryx achieves performance comparable to the Transformer baseline even when processing only a tiny fraction (<10%) of the tokens in attention mode. These results suggest that attention and linear recurrent models can share internal representations, and motivate sequence-axis hybridization as a promising direction.
Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1