Hybrid attention is rapidly becoming the dominant architectural choice for frontier models. Despite their successes, such architectures complicate model design by introducing additional choices, including the types and locations of layers. This combinatorially large design space has been studied empirically, but given the vast expense of training and evaluating models with different configurations, a principled understanding of the effects of layer selection and ordering is needed. We provide the first such analysis, showing that the dominant hybrid architecture is highly task-specific. Concretely, for any layer configuration, we prove that there exists a task where we can construct a model with this configuration that provably outperforms any other model of any different configuration under fixed resource constraints. Empirically, we observe that for standard training, learned hybrid architectures exhibit separations between layer orderings that match our theoretical findings. For instance, under the same memory budget, the optimal architecture predicted by our theory for a segmented retrieval task consistently achieves perfect accuracy, whereas the next strongest plateaus below 0.5. These results extend to deeper models, where ``containing'' the dominant architecture for a task indicates whether the model attains high performance. In addition, we investigate the mechanisms learned by hybrids, observing that Transformer layers contribute the largest share of performance on associative recall across architectures. This work establishes insights into task-dependent hybrid architecture design relevant to the training and scaling of frontier-scale hybrids.
Figures & tables
Figure 1: Structure of tasks that exhibit performance differences among different hybrid architectures. Large gaps need SSM layers to communicate each segment’s query, while the dense computation per segment is done by the Transformers.
Figure 2: Results for the TS tasks where an inference memory proxy is held constant. The annotations represent the relationship between window storage and state size for Transformer-Mamba hybrids. The only architecture with high performance on the task is the TS architecture. Due to fixed memory, models with larger token dimension have smaller window and state size, making the tasks difficult. Conversely, small token dimension limits expressivity.
Figure 3Figure 4
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
V
The vocabulary space.
U
The intermediate space used between architecture layers.
Y
The output space.
S
An SSM layer.
T
A TF layer.
Σ
Defined to be {S,T} , the possible letters in an architecture.
Appendix
Table 2: Notation
Figure 6: Extended results for segmented retrieval.
Figure 7: Hybrid performance on exact copy for 1024 dimensions of memory.
Figure 8: Hybrid performance on exact copy for 256 dimensions of memory.
Model
Performance
Model
Performance
TTTST
0.911
TTTTT
0.533
TTSTT
0.934
TTTTS
0.840
TTSTS
0.884
TTTSS
0.776
TTSST
0.890
TTSSS
0.787
TSTTT
0.922
STTTT
0.527
TSTTS
0.923
STTTS
0.842
Appendix
Table 3: Quantitative results for a deeper hybrids on the TST task. The left architectures contain TST as a sub-architecture, while the other three do not.
Figure 9: Results for training various architectures with different token dimensions on the TST task. All models used the same window size and state size. As expected, only the TST model attains a high accuracy.
Figure 10: Performance on segmented retrieval, including Gated DeltaNet ( G ) hybrids. Only the 15:1 ratio of window to SSM state for TS and ST hybrids is shown.
Figure 11: Performance of architectures trained on selective copy. We observe that the ST architecture is largely dominant, as expected.
Figure 12: Token accuracy of T∗S∗ and S∗T∗ architectures on (2,1) associative recall. Small differences in architectures affect performance dramatically.
Figure 13: Learning rate sweep for select architectures trained on associative recall.
Figure 14: Structure of Exact Copy. At each position t>l , the TF layer retrieves xt−l . The first SSM layer counts consecutive matches xt=xt−l in Ht , while the second sets yt=1 once Ht=(c−1)l and retains this value thereafter. Here, abc repeats three times consecutively, giving H10=6 and yL=1 despite subsequent mismatches.
Figure 15: Structure of Segmented Retrieval. At each internal boundary tj , the TF layer retrieves vj=xtj−λ . The SSM layer retains the most recent retrieved token belonging to N and outputs it at position L . Here, the retrieved tokens are A,u,B,v , so the output is B .
Hybrid sequence models--combining Transformer and state-space model layers--seek to gain the expressive versatility of attention as well as the computational efficiency of state-space model layers. Despite burgeoning interest in hybrid models, we lack a basic understanding of the settings where--and underlying mechanisms through which--they offer benefits over their constituent models. In this paper, we study this question, focusing on a broad family of core synthetic tasks. For this family of tasks, we prove the existence of fundamental limitations for non-hybrid models. Specifically, any Transformer or state-space model that solves the underlying task requires either a large number of parameters or a large working memory. On the other hand, for two prototypical tasks within this family--namely selective copying and associative recall--we construct hybrid models of small size and working memory that provably solve these tasks, thus achieving the best of both worlds. Our experimental evaluation empirically validates our theoretical findings. Importantly, going beyond the settings in our theoretical analysis, we empirically show that learned--rather than constructed--hybrids outperform non-hybrid models with up to 6x as many parameters. We additionally demonstrate that hybrid models exhibit stronger length generalization and out-of-distribution robustness than non-hybrids.
John Cooper, Ilias Diakonikolas, Mingchen Ma +1
Department of Computer Sciences, University of Wisconsin-Madison
Hybrid language models that interleave attention with recurrent components are increasingly competitive with pure Transformers, yet standard LoRA practice applies adapters uniformly without considering the distinct functional roles of each component type. We systematically study component-type LoRA placement across two hybrid architectures -- Qwen3.5-0.8B (sequential, GatedDeltaNet + softmax attention) and Falcon-H1-0.5B (parallel, Mamba-2 SSM + attention) -- fine-tuned on three domains and evaluated on five benchmarks. We find that the attention pathway -- despite being the minority component -- consistently outperforms full-model adaptation with 5-10x fewer trainable parameters. Crucially, adapting the recurrent backbone is destructive in sequential hybrids (-14.8 pp on GSM8K) but constructive in parallel ones (+8.6 pp). We further document a transfer asymmetry: parallel hybrids exhibit positive cross-task transfer while sequential hybrids suffer catastrophic forgetting. These results establish that hybrid topology fundamentally determines adaptation response, and that component-aware LoRA placement is a necessary design dimension for hybrid architectures.
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Transformers face quadratic complexity and memory issues with long sequences, prompting the adoption of linear attention mechanisms using fixed-size hidden states. However, linear models often suffer from limited recall performance, leading to hybrid architectures that combine linear and full attention layers. Despite extensive hybrid architecture research, the choice of linear attention component has not been deeply explored. We systematically evaluate various linear attention models across generations - vector recurrences to advanced gating mechanisms - both standalone and hybridized. To enable this comprehensive analysis, we trained and open-sourced 72 models: 36 at 340M parameters (20B tokens) and 36 at 1.3B parameters (100B tokens), covering six linear attention variants across five hybridization ratios. Benchmarking on standard language modeling and recall tasks reveals that superior standalone linear models do not necessarily excel in hybrids. While language modeling remains stable across linear-to-full attention ratios, recall significantly improves with increased full attention layers, particularly below a 3:1 ratio. Our study highlights selective gating, hierarchical recurrence, and controlled forgetting as critical for effective hybrid models. We recommend architectures such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1 to achieve Transformer-level recall efficiently. Our models are open-sourced at https://huggingface.co/collections/m-a-p/hybrid-linear-attention-research-686c488a63d609d2f20e2b1e.
Dustin Wang, Rui-Jie Zhu, Steven Abreu +9
1UC Santa Cruz · 2ByteDance Seed · University of Groningen +3