Olmo Hybrid: From Theory to Practice and Back

William Merrill; Yanhong Li; Tyler Romero; Anej Svete; Caia Costello; Pradeep Dasigi; Dirk Groeneveld; David Heineman

cs.LGApr 3, 2026

Olmo Hybrid: From Theory to Practice and Back

Authors: William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, +14 more

Organizations: Allen Institute for AI · ETH Zürich · Lambda ML Team · University of Washington · University of Cambridge

Abstract

Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention. Yet there is no consensus on whether the potential benefits of these new architectures justify the risk and effort of scaling them up. To address this, we provide evidence for the advantages of hybrid models over pure transformers on several fronts. First, theoretically, we show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both, such as code execution. Putting this theory to practice, we train Olmo Hybrid, a 7B-parameter model largely comparable to Olmo 3 7B but with the sliding window layers replaced by Gated DeltaNet layers. We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations, demonstrating the benefit of hybrid models in a controlled, large-scale setting. We find that the hybrid model scales significantly more efficiently than the transformer, explaining its higher performance. However, its unclear why greater expressivity on specific formal problems should result in better scaling or superior performance on downstream tasks unrelated to those problems. To explain this apparent gap, we return to theory and argue why increased expressivity should translate to better scaling efficiency, completing the loop. Overall, our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm: not merely to reduce memory during inference, but as a fundamental way to obtain more expressive models that scale better during pretraining.

Figures & tables

Layer Type	Elements	FP16 Size	vs. GDN
Multi-Head Attention ( $32$ K seq, $32$ KV heads, $d_{h}{=}128$ )	$268.4$ M	$512$ MiB	$485\times$
Grouped-Query Attention ( $32$ K seq, $8$ KV heads, $d_{h}{=}128$ )	$67.1$ M	$128$ MiB	$121\times$
Grouped-Query SWA ( $4096$ window, $8$ KV heads, $d_{h}{=}128$ )	$8.39$ M	$16.0$ MiB	$15.2\times$
Olmo Hybrid GDN ( $30$ heads, $d_{k}{=}96$ , $d_{v}{=}192$ )	$0.55$ M	$1.05$ MiB	—

Table 1: Per-layer inference state size comparison under the Olmo 3 configuration stored in fp16.

Figure 1: Olmo Hybrid 7B is more efficient than OLMo 3 7B during pretraining, reaching the same Common Crawl loss in 35% fewer tokens and the same MMLU accuracy using 49% fewer tokens (and thus also 35% and 49% fewer FLOPs, respectively).

	Base Aggregate Scores						Held-out Scores
Model		Math	Code	$\textbf{\penalty\ MC}_{\textbf{\penalty\ STEM}}$	$\textbf{\penalty\ MC}_{\textbf{\penalty\ Non-STEM}}$	GenQA	LBPP	BBH	MMLU Pro	DM Math
Olmo 3 7B Pretrain		23.3	19.6	64.0	71.9	68.5	6.3	48.6	31.2	15.7
Olmo Hybrid 7B Pretrain		27.0	17.1	67.4	75.5	66.8	2.9	52.2	35.7	13.2
Olmo 3 7B Midtrain		59.8	32.1	67.3	78.2	71.3	17.8	65.9	36.6	23.8
Olmo Hybrid 7B Midtrain		61.2	32.9	70.9	81.3	73.6	17.7	68.6	43.1	23.7
Olmo 3 7B LC		54.6	30.9	66.2	78.2	72.5	17.7	64.0	37.2	23.6

Table 2: Base model evaluations across the training pipeline for Olmo Hybrid vs. Olmo 3. After mid-training, Olmo Hybrid outperforms Olmo 3 7B across every evaluation domain. On evaluations held out from Olmo 3 development (right), Olmo Hybrid outperforms Olmo 3 on MMLU Pro and BBH but loses slightly on LBPP and DM Math. Overall, the results suggest that Olmo Hybrid 7B generally outperforms Olmo 3 7B.

		RULER Scores
Model	LC Method	4k	8k	16k	32k	64k
Olmo 3 7B	YaRN	95.8	89.3	83.2	78.9	70.9
Olmo Hybrid 7B	YaRN	92.8	91.3	90.0	84.7	76.9
Olmo Hybrid 7B	DroPE	92.2	89.8	88.4	86.2	85.0

Table 3: Results for Olmo Hybrid after long-context tension with YaRN and DroPE compared against Olmo 3. Both methods outperform Olmo 3 7B at long sequence lengths, with the gains from DroPE being particularly notable.

Figure 2: Compute-performance tradeoff of open-weight hybrid, RNN, and transformer base models on the average of OlmoBaseEval task suites. Olmo Hybrid 7B is on the Pareto frontier of open-weight dense models. Training compute was estimated using the

C=6ND

heuristic from Kaplan et al. (2020) , with reported token and parameter counts. Per-benchmark results are reported in Table 6 . While theoretical compute for MoE-based SSMs is low, effective training efficiency can be roughly 50–80% of a dense model ( Rajbhandari et al., 2022 ) , so we draw our frontier over dense models only. For this plot, we show only models obtaining >50% average performance on OlmoBaseEval .

Panel of Figure 3 — Figure 3: Expressive power of transformers, linear RNNs, and hybrid models relative to circuit complexity classes. Dashed lines represent unproven but conjectured separations between classes (e.g., $\mathsf{TC}^{0}\neq\mathsf{NC}^{1}$ ). Notably, transformers can express recall ( Section 3.1 ), and DeltaNet (or GDN) with negative eigenvalues can express state tracking ( Section 3.2 ) ( Grazzi et al., 2025 ) . Hybridizing gives both capabilities, and we prove that it also unlocks state-based recall, a problem that neither model can express on its own ( Section 3.3 ). With the addition of padding tokens, transformers can express exactly the class $\mathsf{TC}^{0}$ , whereas hybrid models can capture all of $\mathsf{NC}^{1}$ , which enables solving boolean formula evaluation ( Section 3.4 ).

Figure 6: Let

\mathbf{k}_{t}^{\top}=(1,-1)/\sqrt{2}

; crucially,

\left\lVert\mathbf{k}_{t}\right\rVert=1

. Multiplying the low-rank term in the GDN update by 2 allows the transition matrix to have a negative eigenvalue, meaning it can implement dynamics that alternate between states ( Grazzi et al., 2025 ) . This is useful for expressing state tracking tasks like parity and the

A_{5}

word problem.

Figure 7: Synthetic task results. (a) State tracking accuracy vs. number of updates

n

: linear RNN and hybrid remain near-perfect, while the transformer falls quickly. (b) Recall accuracy vs. bit-array size

m

: transformer and hybrid maintain perfect recall, while the linear RNN degrades for large

m

. (c) State-based recall accuracy vs. number of swaps

n

(

m=n

): the hybrid remains robust, while both the transformer and linear RNN degrade for large

m,n

Figure 8: Scaling law fits

L(N,D)=E+A/N^{\alpha}+B/D^{\beta}

for all architectures. Transformer:

\alpha=0.252

\beta=0.213

R^{2}=0.998

. Hybrid:

\alpha=0.226

\beta=0.219

R^{2}=0.999

. Pure GDN:

\alpha=0.183

\beta=0.227

R^{2}=0.999

. (a) Loss vs. compute; points represent model checkpoints at different model sizes (shown by opacity) and the bold curve shows the fitted scaling law at the compute-optimal frontier. (b) Loss vs. parameter count; points represent checkpoints at different Chinchilla multiples (

D/N

ratios, shown by opacity) and the fitted curve is evaluated at the largest multiple. (c) Loss vs. data budget; points represent checkpoints at different model sizes (shown by opacity) and the fitted curve is evaluated at the largest model size. Loss axes are plotted on a logarithmic scale so that the power-law relationships appear linear.

Architecture	Model	$N$	$D$	Observed	Predicted	Error (%)
Transformer	Olmo 3	7B	5.9T	2.17	2.16	0.28
Transformer	Olmo 3	32B	5.5T	2.02	2.05	1.69
Hybrid GDN	Olmo Hybrid	7B	5.5T	2.14	2.14	0.28

Table 4: Scaling law prediction validation against final models. Predicted loss is computed from the fitted law

L(N,D)=E+A/N^{\alpha}+B/D^{\beta}

. Extrapolation is generally strong despite differences in the learning rate schedule between our scaling studies and the large-scale training runs.

Figure 9: Scaling law coefficients

E

A

B

with 95% bootstrap CIs for the fixed-exponent fit (

\alpha{=}\beta{=}0.22

shared), which yields tight confidence intervals. The data efficiency coefficient

B

shows a robust advantage for Hybrid GDN over the transformer (

83.7

vs.

94.9

, non-overlapping CIs, marked

*

), while differences in

E

and

A

are not statistically conclusive.

Figure 10: Projected savings factors across model scales (target loss

=2.474

, selected as the minimum of all observed training losses). (a) Parameter savings (

N_{\mathrm{ref}}/N_{\mathrm{arch}}

) vs training tokens: fewer parameters are needed to reach the same loss at a given data budget. (b) Data savings (

D_{\mathrm{ref}}/D_{\mathrm{arch}}

) vs model size: fewer training tokens are needed to reach the target loss at a given model size. (c) Compute savings (

C_{\mathrm{ref}}/C_{\mathrm{arch}}

C\propto N{\cdot}D

) vs target loss: for each target loss the minimum total compute is found by optimising over model size

N

; harder targets (lower loss) are on the right. Dashed vertical lines mark the observed losses of our final production models. The savings factor grows with difficulty for both Pure GDN and Hybrid models. Values above

1\times

indicate an advantage over Transformer. Note that estimates at lower loss values (right side of the plot) involve extrapolation beyond the fitting range and are therefore less certain.

Figure 11: Impact of expressivity

1-\epsilon

on loss under three instantiations of the expressivity-aware quantization model; both axes are log-scaled. In the first row,

\Delta^{\prime}<\Delta

, so decreasing

\epsilon

lowers irreducible loss ( Corollary 4.2 ) and reduces loss everywhere ( Corollary 4.1 ), but especially for large

N

and

D

. In the second row,

\Delta^{\prime}=\Delta

, but inexpressible tasks require more parameters and tokens than expressible tasks. Thus, while increasing expressivity shifts down the scaling curve everywhere, irreducible loss remains the same, so for large enough

N,D

, the loss reduction diminishes. Finally, the third row incorporates both effects of expressivity. This means decreasing

\epsilon

both causes an initial difference in loss and lowers irreducible loss, leading to a visible gap in loss as

\epsilon

decreases across values of

N

and

D

Panel of Figure 12 — Figure 12: Architecture configurations evaluated in our ablation study. We compare pure architectures (Transformer, GDN, Mamba2) against hybrid variants with different linear-to-attention ratios (1:1, 3:1, 7:1), RNN backbones (GDN vs. Mamba2 at 3:1), and placement strategies (interleaved vs. middle). The highlighted configuration (3:1 interleaved with GDN) was selected for the final Olmo Hybrid.

Architecture	Attn %	60M	100M	190M	370M	600M	760M	1B
Pure Architectures
Transformer	100%	1.155	1.037	0.950	0.839	0.781	0.747	0.682
GDN	0%	1.080 †	0.990 †	0.895 †	0.795	0.761 †	0.722 †	0.677 †
Mamba2	0%	1.146	1.036	0.941	0.843	0.787	0.750	0.718
Hybrid: Interleaved Attention
GDN (1:1)	50%	1.093	0.992	0.896	0.804	0.787	0.724	0.674

Table 5: Architecture ablation results — averages. Average OlmoBaseEval BPB (across Math, Code, QA) for pure and hybrid architectures at 7 representative scales. Bold indicates best; underline second best; † best within section.

\bigstar

marks our selected architecture. Note that per-size comparisons should be interpreted with care, as architectures at the same nominal scale differ in actual parameter count (see Table 22 ).

	Knowledge & Reasoning					Math				Code			Chat & IF
Model	MMLU	PopQA	BBH	GPQA	Zebra	MATH	$\Omega$	AIME’25	AIME’24	HE+	MBPP+	LCB	IFEval	IFB	AE3
Olmo 3 Think SFT	74.9	20.8	84.1	45.8	57.9	94.4	37.8	57.6	69.6	88.2	63.2	67.8	77.9	30.0	43.9
Olmo Hybrid Think SFT	80.5	25.1	84.6	47.0	55.1	93.8	35.1	55.2	66.2	86.3	63.7	65.5	80.4	31.6	49.0
Olmo 3 Instruct SFT	67.1	16.5	51.0	30.0	18.0	65.1	14.4	7.2	6.7	69.8	56.5	20.0	81.7	27.4	21.8
Olmo Hybrid Instruct SFT	71.9	16.8	47.3	36.8	17.0	66.7	16.0	8.8	6.7	69.2	55.3	21.3	81.5	29.0	25.6
Olmo 3 DPO	69.1	20.7	69.3	37.9	28.4	79.6	22.8	20.4	23.5	72.9	55.9	18.8	82.0	29.3	43.3

Table 7: Post-training evaluation comparison between Olmo 3 (dense) and Olmo Hybrid (hybrid) across three training stages: Think SFT, Instruct SFT, and DPO.

\Omega

= Omega Full. HE+ = HumanEvalPlus. IFB = IFBench. AE3 = AlpacaEval 3.

Model	Metric	1K	4K	8K	16K	32K
Olmo 3 7B (MHA)	Tokens/sec/node	2,909	2,164	1,510	1,690	1,247
	Inference GPUs (RL batch 1K)	32	48	56	64	96
	Generation wall time (h)	10.5	56.7	162.5	283.7	768.6
	GPU-hours ( $\times$ 1K)	0.3	2.7	9.1	18.2	73.8
Olmo Hybrid 7B	Tokens/sec/node	3,039	3,077	2,736	1,876	1,422
	Inference GPUs (RL batch 1K)	8	16	24	40	64

Table 8: Inference-bound scaling for async RL generation across context lengths. In async RL, each training step requires generating a full batch of rollouts before the policy update; generation throughput is the bottleneck. We measure the minimum GPU allocation needed to serve an RL batch of 1,024 rollouts concurrently and project generation wall time and total GPU-hours over 1,725,440 episodes (1,684 steps).

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Model	# Heads	$d_{\textrm{model}}$	# Parameters	Training Throughput (TPS)
Olmo 3	32	4096	6.8B	8.0K
Olmo Hybrid	32	4096	7.7B	7.7K
	31	3968	7.4B	7.7K
	30	3840	7.0B	8.2K

Appendix

Table 9: Rough training throughput measurements run before launching the Olmo Hybrid pretraining run.

Figure 13: SpikeScore grows throughout training for Olmo 3, but it stays more stable when training Olmo Hybrid.

	7B Think SFT	7B Instruct SFT	7B Instruct DPO
Instances	2,932,239	2,153,716	259,922
Total Tokens	47.6B	3.4B	N/A
Batch Size	1M tokens	1M tokens	N/A
Learning Rate	$2.5\times 10^{-5}$	$2.5\times 10^{-5}$	$1\times 10^{-6}$
Num. GPUs	64	64	32
Max Sequence Length	32K	32K	16K

Appendix

Table 10: Training hyperparameters for Olmo Hybrid 7B. Total tokens includes masked tokens (e.g. prompts). Think SFT total tokens is computed from the Olmo 3 baseline (45.4B) plus the

3\times

tool-use upscale:

45.4\text{B}\times(1+2\times 2.47\%)=47.6\text{B}

	Olmo 3	Olmo Hybrid
Data Parallel	HSDP (shard within node)	FSDP (full sharding)
Context Parallel	Ring (degree ${}=8$ )	Ulysses (degree ${}=2$ )
Activation Checkpointing	Selected modules ( feed_forward )	Budget-based (0.1)

Appendix

Table 11: Differences between Olmo 3 and Olmo Hybrid SFT training configurations.

	Task	Format	Metric	Temp	Top-p	Ans. Extract	Max Toks	N	# Sub

Chat Suite

	IF Eval ( 2023 )	CoT	Custom	0.6	0.95	Custom	32768	1	-
	IFBench ( 2025 )	CoT	Custom	0.6	0.95	Custom	32768	1	-
	MATH 500 ( 2022 ; 2023 )	CoT EM	EM Flex	0.6	0.95	Minerva	32768	1	-

Appendix

Table 13: Details of the post-training evaluation suite , as used in OLMo 3 ( Olmo Team, 2025 ) . We mark tasks with * to indicate new additions compared to the OLMo 2 suite ( Olmo Team, 2024 ) . All evaluation generations have thinking traces (text between <think>...</think> ) stripped before passing to the answer scorer. We use zero-shot setting for all metrics.

Shared hyperparameters	Value
Layers ( $L$ )	$4$
Model width ( $d_{\text{model}}$ )	$256$
FFN intermediate size ( $d_{\text{ff}}$ )	$1024$
Attention heads ( $H$ )	$4$
Head dimension ( $d_{\text{head}}$ )	$64$
Max positions	$1024$ (Recall) / $4096$ (State Tracking, State-based Recall)

Appendix

Table 14: Architecture hyperparameters used for synthetic evaluations.

$n$	Transformer	Linear RNN (neg. eig.)	Linear RNN (pos. eig.)	Hybrid (neg. eig.)	Hybrid (pos. eig.)
4	0.51172	1.00000	1.00000	1.00000	1.00000
8	0.27734	1.00000	1.00000	1.00000	1.00000
16	0.17188	1.00000	0.97266	1.00000	1.00000
32	0.23047	1.00000	0.64453	1.00000	1.00000
64	0.19922	1.00000	0.21484	1.00000	0.96094
128	0.23047	1.00000	0.22266	1.00000	0.36328

Appendix

Table 15: State tracking accuracy (varying number of updates

n

$m$	Transformer	Linear RNN (neg. eig.)	Linear RNN (pos. eig.)	Hybrid (neg. eig.)	Hybrid (pos. eig.)
4	1.00000	1.00000	1.00000	1.00000	1.00000
8	1.00000	1.00000	1.00000	1.00000	1.00000
16	1.00000	1.00000	1.00000	1.00000	1.00000
32	1.00000	1.00000	1.00000	1.00000	1.00000
64	1.00000	0.83203	0.80078	1.00000	1.00000
128	0.96484	0.67578	0.67188	1.00000	1.00000

Appendix

Table 16: Recall accuracy (varying bit-array size

m

$n$	Transformer	Linear RNN (neg. eig.)	Linear RNN (pos. eig.)	Hybrid (neg. eig.)	Hybrid (pos. eig.)
4	0.82031	1.00000	0.76953	1.00000	0.85938
8	0.75000	1.00000	0.99219	1.00000	0.99219
16	0.73828	1.00000	0.85547	1.00000	0.93750
32	0.73828	1.00000	0.64453	1.00000	0.68750
64	0.62891	0.78125	0.59766	1.00000	0.61328
128	0.54297	0.63672	0.57422	1.00000	0.57031

Appendix

Table 17: State-based recall accuracy (varying number of swaps

n

, with

m=n

Architecture	$E$	$A$	$\alpha$	$B$	$\beta$	$a_{\text{opt}}$	$b_{\text{opt}}$	$R^{2}$
Free fit (all five parameters fit independently)
Olmo 3	1.65 [1.11, 1.87]	108.83 [21.2, 448.7]	0.25 [0.14, 0.34]	83.45 [29.6, 495.0]	0.21 [0.15, 0.30]	0.46 [0.33, 0.68]	0.54 [0.32, 0.67]	0.9976
Hybrid	1.60 [1.25, 1.72]	71.14 [24.8, 145.3]	0.23 [0.15, 0.27]	81.72 [37.4, 219.6]	0.22 [0.18, 0.27]	0.49 [0.41, 0.64]	0.51 [0.36, 0.59]	0.9993
Pure GDN	1.46 [1.16, 1.75]	36.24 [19.0, 113.3]	0.18 [0.14, 0.26]	102.72 [52.1, 214.8]	0.23 [0.19, 0.26]	0.55 [0.45, 0.65]	0.45 [0.35, 0.55]	0.9994
Fixed-exponent fit ( $\alpha=\beta=0.22$ shared; only $E$ , $A$ , $B$ refit)
Olmo 3	1.55 [1.52, 1.58]	66.63 [61.9, 69.4]	0.22	94.85 [89.3, 101.4]	0.22	0.50	0.50	0.9975

Appendix

Table 18: Fit Chinchilla scaling law parameters for

L(N,D)=E+A/N^{\alpha}+B/D^{\beta}

. 95% bootstrap CI shown below each estimate. Top : unconstrained fit where all five parameters are fit independently per architecture. Bottom : fixed-exponent fit where

\alpha=\beta=0.22

(in gray cells) (approximately the mean of the unconstrained estimates) are shared across architectures; only

E

A

B

are refit. Equal exponents imply

a_{\text{opt}}=b_{\text{opt}}=0.5

	Transformer			Hybrid GDN			Pure GDN
FLOPs	$N^{*}$	$D^{*}$	Loss	$N^{*}$	$D^{*}$	Loss	$N^{*}$	$D^{*}$	Loss
$10^{18}$	0.2	0.8	3.57 [3.52, 3.62]	0.2	0.7	3.46 [3.42, 3.49] [ $-$ 0.19, $-$ 0.05]	0.2	1.0	3.53 [3.50, 3.55] [ $-$ 0.10, $+$ 0.01]
$10^{19}$	0.6	2.9	3.12 [3.10, 3.14]	0.7	2.3	3.04 [3.02, 3.05] [ $-$ 0.11, $-$ 0.05]	0.6	2.9	3.10 [3.09, 3.10] [ $-$ 0.05, $+$ 0.00]
$10^{20}$	1.6	10.2	2.78 [2.75, 2.79]	2.2	7.4	2.71 [2.69, 2.72] [ $-$ 0.10, $-$ 0.04]	2.0	8.2	2.76 [2.74, 2.77] [ $-$ 0.04, $+$ 0.01]
$10^{21}$	4.7	35.5	2.51 [2.45, 2.55]	7.0	24.0	2.46 [2.41, 2.48] [ $-$ 0.11, $+$ 0.01]	7.3	22.9	2.49 [2.46, 2.52] [ $-$ 0.07, $+$ 0.04]
$10^{22}$	13.5	123.6	2.31 [2.21, 2.36]	21.6	77.2	2.27 [2.18, 2.30] [ $-$ 0.14, $+$ 0.06]	26.0	64.1	2.27 [2.23, 2.33] [ $-$ 0.10, $+$ 0.08]

Appendix

Table 19: Compute-optimal model size (

N^{*}

, billions), dataset size (

D^{*}

, billions of tokens), and predicted loss for all three architectures. Each architecture’s fitted

a_{\text{opt}},b_{\text{opt}}

determine its compute-optimal allocation (

C=6ND

). For each loss entry: [loss CI] on the second line; [

\Delta

CI] on the third line (loss reduction relative to the transformer). All CIs are 95% bootstrap intervals.

	Transformer	Hybrid		Pure GDN
$N$	$D$	$D$	Savings	$D$	Savings
1B	776.5B	587.1B	1.32 $\times$ [0.67, 4.02]	949.6B	0.82 $\times$ [0.45, 2.50]
3B	90.0B	57.2B	1.57 $\times$ [1.11, 2.50]	79.5B	1.13 $\times$ [0.77, 1.63]
7B	35.1B	20.9B	1.68 $\times$ [0.91, 3.47]	27.0B	1.30 $\times$ [0.68, 2.30]
13B	21.5B	12.3B	1.75 $\times$ [0.84, 4.12]	15.2B	1.41 $\times$ [0.65, 2.75]
30B	13.1B	7.2B	1.82 $\times$ [0.77, 4.98]	8.4B	1.56 $\times$ [0.61, 3.44]

Appendix

Table 20: Projected token requirements across model scales (target loss

=2.474

, selected as the minimum of all observed training losses). Savings

>1\times

means fewer tokens needed than the transformer. 95% bootstrap CI shown below each estimate. See also Figure 10 (b).

Figure 14: Scaling law fit diagnostics. (a) Residuals are small and unbiased for all architectures. (b) Mean absolute relative error stays below 1% across all model sizes, confirming reliable fits. Overall: Olmo 3

R^{2}=0.9976

; Hybrid

R^{2}=0.9993

; Pure GDN

R^{2}=0.9994

Figure 15: Extended downstream eval training curves comparing Olmo Hybrid 7B and Olmo 3 7B across 6 benchmarks, sorted by token efficiency (see Figure 1 for CE loss and MMLU training curves). Olmo Hybrid reaches Olmo 3’s final performance using 19–58% fewer tokens depending on the benchmark.

Figure 16: Evaluation metrics vs. training FLOPs. (a) Average validation cross-entropy loss across 11 held-out corpora (C4, Dolma Books/CC/pes2o/Reddit/Stack/Wiki, ICE, M2D2 S2ORC, Pile, Wikitext-103; lower is better); this is the same loss used to fit the Chinchilla scaling laws. (b) Base Easy Suite average BPB (lower is better). (c)–(e) Per-cluster Base Easy BPB for Math, Code, and QA respectively. The thick line is the fitted compute-optimal frontier.

Figure 17: Evaluation metrics vs. training FLOPs for pure architectures (Transformer, pure GDN, pure Mamba2). Panels follow the same layout as Figure 16 : (a) average validation cross-entropy loss, (b) Base Easy Suite average BPB, and (c)–(e) per-cluster BPB for Math, Code, and QA. The thick line is the fitted compute-optimal frontier.

Figure 18: Evaluation metrics vs. training FLOPs for hybrid architectures (GDN hybrid with interleaved and middle placement, Mamba2 hybrid). Panels follow the same layout as Figure 16 : (a) average validation cross-entropy loss, (b) Base Easy Suite average BPB, and (c)–(e) per-cluster BPB for Math, Code, and QA. The thick line is the fitted compute-optimal frontier.

Figure 19: Evaluation metrics vs. training FLOPs for GDN hybrid models with varying linear-to-attention ratios (1:1, 3:1, 7:1). Panels follow the same layout as Figure 16 : (a) average validation cross-entropy loss, (b) Base Easy Suite average BPB, and (c)–(e) per-cluster BPB for Math, Code, and QA. The thick line is the fitted compute-optimal frontier.

Figure 20: Evaluation metrics vs. training FLOPs for GDN ablations (e.g., GDN with and without negative eigenvalues). Panels follow the same layout as Figure 16 : (a) average validation cross-entropy loss, (b) Base Easy Suite average BPB, and (c)–(e) per-cluster BPB for Math, Code, and QA. The thick line is the fitted compute-optimal frontier.

Figure 21: Evaluation metrics vs. training FLOPs for GDN hybrid ablations (e.g., hybrid GDN with and without negative eigenvalues). Panels follow the same layout as Figure 16 : (a) average validation cross-entropy loss, (b) Base Easy Suite average BPB, and (c)–(e) per-cluster BPB for Math, Code, and QA. The thick line is the fitted compute-optimal frontier.

Architecture	Attn %	60M	100M	190M	370M	600M	760M	1B
$d$		384	512	768	1024	1280	1536	2048
$h$		8	8	12	16	16	16	16
$l$		8	12	12	16	16	16	16
Pure Architectures
Transformer	100%	57	102	190	371	548	758	1279
GDN	0%	78	140	276	574	780	1011	1549

Appendix

Table 22: Architecture configurations and non-embedding parameter counts (millions).

d

: model dimension,

h

: number of attention heads,

l

: number of layers. Architectures at the same nominal scale can differ substantially in parameter count due to differences in layer composition.

Explore similar work

Jun 18, 2026

Comparing Transformers and Hybrid Models at the Token Level

Hybrid language models that mix attention and recurrent layers have shown promise: theoretically, recurrent layers ameliorate the limitations of pure transformers on state tracking, and empirically, hybrids can outperform pure transformers in loss and downstream evaluations \citep{waleffe2024empirical,merrill2026olmohybrid}. Yet it remains unclear which data or capabilities drive these gains, and to what degree they reflect the theoretical advantages motivating hybrid models. We address this question using the open weights from Olmo 3 \citep{olmo2025olmo3} and Olmo Hybrid \citep{merrill2026olmohybrid}: we compare the loss of a matched transformer and hybrid at the same target tokens under the same prefixes, stratifying the results by natural token tags, copy features, delimiter structure, and controlled synthetic probes. The hybrid has lower loss on most tag families, but the gains are not uniform: they are largest for open-class content words and smaller for many closed-class function words. Across prose, code, and markup, the hybrid's loss advantage is larger on opening delimiters than on the corresponding closing delimiters, and nearly vanishes on repeated

n

-grams. Synthetic probes show the same split: the hybrid is favored on pronoun-memory and entity-tracking tasks, whereas the transformer is favored on bracket-matching tasks that require choosing closing delimiters. These patterns suggest that the recurrent layers in hybrids improve predictions that leverage the semantic state of a document, whereas attention helps on tokens predictable by

n

-gram copying or syntactic bracket matching. We conclude with proof-of-concept filtered evaluations showing how token-level decompositions can sharpen pretraining diagnostics for hybrid architectures.

Yanhong Li, William Merrill

Allen Institute for AI

LLM Evaluation Transformer

Ask PDF

Mar 9, 2026

Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models

Hybrid sequence models--combining Transformer and state-space model layers--seek to gain the expressive versatility of attention as well as the computational efficiency of state-space model layers. Despite burgeoning interest in hybrid models, we lack a basic understanding of the settings where--and underlying mechanisms through which--they offer benefits over their constituent models. In this paper, we study this question, focusing on a broad family of core synthetic tasks. For this family of tasks, we prove the existence of fundamental limitations for non-hybrid models. Specifically, any Transformer or state-space model that solves the underlying task requires either a large number of parameters or a large working memory. On the other hand, for two prototypical tasks within this family--namely selective copying and associative recall--we construct hybrid models of small size and working memory that provably solve these tasks, thus achieving the best of both worlds. Our experimental evaluation empirically validates our theoretical findings. Importantly, going beyond the settings in our theoretical analysis, we empirically show that learned--rather than constructed--hybrids outperform non-hybrid models with up to 6x as many parameters. We additionally demonstrate that hybrid models exhibit stronger length generalization and out-of-distribution robustness than non-hybrids.

John Cooper, Ilias Diakonikolas, Mingchen Ma +1

Department of Computer Sciences, University of Wisconsin-Madison

Sequence Modeling Transformer Expressivity

Ask PDF

Mar 15, 2026

M $^2$ RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling

Transformers are highly parallel but are limited to computations in the TC

^0

complexity class, excluding tasks such as entity tracking and code execution that provably require greater expressive power. Motivated by this limitation, we revisit non-linear Recurrent Neural Networks (RNNs) for language modeling and introduce Matrix-to-Matrix RNN (M

^2

RNN): an architecture with matrix-valued hidden states and expressive non-linear state transitions. We demonstrate that the language modeling performance of non-linear RNNs is limited by their state size, and show how the state size expansion mechanism enables efficient use of tensor cores. Empirically, M

^2

RNN achieves perfect state tracking generalization at sequence lengths not seen during training. These benefits also translate to large-scale language modeling. In hybrid settings that interleave recurrent layers with attention, Hybrid M

^2

RNN outperforms equivalent Gated DeltaNet hybrids by

0.4

0.5

perplexity points on a 7B MoE model, while using

3\times

smaller state sizes for the recurrent layers. Notably, replacing even a single recurrent layer with M

^2

RNN in an existing hybrid architecture yields accuracy gains comparable to Hybrid M

^2

RNN with minimal impact on training throughput. Further, the Hybrid Gated DeltaNet models with a single M

^2

RNN layer also achieve superior long-context generalization, outperforming state-of-the-art hybrid linear attention architectures by up to

8

points on LongBench. Together, these results establish non-linear RNN layers as a compelling building block for efficient and scalable language models.

Mayank Mishra, Shawn Tan, Ion Stoica +2

UC Berkeley · Learning Machine · MIT-IBM Watson Lab +2

Recurrent Neural Networks Language Modeling Autoregressive Language Modeling Long-Context Language Modeling

Ask PDF

Abstract

Explore similar work

Comparing Transformers and Hybrid Models at the Token Level

Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models

M2^22RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling

M $^2$ RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling