Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves
Authors: Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting
Organizations: Aleph Alpha Research, Heidelberg, Germany · Lab1141, Germany · Work started at Technical University of Munich, Germany · Research Group Social Computing, Technical University of Munich, Germany · Computer Science Department, TU Darmstadt, Germany · Hessian.AI, Germany
Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes depth-recurrence due to partially fixed layer stacks, and ignores the bottleneck of constant hidden-sizes that restricts many-step latent reasoning. To address this, we introduce a modular framework of depth-recurrent attention mixtures (Dreamer), combining sequence attention, depth attention, and sparse expert attention. It alleviates the hidden-size bottleneck through attention along depth, decouples scaling dimensions, and allows depth-recurrent models to scale efficiently and effectively. Across language reasoning benchmarks, our models require 2 to 8x fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched SOTA, and outperform ca. 2x larger SOTA models with the same training tokens. We further present insights into knowledge usage across depths, e.g., showing 2 to 11x larger expert selection diversity than SOTA MoEs.
Figures & tables
Figure 1: D epth- re current a ttention m ixtu re ( Dreamer ). This instance combines attention over sequence (horizontal), depth (vertical), and sparse experts (right) in a single recurring layer, to facilitate knowledge reuse, alleviate hidden- and layer-size bottlenecks, improve reasoning and data efficiency, and decouple scaling dimensions (params, FLOPs, depth, and depth-wise memory).
Figure 2: Dreamer, as high-level folded view, combining depth recurrence (DR) with depth attention (DA), sequence attention (SA), and expert attention (EA).
Parameters (B)
FLOPs (B/token)
Memory (GB)
Depth ▹
8
16
24
32
8
16
24
32
8
16
24
32
Transformer
0.454
0.908
1.362
1.816
0.601
0.939
1.277
1.615
1.511
2.452
3.394
4.336
Dreamer − DA
0.455
0.908
1.362
1.817
0.600
0.939
1.279
1.620
1.512
2.460
3.411
4.379
Dreamer
0.454
0.908
1.363
1.816
0.601
0.941
1.273
1.613
1.511
2.461
3.414
4.380
Table 1: Model resource requirements. FLOPs per token and peak memory are measured over 1024 generated tokens. Parameters exclude vocab. As intended, our matching closely aligns resources.
Figure 3: Inference FLOPs per token and peak memory (log-scaled) for Dreamer and Transformer (depth 16) over context lengths, decomposed into Dreamer’s per-component contributions (SA, DA, EA). Resource requirements for Dreamer and Transformer stay tightly matched across context lengths, with SA dominating. DA memory usage (18 MB) stays constant due to per-token kv-cache eviction.
Model
Depth
Avg
MathQA
GSM8K
MATH
MMLU
Maths-
(0-shot)
(0-shot)
(0-shot)
(math, 0-shot)
College
Acc/DE ↑
Acc/DE ↑
Acc/DE ↑
Acc/DE ↑
Acc/DE ↑
PPL ↓
Transformer
8
26.8 / 1.0
19.9 / 1.0
30.4 / 1.0
34.3 / 1.0
22.7 / 1.0
7.47
Dreamer − DA
8
34.6 / 2.3
29.9 / 3.6
35.3 / 1.8
37.5 / 1.5
35.5 / 2.8
7.32
Dreamer
8
35.6 / 2.8
32.6 / 3.3
38.4 / 2.5
38.7 / 2.3
34.0 / 3.4
7.36
Transformer
16
32.3 / 1.0
22.9 / 1.0
43.8 / 1.0
40.9 / 1.0
26.6 / 1.0
6.74
Table 2: Natural language reasoning & math accuracies (Acc), validation perplexity (PPL), and data efficiency (DE). DE is the saving factor of training tokens to reach Transformer’s best accuracy. All models are pre-trained on ∼ 100B tokens. Models of same depth are tightly FLOP-, parameter-, and memory-matched. Dreamer consistently outperforms modern Transformers, even without DA.
Model
Avg
MathQA
GSM8K
MATH
MMLU
(depth 16)
(0-shot)
(0-shot)
(0-shot)
(math, 0-shot)
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Transformer
32.3
22.9
43.8
40.9
26.6
Dreamer − DA − EA
33.2
24.4
45.5
47.4
35.3
Dreamer − DA − DPE
33.5
31.7
40.9
43.4
29.4
Dreamer − DA − AMoE
34.1
26.4
45.0
45.0
34.5
Table 3: Dreamer ablations. “ − revDPE”: no half-reversed depth-position encoding, instead standard RoPE. “ − AMoE”: no linear MoE for attention projections, instead standard attention projections. “ − DPE”: no depth-position for EA, instead pure low-rank EA. “ − EA”: no EA, instead modern MoE.
Model
Avg
HellaSwag
MMLU
PIQA
TriviaQA
(depth 32)
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Transformer
34.5
38.1
29.3
69.0
1.6
Dreamer − DA
36.2
39.6
30.6
69.5
5.1
Dreamer
37.4
40.6
30.6
71.4
7.0
Table 4: Broad knowledge and language modeling benchmark results. We compare Dreamer to modern Transformer architectures. All models are pre-trained on ca. 100B tokens, have depth 32, and are tightly FLOP-, parameter-, and memory-matched.
Figure 4: Map of average depth attention scores, normalized per depth.
Figure 5: Distribution of depths per expert. Experts are sorted by the number of depths in their top 90th percentile of sorted P(depth∣expert) , as a measure of depth-generalization. Within these groups, experts are sorted by sampled depth from their distributions. Observations: Higher depths use more depth-specialized experts. Ca. 50% of experts are widely depth-generalized.
Figure 6: Lorenz curves (log-scaled) for EA expert routing. This shows how many expert routings occurred vs. how many unique experts were used, for Dreamer per depth (colored), Dreamer globally (thin dotted), and normal MoEs per depth (= globally) (thick dashed). Observations: Dreamer uses 2–11 × more experts per depth than normal MoEs. Global balance remains strong (Gini: 0.075).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Transformer
Dreamer − DA
Dreamer
Depth
8, 16, 24, 32
8, 16, 24, 32
8, 16, 24, 32
layers
= depth
1
1
DR iterations
1
= depth
= depth
EA expert intermediate size
512
504
480, 480, 472, 472
EA experts
32
257, 517, 778, 1039
266, 537, 822, 1097
EA active experts
8
Appendix
Table 5: Hyperparameters.
Dataset
Paper
License
EleutherAI/hendrycks_math
[ 38 ]
MIT
bethgelab/CuratedThoughts
[ 39 ]
-
(OpenR1-Math-220k-default,
OpenThoughts-114k-math-default)
microsoft/orca-math-word-problems-200k
[ 40 ]
MIT
TIGER-Lab/MathInstruct
[ 41 ]
MIT
Appendix
Table 6: Natural language reasoning & math training datasets.
Benchmark
Paper
License
MathQA
[ 50 ]
Apache 2.0
GSM8K
[ 42 ]
MIT
MATH
[ 38 ]
MIT
MMLU (math:
[ 51 ]
MIT
abstract_algebra,
elementary_mathematics,
Appendix
Table 7: Natural language reasoning & math test benchmarks.
Dataset
Paper
License
nvidia/Nemotron-CC-v2
[ 52 ]
NVIDIA Data Agreement for Model Training
(High-Quality-Synthetic)
Appendix
Table 8: Knowledge & natural language training datasets.
Benchmark
Paper
License
HellaSwag
[ 53 ]
MIT
MMLU
[ 51 ]
MIT
PIQA
[ 54 ]
-
TriviaQA
[ 55 ]
-
Appendix
Table 9: Knowledge & natural language test benchmarks.
Context Length
Transformer
Dreamer
SA
DA
EA
1,024
0.939
0.941
0.273
0.021
0.384
16,384
1.951
1.954
1.286
0.021
0.384
131,072
9.512
9.514
8.846
0.021
0.384
1,048,576
69.993
69.996
69.328
0.021
0.384
Appendix
Table 10: Inference FLOPs per token (in billions) for Dreamer and Transformer (depth 16) at varying context lengths, decomposed into Dreamer’s components (SA, DA, EA).
Context Length
Transformer
Dreamer
SA
DA
EA
1,024
2.452
2.461
0.281
0.018
1.593
16,384
3.585
3.585
1.405
0.018
1.584
131,072
12.040
12.041
9.861
0.018
1.584
1,048,576
79.688
79.689
77.507
0.018
1.584
Appendix
Table 11: Inference peak memory (in GB) for Dreamer and Transformer (depth 16) at varying context lengths, decomposed into Dreamer’s components (SA, DA, EA). Peak memory includes parameters, activations, and kv-cache.
Figure 7: Intermediate reasoning benchmark results (0-shot) during training, with FLOP-/parameter-/memory-matched architectures per depth. Dreamer consistently outperforms modern Transformers.
Figure 8: Distribution of experts per depth. Experts are sorted like in Figure 5 . Observations: Higher depths tend to use more depth-specialized experts. However, there are some exceptions, like the second to last depth, which also use some specific depth-generalized experts.