Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves
Authors: Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting
Organizations: Aleph Alpha Research, Heidelberg, Germany · Lab1141, Germany · Work started at Technical University of Munich, Germany · Research Group Social Computing, Technical University of Munich, Germany · Computer Science Department, TU Darmstadt, Germany · Hessian.AI, Germany
Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes depth-recurrence due to partially fixed layer stacks, and ignores the bottleneck of constant hidden-sizes that restricts many-step latent reasoning. To address this, we introduce a modular framework of depth-recurrent attention mixtures (Dreamer), combining sequence attention, depth attention, and sparse expert attention. It alleviates the hidden-size bottleneck through attention along depth, decouples scaling dimensions, and allows depth-recurrent models to scale efficiently and effectively. Across language reasoning benchmarks, our models require 2 to 8x fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched SOTA, and outperform ca. 2x larger SOTA models with the same training tokens. We further present insights into knowledge usage across depths, e.g., showing 2 to 11x larger expert selection diversity than SOTA MoEs.
Figures & tables
Figure 1: D epth- re current a ttention m ixtu re ( Dreamer ). This instance combines attention over sequence (horizontal), depth (vertical), and sparse experts (right) in a single recurring layer, to facilitate knowledge reuse, alleviate hidden- and layer-size bottlenecks, improve reasoning and data efficiency, and decouple scaling dimensions (params, FLOPs, depth, and depth-wise memory).
Figure 2: Dreamer, as high-level folded view, combining depth recurrence (DR) with depth attention (DA), sequence attention (SA), and expert attention (EA).
Parameters (B)
FLOPs (B/token)
Memory (GB)
Depth ▹
8
16
24
32
8
16
24
32
8
16
24
32
Transformer
0.454
0.908
1.362
1.816
0.601
0.939
1.277
1.615
1.511
2.452
3.394
4.336
Dreamer − DA
0.455
0.908
1.362
1.817
0.600
0.939
1.279
1.620
1.512
2.460
3.411
4.379
Dreamer
0.454
0.908
1.363
1.816
0.601
0.941
1.273
1.613
1.511
2.461
3.414
4.380
Table 1: Model resource requirements. FLOPs per token and peak memory are measured over 1024 generated tokens. Parameters exclude vocab. As intended, our matching closely aligns resources.
Figure 3: Inference FLOPs per token and peak memory (log-scaled) for Dreamer and Transformer (depth 16) over context lengths, decomposed into Dreamer’s per-component contributions (SA, DA, EA). Resource requirements for Dreamer and Transformer stay tightly matched across context lengths, with SA dominating. DA memory usage (18 MB) stays constant due to per-token kv-cache eviction.
Model
Depth
Avg
MathQA
GSM8K
MATH
MMLU
Maths-
(0-shot)
(0-shot)
(0-shot)
(math, 0-shot)
College
Acc/DE ↑
Acc/DE ↑
Acc/DE ↑
Acc/DE ↑
Acc/DE ↑
PPL ↓
Transformer
8
26.8 / 1.0
19.9 / 1.0
30.4 / 1.0
34.3 / 1.0
22.7 / 1.0
7.47
Dreamer − DA
8
34.6 / 2.3
29.9 / 3.6
35.3 / 1.8
37.5 / 1.5
35.5 / 2.8
7.32
Dreamer
8
35.6 / 2.8
32.6 / 3.3
38.4 / 2.5
38.7 / 2.3
34.0 / 3.4
7.36
Transformer
16
32.3 / 1.0
22.9 / 1.0
43.8 / 1.0
40.9 / 1.0
26.6 / 1.0
6.74
Table 2: Natural language reasoning & math accuracies (Acc), validation perplexity (PPL), and data efficiency (DE). DE is the saving factor of training tokens to reach Transformer’s best accuracy. All models are pre-trained on ∼ 100B tokens. Models of same depth are tightly FLOP-, parameter-, and memory-matched. Dreamer consistently outperforms modern Transformers, even without DA.
Model
Avg
MathQA
GSM8K
MATH
MMLU
(depth 16)
(0-shot)
(0-shot)
(0-shot)
(math, 0-shot)
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Transformer
32.3
22.9
43.8
40.9
26.6
Dreamer − DA − EA
33.2
24.4
45.5
47.4
35.3
Dreamer − DA − DPE
33.5
31.7
40.9
43.4
29.4
Dreamer − DA − AMoE
34.1
26.4
45.0
45.0
34.5
Table 3: Dreamer ablations. “ − revDPE”: no half-reversed depth-position encoding, instead standard RoPE. “ − AMoE”: no linear MoE for attention projections, instead standard attention projections. “ − DPE”: no depth-position for EA, instead pure low-rank EA. “ − EA”: no EA, instead modern MoE.
Model
Avg
HellaSwag
MMLU
PIQA
TriviaQA
(depth 32)
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Transformer
34.5
38.1
29.3
69.0
1.6
Dreamer − DA
36.2
39.6
30.6
69.5
5.1
Dreamer
37.4
40.6
30.6
71.4
7.0
Table 4: Broad knowledge and language modeling benchmark results. We compare Dreamer to modern Transformer architectures. All models are pre-trained on ca. 100B tokens, have depth 32, and are tightly FLOP-, parameter-, and memory-matched.
Figure 4: Map of average depth attention scores, normalized per depth.
Figure 5: Distribution of depths per expert. Experts are sorted by the number of depths in their top 90th percentile of sorted P(depth∣expert) , as a measure of depth-generalization. Within these groups, experts are sorted by sampled depth from their distributions. Observations: Higher depths use more depth-specialized experts. Ca. 50% of experts are widely depth-generalized.
Figure 6: Lorenz curves (log-scaled) for EA expert routing. This shows how many expert routings occurred vs. how many unique experts were used, for Dreamer per depth (colored), Dreamer globally (thin dotted), and normal MoEs per depth (= globally) (thick dashed). Observations: Dreamer uses 2–11 × more experts per depth than normal MoEs. Global balance remains strong (Gini: 0.075).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Transformer
Dreamer − DA
Dreamer
Depth
8, 16, 24, 32
8, 16, 24, 32
8, 16, 24, 32
layers
= depth
1
1
DR iterations
1
= depth
= depth
EA expert intermediate size
512
504
480, 480, 472, 472
EA experts
32
257, 517, 778, 1039
266, 537, 822, 1097
EA active experts
8
Appendix
Table 5: Hyperparameters.
Dataset
Paper
License
EleutherAI/hendrycks_math
[ 38 ]
MIT
bethgelab/CuratedThoughts
[ 39 ]
-
(OpenR1-Math-220k-default,
OpenThoughts-114k-math-default)
microsoft/orca-math-word-problems-200k
[ 40 ]
MIT
TIGER-Lab/MathInstruct
[ 41 ]
MIT
Appendix
Table 6: Natural language reasoning & math training datasets.
Benchmark
Paper
License
MathQA
[ 50 ]
Apache 2.0
GSM8K
[ 42 ]
MIT
MATH
[ 38 ]
MIT
MMLU (math:
[ 51 ]
MIT
abstract_algebra,
elementary_mathematics,
Appendix
Table 7: Natural language reasoning & math test benchmarks.
Dataset
Paper
License
nvidia/Nemotron-CC-v2
[ 52 ]
NVIDIA Data Agreement for Model Training
(High-Quality-Synthetic)
Appendix
Table 8: Knowledge & natural language training datasets.
Benchmark
Paper
License
HellaSwag
[ 53 ]
MIT
MMLU
[ 51 ]
MIT
PIQA
[ 54 ]
-
TriviaQA
[ 55 ]
-
Appendix
Table 9: Knowledge & natural language test benchmarks.
Context Length
Transformer
Dreamer
SA
DA
EA
1,024
0.939
0.941
0.273
0.021
0.384
16,384
1.951
1.954
1.286
0.021
0.384
131,072
9.512
9.514
8.846
0.021
0.384
1,048,576
69.993
69.996
69.328
0.021
0.384
Appendix
Table 10: Inference FLOPs per token (in billions) for Dreamer and Transformer (depth 16) at varying context lengths, decomposed into Dreamer’s components (SA, DA, EA).
Context Length
Transformer
Dreamer
SA
DA
EA
1,024
2.452
2.461
0.281
0.018
1.593
16,384
3.585
3.585
1.405
0.018
1.584
131,072
12.040
12.041
9.861
0.018
1.584
1,048,576
79.688
79.689
77.507
0.018
1.584
Appendix
Table 11: Inference peak memory (in GB) for Dreamer and Transformer (depth 16) at varying context lengths, decomposed into Dreamer’s components (SA, DA, EA). Peak memory includes parameters, activations, and kv-cache.
Figure 7: Intermediate reasoning benchmark results (0-shot) during training, with FLOP-/parameter-/memory-matched architectures per depth. Dreamer consistently outperforms modern Transformers.
Figure 8: Distribution of experts per depth. Experts are sorted like in Figure 5 . Observations: Higher depths tend to use more depth-specialized experts. However, there are some exceptions, like the second to last depth, which also use some specific depth-generalized experts.
Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.
Yuxiang Wang, Kunyu Feng, Yingda Shen +3
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2
Scaling depth is a key driver for large language models (LLMs). Yet, as LLMs become deeper, they often suffer from signal degradation: informative features formed in shallow layers are gradually diluted by repeated residual updates, making them harder to recover in deeper layers. We introduce mixture-of-depths attention (MoDA), a mechanism that allows each attention head to attend to sequence KV pairs at the current layer and depth KV pairs from preceding layers. We further describe a hardware-efficient algorithm for MoDA that resolves non-contiguous memory-access patterns, achieving 97.3% of FlashAttention-2's efficiency at a sequence length of 64K. Experiments on 1.5B-parameter models demonstrate that MoDA consistently outperforms strong baselines. Notably, it improves average perplexity by 0.2 across 10 validation benchmarks and increases average performance by 2.11% on 10 downstream tasks, with a negligible 3.7% FLOPs computational overhead. We also find that combining MoDA with post-norm yields better performance than using it with pre-norm. These results suggest that MoDA is a promising primitive for depth scaling. Code is released at https://github.com/hustvl/MoDA .
Lianghui Zhu, Yuxin Fang, Bencheng Liao +10
School of EIC, Huazhong University of Science & Technology · ByteDance Seed
Large Language Models can perform multi-step reasoning and improve task performance through different forms of intermediate computation, from token-based traces to computation carried out in latent space. However, a question remains open: do these different forms of thinking rely on the same underlying mechanism? To address this, we train and compare five variants of the same GPTNeoX backbone from scratch on an extended multi-hop reasoning task (ProsQA-Ext): a vanilla model, a Chain-of-Thought (CoT) model, a Pause Token model, and two latent-reasoning models that are optimized end-to-end without intermediate reasoning traces. We find that, strong in-distribution (ID) performance does not guarantee depth generalization. Vanilla, CoT, and Pause Token models solve ID problems well, but rely largely on local graph features and generalize poorly to out-of-distribution (OOD) problems with longer hops. In contrast, latent variants generalize better and show internal dynamics consistent with forward reachability propagation on the graph. Causal interventions and circuit analysis localize this computation to a sparse recurrent search circuit in the bottleneck latent model: an attention head retrieves graph relations, an MLP and the residual stream update the reachability state across recurrent steps, while multiple attention heads together then do the candidate matching. Together, these results show that different thinking mechanisms can learn distinct computational solutions, even at similar ID performance. In this setting, latent recurrence supports a reusable forward-search algorithm that generalizes beyond the training depth.