Language models typically increase per-token computation through deeper or looped layers and latent reasoning steps, while richer residual connections improve information flow across layers. We instead increase computational capacity: the number of active computational states a model maintains. We introduce the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently along time, alongside the vertical residual stream that flows through depth. Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than adding or rerouting residual pathways through depth, it provides companion computational states that persist across tokens. Unlike hybrid Transformer-SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing, these latents directly condition the vertical stream's attention, KV-cache writes, MLP computation, and decoding. On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer. These gains extend across architectural variants. The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with the same enhancements) to 13.61. On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids, it improves perplexity at every tested native state size, with Lattice-GDN and Lattice-Triadic-GDN variants outperforming pure baselines with 4x larger native states. These results identify persistent per-layer state as a complementary axis for scaling language-model computation.
Figures & tables
Figure 1: Latent Lattice Language Model. Blue arrows denote the vertical latent htℓ flowing through depth; orange arrows denote the horizontal latents rtℓ , one per Lattice layer, flowing through time. The arrow from the top Lattice layer to the bottom layer of the next timestep illustrates the optional top-to-bottom feedback; the other two enhancements (input injection and the depth-wise readout stream) are omitted for clarity. See the project webpage for animations and a comparison with the base Transformer.
Approach
Params [-0.2ex] (M)
GFLOP/token
FineWeb Edu
Pile
LAMBADA
Nemo CC
Train
Infer
OAI
Stand.
400M-parameter models
Base-24L (width dmodel=1024 )
406.7
0.71
0.71
15.410
26.276
41.172
73.648
14.768
LLLM (945,256)
406.5
0.72
0.72
14.545
25.031
38.835
65.780
14.278
LLLM (908,256) inj+vs
405.6
0.72
0.72
14.400
24.100
38.513
63.259
14.116
LLLM-410 J3 (945,256)
406.9
1.85
1.85
14.026
24.107
37.531
62.923
13.694
Table 1: Language-modeling Perplexity ( ↓ ) after 20B-token pretraining. LLLM (dmodel,dr) denotes a vertical (residual-stream) width dmodel and horizontal-latent width dr ; all models are 24 layers unless noted and use a 2048-token context. FineWeb-Edu is in-distribution; the other sets are out-of-distribution. The 400M LLLM (908,256) inj+vs (input injection + depth-wise readout stream) matches or outperforms the 600M Transformers. Results at 30B tokens are in Tab. 3 .
Figure 2: Recurrent backbones: FineWeb-Edu perplexity. Pure and Lattice GDN, Mamba2, Triadic GDN, and GDN–Mamba2 hybrid models ( ∼ 405M parameters, 10B FineWeb-Edu tokens) across multiple native-state sizes per mixer. Each Lattice model is iso-parameter with its pure counterpart and lowers perplexity at every state size. See Sec. 4.3 .
Figure 3: Lattice of Latents vs. prior residual-stream enhancements. On its own, the Lattice lowers the base Transformer’s perplexity more than HC, mHC, or value residuals, and its gains compound with theirs: Lattice + HC + x0 reaches 13.76 , the best overall. 400M models, 20B FineWeb-Edu tokens. See Sec. 4.2 and Appendix Sec. A.3.3 .
Figure 4: How the horizontal latent acts. Transformer instantiation: the recurrent lattice latent r^tℓ conditions the layer’s QKV and MLP computations; when the optional horizontal decoding head is enabled, the topmost horizontal latent also contributes to output decoding (not shown).
Figure 5: Horizontal latent update. On top of the Mamba-style affine update, we further introduce input injection, vstream (not shown) and an optional cross-timestep nonlinear flow. See webpage for animations.
Figure 6: Deconstructing the Lattice. Here we study a challenging regime where the vertical residual stream width is significantly reduced to 256-dim. LLLM ablations show how the proposed horizontal update enhancements (Sec. 3.2.1 and Sec. 3.2.2 ) recover most of the lost performance; trace the red bars from left to right. Eval post 10B-token pretraining. Note: 406M Lattice model beats transformer (with perplexity = 15.63, see Fig. 7 ). See Sec. 4.4 for more details.
Figure 7: LLLM vs. larger-depth and larger-width Transformers. Acting as a companion to the residual stream, the Lattice enables better performance than a deeper Transformer or naively increasing the residual-stream width; even 406 M LLLM (945,256) outperforms both ∼ 600 M alternatives on 10B-token pretraining. See Sec. 4.4 .
Figure 8: Allocating vertical and horizontal latents. Comparing Transformers with LLLMs at different allocations of vertical and horizontal latent widths (dmodel,dr) after 10B-token pretraining. See Sec. 4.4 .
Native state size
Entropy effective rank rent ( ↓ )
Stable rank rstable ( ↓ )
Query alignment gain G1 ( ↑ )
16K
6.250→5.649
1.518→1.353
3.693→5.579
32K
10.077→8.403
1.629→1.440
5.613→7.493
64K
14.498→11.842
1.798→1.555
6.269→8.925
128K
22.965→19.444
2.055→1.766
7.637→10.714
256K
37.915→33.418
2.504→2.128
7.957→13.321
Table 2: Native-memory geometry and query alignment in GDN. Each cell reports pure → Lattice , at matched native-state size and approximately matched parameters. Lower ranks mean the memory is concentrated in fewer directions; higher G1 means queries align more with the leading memory direction. The Lattice improves all three metrics at every state size. Values are averaged over all 22 layers and all heads, 1024 FineWeb-Edu sequences of 2048 tokens, and 15 positions per sequence just after a state update. See Sec. 4.4 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Approach
Params [-0.2ex] (M)
FineWeb-Edu [-0.2ex] PPL ↓
Pile [-0.2ex] PPL ↓
LAMBADA OpenAI
LAMBADA Standard
PPL ↓
Acc. ↑
PPL ↓
Acc. ↑
400M-parameter models
Base-24L (width dv=1024 )
407
14.719
26.518
40.759
34.83
69.346
26.49
LLLM (945,256)
407
14.050
25.081
37.605
37.36
64.314
28.80
LLLM (908,256) re+vs
406
13.921
24.148
37.451
37.73
61.598
29.89
500M-parameter models
Appendix
Table 3: Language-modeling quality after 30B-token FineWeb-Edu pretraining. We report perplexity ( ↓ ) on FineWeb-Edu, Pile, and the OpenAI and Standard LAMBADA variants, together with LAMBADA last-word accuracy ( ↑ ). Models are grouped by parameter scale. The performance trends are similar to the 20b token results in Tab. 1 .
Figure 9: Ablating top-to-bottom residual-stream flow. Feeding each token’s top-layer state back into the bottom of the network at the next token adds a nonlinear path through the residual stream. Jacobi on vertical applies this feedback to the vertical residual stream, analogous to the concurrent Full-Bandwidth Transformer ( Wang et al., 2026 ) ; Jacobi on Lattice applies it to the horizontal latents; the last bar applies both. We call it “Jacobi” because the feedback is trained and evaluated with three parallel, teacher-forced Jacobi sweeps (J3; App. A.2.3 ), at the cost of extra forward passes. Both forms of feedback help, and they compound: 16.94 (Transformer) and 15.52 (single-pass Lattice) fall to 14.92 with feedback on both. FineWeb-Edu val perplexity, trained on 10B tokens.
Approach
Params (M)
LAMBADA
ARC
WG
PIQA
HS
SciQ
RACE
Avg. ↑
OpenAI
Standard
Easy
Challenge
0-shot accuracy
400M-parameter models
Base-24L (width dv=1024 )
407
33.0
24.9
53.7
27.8
50.6
66.8
40.2
81.4
30.7
45.46
LLLM (945,256)
407
35.9
26.9
56.4
29.0
54.2
68.4
42.4
84.4
31.3
47.67
LLLM (908,256) re+vs
406
35.1
28.2
56.0
29.6
51.6
68.0
43.6
82.6
31.7
47.38
Appendix
Table 4: Zero- and five-shot downstream task accuracy at the 20B-token FineWeb-Edu pretraining scale. Models are grouped by parameter scale and shown in the same order under both evaluation settings. All task values are accuracies ( % ; ↑ ). LAMBADA results are reported on the OpenAI and Standard variants, while ARC results are reported on ARC-Easy and ARC-Challenge. WG / HS denote WinoGrande / HellaSwag, respectively. Avg. is reported average across 9 tasks.
Figure 10: Stateless vs. Lattice modulation: is recurrent history needed? We replace the horizontal latent in FiLM modulation with a stateless signal computed from either the token embedding (input-conditioned) or the incoming residual stream (residual-stream-conditioned), at matched ∼ 406M parameters. Lattice-conditioned modulation, which carries history across timesteps, clearly outperforms both stateless variants. FineWeb-Edu val perplexity, trained on 10B tokens.
Approach
GSM [-0.2ex]8K
SVAMP [-0.2ex]1K
Multi [-0.2ex]Arith
ASDiv
GSM [-0.2ex]Hard
400M-parameter models
Base-24L ( dv=1024 )
29.7
28.4
74.0
41.6
6.4
LLLM (945,256)
44.5
42.0
90.8
51.4
10.8
LLLM (908,256) re+vs
43.6
42.6
87.7
52.0
10.0
LLLM J3 (945,256)
44.9
45.2
89.2
51.5
9.7
600M-parameter models
Appendix
Table 5: Finetuned mathematical-reasoning Acc.
Figure 11: Short-Conv vs. Lattice Static and dynamic width-4 short convolutions on q,k,v improve the Transformer; the Lattice matches dynamic convolution with fewer parameters, and the two compose: Lattice + dynamic convolution reaches 14.99 . Parameter counts are shown below the bars. FineWeb-Edu val perplexity, trained on 10B tokens. See App. A.3.1 .
Figure 12: Zero-shot downstream accuracy of recurrent backbones. Nine-task average accuracy (%) of pure and Lattice Mamba2, GDN, and no-SWA GDN–Mamba2 hybrid models ( ∼ 405M parameters, 10B FineWeb-Edu tokens) across native-state sizes. Lattice boosts pure backbone performance on the nine-task average results. See Sec. 4.3 .
Figure 13: Five-shot downstream accuracy of recurrent backbones. Nine-task average accuracy (%) of pure and Lattice Mamba2, GDN, and no-SWA GDN–Mamba2 hybrid models ( ∼ 405M parameters, 10B FineWeb-Edu tokens) across native-state sizes. Lattice boosts pure backbone performance on the nine-task average results. See Sec. 4.3 .
Figure 14: Lattice of Latents vs. prior residual-stream enhancements: learned value residuals. Complementing Fig. 3 , we add learned value residuals (“value”), alone and on top of HC + x0, for Transformers (gray) and LLLMs (red). The Lattice lowers perplexity in every configuration ( 3.0 – 7.0% ). FineWeb-Edu validation perplexity, ∼ 400M models, trained on 20B tokens. See Appendix Sec. A.4.2 for details on the learned value residual.
Hyperparameter
Value
Optimizer
AdamW ( β1=0.9,β2=0.95 )
Weight decay
0.1 (none on biases/backbone norms/ A_log / dt_bias )
Peak LR / floor
1×10−3 / 10% of peak
LR schedule
cosine-with-min-lr, 2% linear warmup
Grad clip
1.0
Global batch / seq. length
512 / 2048 ( ≈1.05 M tokens/step)
Appendix
Table 6: Training hyperparameters shared across all runs unless otherwise noted.
Native state
16K
32K
64K
128K
256K
GDN (H,dk,dv)
16,16,64
16,32,64
16,64,64
8,128,128
4,256,256
dmodel pure / Lattice
1120/998
1086/963
1024/898
1024/899
1024/900
Mamba2 (H,dstate,dhead)
16,16,64
16,32,64
16,64,64
16,128,64
4,256,256
dmodel pure / Lattice
1155/1036
1153/1034
1148/1029
1139/1019
1122/1003
Hybrid dmodel pure / Lattice
1137/1028
1118/1008
1082/971
1079/969
1071/961
Appendix
Table 7: Recurrent-backbone geometries and widths. Native state per mixer and residual width of the pure and input-injection + vstream Lattice models (all ∼ 405M parameters). Hybrid GDN and Mamba2 mixers use the GDN and Mamba2 geometries of the same column. Triadic GDN pairs use joint geometries (H,Edk,dv)=(16,16,64) , (16,32,64) , (16,64,64) , (32,64,32) , (16,128,64) , and (8,256,128) .