Transformer language models are commonly scaled by making their blocks wider or stacking them deeper. In this work, we explore a complementary path: branching, the parallel composition of full Transformer blocks. We introduce the Multi-Branch Transformer (MBT), which processes the input through shared initial layers, executes independently parameterized branches of sequential blocks in parallel, and fuses their outputs before shared final layers. At comparable training compute, we study three MBT variants: Wide-Shallow preserves block width, Narrow-Deep preserves sequential depth, and Balanced interpolates between them. In controlled pretraining experiments at four mixture-of-experts scales (0.8B to 2.1B total parameters), 2-branch Wide-Shallow MBTs consistently outperform sequential Transformers under matched parameter and training-compute budgets. Balanced MBTs improve further and retain comparable quality as the number of branches increases. To improve inference efficiency at long contexts, we introduce Joint-KV, which constructs a shared KV cache from branch representations at each depth. Our final recipe combines Balanced MBTs with Joint-KV, supported by an inference engine that exploits branch grouping and shared-cache reuse. Fitted scaling curves indicate that standard Transformers require up to 1.26x as much training body compute to match our final models' validation loss. These models reduce full-model KV-cache storage by up to 34% and deliver up to 51.4% higher decoding throughput on a single H200 GPU. Ablation studies show that branching yields gains beyond changes in width and depth, enables an effective KV-sharing strategy (Joint-KV), remains robust across fusion functions, and performs best in the middle layers. These findings show that branching offers a practical path for scaling language models toward both better modeling quality and faster inference.
Figures & tables
Figure 2: Comparison of Transformer computation structures. (a) Standard sequential blocks; (b) within-block parallel attention and FFN, as in PaLM [ 14 ] ; (c) wider, shallower stacks; and (d) MBTs with concurrent, independently parameterized branches of full Transformer blocks. Depth denotes the number of sequential blocks along a computation path, excluding fusion. MBTs explore an orthogonal axis of parallelism at the granularity of full Transformer blocks.
Figure 3: Three width–depth allocations for MBTs. Relative to a standard Transformer under matched compute, (1) Wide-Shallow MBT preserves block width and redistributes the middle blocks across branches; (2) Narrow-Deep MBT preserves the depth along each branch using narrower blocks; and (3) Balanced MBT uses intermediate width and depth.
Figure 4: KV-cache sharing and serving of MBTs. Top: KV-cache construction at one branch depth. Independent-KV constructs a separate cache from each branch input. Joint-KV projects the concatenated branch inputs into one shared cache, while retaining branch-local queries. Bottom: naive versus optimized serving for MBTs (Example in figure: 4-branch MBT at scale W3). (1) Branch grouping executes same-depth blocks in one stage. (2) We call the KV cache of one KV projection a KV bank . Independent-KV stores one bank per branch block, and Joint-KV stores one bank per branch depth. (3) All branches read a shared KV bank in one attention call. More details are in Figure 12 in Appendix B.1 .
Shared backbone
Tokens
Selected MBT
Parameters (B / M)
Loss ↓
Scale
L
dmodel
dffn
(B)
Allocation
Standard
MBT
Standard
MBT
W1
12
1024
512
27.845
4/(2+2)/4
0.799 / 139.2
0.801 / 141.3
2.653
2.622
W2
14
1152
576
41.088
5/(2+2)/5
1.134 / 205.4
1.136 / 208.1
2.554
2.525
W3
16
1280
640
57.942
5/(3+3)/5
1.555 / 289.7
1.558 / 293.0
2.471
2.444
W4
18
1408
704
78.841
6/(3+3)/6
2.074 / 394.2
2.077 / 398.2
2.392
2.367
Table 1: Scaling ladder and best Wide–Shallow 2-branch MBTs from Figure 5 . Standard and MBT share the backbone dimensions, physical block count, and training tokens at each scale.
Figure 5: Scaling the parallel block ratio. All four scales exhibit a consistent U-shaped trend: allocating approximately 1/3 of blocks to parallel branches reduces loss by 0.025–0.031 relative to Standard, while increasing this fraction to approximately 79% maintains performance on par with Standard.
Figure 6: Compute scaling and downstream performance. Best Wide-Shallow two-branch MBTs versus Standard across W1–W4. (a) Validation loss versus training body compute, with power-law fits. (b) Zero-shot task scores averaged over W1–W4. MBTs improve compute efficiency and mean accuracy on all nine tasks.
Model
K (# branches)
Allocation
d
m
f
L/D
Body
Loss ↓
Standard
—
16 (serial)
1280
640
640
16/16
1.5032
2.471
Wide–Shallow
2
4/(4+4)/4
1280
640
640
16/12
1.5164
2.445
Wide–Shallow
3
4/(3+3+2)/4
1280
640
640
16/11
1.5230
2.457
Wide–Shallow
4
4/(2+2+2+2)/4
1280
640
640
16/10
1.5296
2.451
Balanced
2
4/(6+6)/4
1024
512
512
20/14
1.5301
2.436
Balanced
3
4/(5+5+5)/4
896
432
448
23/13
1.5160
2.446
Table 2: W3 width–depth allocations at comparable compute. Full configurations appear in Table 9 .
Figure 7: Scaling to more parallel branches. Across W1–W4, increasing from 2 to 4 branches largely preserves performance. Balanced MBTs improve the performance through joint width–depth tuning.
Architecture
Block widths
Loss ↓
KV cache
Model
Allocation
d
m
f
Independent
Joint
KiB/token
Saved
Standard
18 (serial)
1408
704
704
2.392
–
49.50
–
K=2
4/(7+7)/4
1152
592
592
2.363
2.361
37.75
23.74%
K=3
4/(6+6+6)/5
896
496
496
2.366
2.366
35.25
28.79%
K=4
4/(6+6+6+6)/5
768
432
432
2.367
2.364
33.75
31.82%
Table 3: W4 Balanced MBTs with Independent-KV and Joint-KV. Cache capacity and savings are for Joint-KV relative to Standard; full W1–W4 results appear in Table 11 .
Figure 8: Joint-KV preserves strong scaling performance. Joint-KV closely tracks Independent-KV across branch counts and improves on Standard while reducing cache capacity.
Figure 9: Decode speedup over Standard. Decode throughput of Balanced Joint-KV MBTs relative to Standard on one H200 GPU, averaged over scales W1–W4, as a function of the number of concurrent requests N , for 4K-, 16K-, 64K-, and 128K-token inputs that each generate 1K tokens. Shaded bands show one standard deviation across scales.
Family
Allocation
Blocks / depth
Outer or all-layer d/m/f
Core or branch d/m/f
Body compute (PFLOP-days)
Standard
12
12/12
1024/512/512
—
0.3665
Global wide–shallow
9
9/9
1280/640/512
—
0.3721
Local wide-core
3/3/3
9/9
1024/512/512
1664/640/640
0.3636
Wide–Shallow MBT
3/(3+3)/3
12/9
1024/512/512
1024/512/512
0.3706
Narrow–Deep MBT
3/(6+6)/3
18/12
1024/512/512
640/320/416
0.3705
Table 4: Example of the ablation setting at W1 with fixed data and approximately matched body compute.
Figure 10: Ablation studies on branching. Positions show loss differences from Standard; labels give absolute loss. The right column fits the scaling laws. We find branching is the key to performance improvement.
Q/KV heads
Body compute
Validation loss ↓
Scale
Allocation
Outer (ratio)
Independent-KV
Joint-KV
KV/token (KiB)
Independent-KV
Joint-KV
Independent-KV
Joint-KV
Δ
W1
3/(5+5)/3
16/2(8:1)
16/1
16/2
5.5
0.3848
0.3867
2.657
2.646
−0.011
W2
3/(6+6)/3
18/2(9:1)
16/1
16/2
6.0
0.7723
0.7763
2.568
2.557
−0.011
W3
4/(6+6)/4
20/2(10:1)
16/1
16/2
7.0
1.4436
1.4499
2.499
2.487
−0.012
W4
4/(7+7)/4
24/3(8:1)
18/1
18/2
9.5
2.6551
2.6664
2.392
2.380
−0.012
Table 5: Joint-KV versus Independent-KV across W1–W4 at matched KV-cache capacity. Each pair shares block allocations, widths, query heads, and training tokens. Training body compute is in PFLOP-days; full configurations in Table 14 .
Fusion
Fused output z
Loss ↓
Concat–projection [ 35 ]
W[h1;h2]
2.445
Channel competition [ 32 ]
2[g⊙W1h1+(1−g)⊙W2h2]
2.446
Joint MLP [ 33 ]
W[h1;h2]+Pϕ(a1+a2)
2.445
Low-rank bilinear [ 34 ]
W[h1;h2]+P(a1⊙a2)
2.446
Joint SwiGLU [ 36 ]
W[h1;h2]+P[ϕ(a1+a2)⊙(b1+b2)]
2.444
Token softmax [ 15 ]
2(β1W1h1+β2W2h2)
2.448
Table 6: Fusion of normalized branch vectors h1,h2 in W3 4/(4+4)/4 . W=[W1W2] , Ai , Bi , and P are learned weight matrices; ai=AiWihi , bi=BiWihi , and ϕ=SiLU . g is a channelwise sigmoid gate and β is a tokenwise softmax over branches; ⊙ denotes elementwise multiplication. The last row tests parameter-free aggregation.
Figure 11: Branching in the middle works best. At W3, we place 8 shared blocks around two 4-block branches while holding all other settings fixed. Shared processing before and after the branches gives the lowest loss.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
W1
W2
W3
W4
Backbone
Blocks L
12
14
16
18
Residual width d
1,024
1,152
1,280
1,408
MoE latent width m
512
576
640
704
Expert hidden width f
512
576
640
704
Query / KV heads
16 / 8
18 / 9
20 / 10
22 / 11
Appendix
Table 7: Standard backbone and pretraining configuration for W1–W4. MBT variants use the same per-scale data and optimization recipe unless their experiment specifies a change.
W1
W2
W3
W4
Task
Standard
MBT
Standard
MBT
Standard
MBT
Standard
MBT
HellaSwag
45.15
46.44
50.29
52.15
53.30
54.65
57.34
58.63
ARC-E
57.37
60.82
62.46
63.72
63.47
67.17
65.66
67.63
ARC-C
31.06
31.40
33.45
35.15
35.58
38.91
37.46
38.40
OBQA
34.40
36.20
35.20
39.20
37.60
39.20
40.60
41.80
SciQ
75.20
77.00
80.50
82.50
84.50
84.70
81.10
86.10
Appendix
Table 8: Per-scale zero-shot results for the Standard and selected 2-branch MBT checkpoints in Figure 6 . Scores are percentages ( ↑ ); the higher score in each pair is bold.
Scale
MBT family
K
P/(branches)/C
d/m/f
L/D
Body / Total
Loss ↓
W1
Wide–Shallow
2
3/(3+3)/3
1024/512/512
12/9
0.3706 / 0.6245
2.630
W1
Wide–Shallow
3
3/(2+2+2)/3
1024/512/512
12/8
0.3726 / 0.6266
2.633
W1
Wide–Shallow
4
3/(2+2+1+1)/3
1024/512/512
12/8
0.3746 / 0.6286
2.628
W1
Balanced
2
3/(5+5)/3
768/368/384
16/11
0.3717 / 0.6257
2.618
W1
Balanced
3
3/(4+4+4)/3
640/368/368
18/10
0.3745 / 0.6284
2.623
W1
Balanced
4
3/(3+3+3+3)/3
640/368/368
18/9
0.3758 / 0.6297
2.626
Appendix
Table 9: Branch-count configurations for Figure 7 . L/D denotes physical block count / depth; d/m/f gives branch residual / MoE latent / expert FFN widths. Compute is body / total PFLOP-days.
Model / allocation
Branch d/m/f
L/D
Loss ↓
Body / Total
Standard (12 blocks)
1024/512/512
12/12
2.653
0.366514 / 0.620471
Wide–Shallow 3/(3+3)/3
1024/512/512
12/9
2.630
0.370569 / 0.624527
Narrow–Deep 3/(6+6)/3
640/352/368
18/12
2.632
0.369463 / 0.623421
Narrow–Deep 3/(6+6+6)/3
512/272/272
24/12
2.650
0.369458 / 0.623416
Narrow–Deep 3/(6+6+6+6)/3
384/256/256
30/12
2.675
0.370701 / 0.624659
Appendix
Table 10: W1 full-depth Narrow–Deep controls, reporting the best final loss for each configuration. Compute is body / total PFLOP-days; L/D denotes physical block count / depth.
Architecture
Loss ↓
Cache and compute
K
Allocation
Branch d/m/f
Independent
Joint
Joint KV saved
Body / Total
W1 : Standard loss 2.653; Standard KV 24 KiB/token
2
3/(5+5)/3
768/368/384
2.618
2.620
18.75%
0.371744 / 0.625702
3
3/(4+4+4)/3
640/368/368
2.623
2.622
29.17%
0.374484 / 0.628442
4
3/(3+3+3+3)/3
640/368/368
2.626
2.633
34.38%
0.375752 / 0.629709
W2 : Standard loss 2.554; Standard KV 31.5 KiB/token
Appendix
Table 11: Balanced MBTs with Independent-KV and Joint-KV. Losses are final validation losses; Joint-KV cache savings are relative to Standard. Compute is body / total PFLOP-days for both variants.
Scale
Model
HS
ARC-E
ARC-C
OBQA
SciQ
WG
BoolQ
LMDA
PIQA
Average
W1
Standard
45.15
57.37
31.06
34.40
75.20
54.85
56.36
37.42
69.80
51.29
Joint-KV, K=2
47.65
56.40
31.66
36.80
79.60
54.30
60.00
38.29
69.70
52.71
Joint-KV, K=3
47.33
57.74
31.06
34.20
76.90
53.75
56.64
38.91
69.53
51.78
Joint-KV, K=4
47.07
58.63
31.31
35.00
75.80
53.20
58.17
36.29
68.55
51.56
W2
Standard
50.29
62.46
33.45
35.20
80.50
53.28
47.22
38.95
71.00
52.48
Joint-KV, K=2
52.29
61.11
35.67
37.00
82.70
53.51
58.04
40.03
70.67
54.56
Appendix
Table 12: Zero-shot downstream performance of Balanced Joint-KV MBTs and Standard. Scores are percentages ( ↑ ); the best displayed score within each group is bold.
Family
Scale
Allocation
Physical blocks / depth
Shared or all-layer d/m/f
Core or branch d/m/f
Standard
W1
12
12 / 12
1024/512/512
—
W2
14
14 / 14
1152/576/576
—
W3
16
16 / 16
1280/640/640
—
W4
18
18 / 18
1408/704/704
—
Global wide–shallow
W1
9
9 / 9
1280/640/512
—
(sequential)
W2
10
10 / 10
1408/704/704
—
Appendix
Table 13: All five families in Figure 10 . Widths are residual / MoE latent / expert FFN ( d/m/f ). Shared widths follow Table 7 ; all configurations retain the per-scale training-token budget.
Q/KV heads
Scale
Allocation
Branch d/m/f
Outer
Joint-KV
Independent-KV
KV/token (KiB)
Body compute
Loss ↓
W1
3/(5+5)/3
768/368/384
16/2
16/2
16/1
5.5
0.3867/0.3848
2.646/2.657
W2
3/(6+6)/3
896/464/464
18/2
16/2
16/1
6.0
0.7763/0.7723
2.557/2.568
W3
4/(6+6)/4
1024/512/512
20/2
16/2
16/1
7.0
1.4499/1.4436
2.487/2.499
W4
4/(7+7)/4
1152/592/592
24/3
18/2
18/1
9.5
2.6664/2.6551
2.380/2.392
Appendix
Table 14: W1–W4 configurations for Section 4.2 . Cache size is identical within each pair. Body compute (PFLOP-days) and loss report Joint-KV / Independent-KV at the same training-token budget.
Figure 12: Quantities changed by the serving mechanisms of Figure 4 for the 4-branch W3 Balanced MBT, relative to Standard. All values are structural counts, not measurements. Left: serial block steps per token when the branches run one after another (Naive) and with branch grouping (Ours). Middle: KV cache stored per token. Right: KV read per decode step with Joint-KV, when each branch reads the shared bank separately (Naive) and with head folding (Ours).
Scale
Standard
K=4
K=3
K=2
W1
12
9
10
11
W2
14
11
12
12
W3
16
12
13
14
W4
18
15
15
15
Appendix
Table 15: Depth, i.e. , serial blocks per decode step, of Standard and of the Joint-KV MBTs in Figure 13 (Eq. 5 ).
Figure 13: Decode speedup of Joint-KV MBTs over Standard at all scales. Decode throughput relative to Standard on one H200 GPU, as a function of the number of concurrent requests N , for 4K-, 16K-, 64K-, and 128K-token inputs that each generate 1K tokens. Rows are scales W1–W4, and all panels share the vertical axis.
W1
W2
W3
W4
Input
N
K=4
K=3
K=2
Drift
K=4
K=3
K=2
Drift
K=4
K=3
K=2
Drift
K=4
K=3
K=2
Drift
4K
1
1.12
1.04
0.96
− 4.2
1.12
1.03
1.02
− 3.0
1.15
1.07
0.99
+0.5
1.09
1.08
1.06
− 1.8
8
1.10
1.03
0.93
+6.2
1.18
1.09
1.11
+2.0
1.21
1.15
1.11
+1.6
1.08
1.10
1.03
+3.3
64
1.23
1.15
1.08
+0.3
1.26
1.19
1.13
+0.7
1.24
1.17
1.12
− 0.1
1.26
1.23
1.19
− 0.2
16K
1
1.15
1.06
0.98
+3.1
1.15
1.05
1.05
+0.9
1.17
1.10
1.01
+2.8
1.10
1.09
1.06
+2.3
8
1.21
1.13
0.95
+0.6
1.33
1.18
1.20
+1.1
1.30
1.23
1.22
− 0.5
1.17
1.18
1.08
+0.3
Appendix
Table 16: Decode speedup of Joint-KV MBTs over Standard at all scales on one H200 GPU. The speedup is the decode throughput of an MBT divided by that of Standard at the same input length and concurrency N . Drift (%) is the change of the Standard throughput between the runs before and after the Joint-KV models. At 128K, each model has its own pair of Standard runs, and the table lists the largest of the three drifts. For each scale and row, the fastest model is in bold and the second fastest is underlined .
Figure 14: Contribution of each engine mechanism. Decode speedup over Standard of the 4-branch W3 Joint-KV MBT under three engines on one H200 GPU, for N concurrent requests that each generate 1K tokens.
Scaling a language model is not only a question of final quality: the architectural choice determines how much computation is spent during training, prompt processing, and autoregressive decoding to achieve certain model quality. An ideal model architecture should lower all above computation costs to facilitate scaling to a larger model, while ensure the larger model indeed outperforms smaller baselines. We introduce KV-Invariant Transformer Expansion (KITE), a scaling paradigm that achieves this goal. It trains the model from a smaller size to a larger size (i.e., saving training costs via upcycling), while places newly added parameters in regions that do not affect attention KV. Consequently, during inference, prefilling KV only relies on the smaller part of the model, so the inference costs are saved. As a concrete instantiation, we present Step Scale Transformer (SST), a two-tower decoder in which one tower produces KV and the other reads them. At comparable cumulative training compute, SST, a 67B MoE model with 2.15B active body parameters per decode token, achieves lower training loss than 47B and 63B MoE Transformers with 1.48B and 2.02B active body parameters, respectively, while reducing estimated inference cost by 6.7% and 31.6%.
Looping in transformer models reuses weights by repeatedly applying the same transformer, or parts of it, making the model deeper without increasing its parameter count. Whether looping is useful depends on which constraint dominates: model size, training compute, or autoregressive inference efficiency. We study these trade-offs jointly in dense decoder-only transformers trained from scratch, varying which parts of the model are looped, how many loops are applied, and how information is passed between loops. Our experiments span models with up to 1B parameters, trained on up to 155B tokens with up to 1.07×1021 FLOPs and 32 loops. We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained. Adding loops is not uniformly beneficial: gains eventually saturate, reverse, or become unstable. This motivates asking whether forcing every loop to use the same weights limits the benefit of deeper looping. We address this by adding small loop-specific low-rank adapters that improve quality, with larger gains as more loops are added. Looped transformers also suffer a decoding slowdown proportional to the number of loops. Parallel Loop Transformers address this by passing hidden states between loops across adjacent token positions, allowing loop stages to run concurrently. However, in our experiments the original formulation does not consistently retain the quality gains of ordinary looping. We find that preserving the current-token embedding and separately normalizing it and the previous-token hidden state before combining them recovers strong performance across model scales and training budgets. Together, these results show that effective looping requires matching the design to the resource constraint and inference setting.
Recent test-time scaling methods improve model performance by increasing inference-time compute. One practical form of this is parallel scaling, where a model samples multiple independent reasoning streams and aggregates their answers using methods like voting, reranking, or verification. While effective, this approach leaves the streams uncoordinated, and a discovery made in one rollout cannot influence any other. Multi-Sequence Verifiers jointly score candidate answers after generation, but the reasoning streams remain independent during inference. ParScale takes a step toward model-internal parallel scaling by learning to aggregate multiple forward passes, but its streams still do not exchange information while processing a sequence. We introduce BatchParScale, an attention mechanism that allows parallel streams to exchange information through a small set of communication positions. We evaluate BatchParScale on a new Parallel Needle-in-a-Haystack retrieval benchmark and in pretraining, fine-tuning, and cooperative multi-agent reinforcement learning. Across these settings, BatchParScale outperforms ParScale on language-model loss and remains competitive on coordination-sensitive MARL tasks. Masked pretraining runs beat an unmasked single-sequence baseline despite local sequence corruption, and LoRA fine-tuning shows the mechanism can be added to a pretrained Transformer. At the same time, fact retrieval performance for pretrained and finetuned BatchParScale models remains unclear.
Hans Peter Lyngsøe Raaschou-jensen, M. Onurcan Kaya, Desmond Elliott
University of Copenhagen · Technical University of Denmark