Block Recursive Transformers for Structured Parameter Sharing

Josep Marques; Hyunyoung A. Cho; Youngeun Kim; Youjia Zhang; Aecheon Jung; Kyuhong Shim; Sungeun Hong

cs.LGOct 18, 2026

Block Recursive Transformers for Structured Parameter Sharing

Authors: Josep Marques, Hyunyoung A. Cho, Youngeun Kim, Youjia Zhang, Aecheon Jung, Kyuhong Shim, Sungeun Hong

Organizations: Sungkyunkwan University · Korea University

Abstract

Recursive Transformers depend not only on how many parameters are shared, but also on how shared parameters are organized across depth. Existing static schemes adopt two extreme reuse topologies: layer-wise repetition, where each layer is reused independently, and global cyclic reuse, where the entire stack is repeatedly applied as a single shared function. We argue that sharing granularity is an important factor in recursive parameter sharing. This study proposes Block Recursive Transformers (BRT), which partition layers into contiguous blocks and recursively reuse each block before moving to the next. This intermediate topology enables local refinement within blocks while preserving depth-wise transitions, without adding routing modules, auxiliary states, or extra training objectives. Across decoder-only models up to 1.7B parameters, with additional encoder-decoder and encoder-only studies, BRT improves validation negative log-likelihood and downstream accuracy over static recursive sharing baselines. Ablations and representation analyses further show that intermediate block granularity better balances local refinement and depth-wise differentiation. Project website: https://aim-skku.github.io/BRT/

Figures & tables

Figure 1: Comparison of sharing strategies. (a) Sequence sharing repeats each layer independently. (b) Cycle sharing reuses the full stack as one global unit. (c) Our BRT recursively reuses contiguous layer blocks before moving to the next block. Sequence and Cycle sharing are boundary cases.

Figure 2: Centered Kernel Alignment (CKA) similarity across depth and recursion steps. (a) Vanilla Transformer shows gradual representation changes across depth. (b) Sequence sharing shows high similarity across layers, suggesting limited depth-wise differentiation. (c) Cycle sharing shows high similarity across recursion steps, consistent with representational saturation under global reuse. (d) Our BRT shows stronger intra-block similarity and lower cross-block similarity, suggesting local refinement while preserving transitions across depth regions.

Model	Recursion		Pretrain		Few-shot Accuracy $\uparrow$
Model	$N_{r}$	Share	Param	$N_{tok}$	ARC	WG	MMLU	HS	PIQA	Avg
Vanilla	-	-	315M	20B	$34.87\pm 0.31$	$50.92\pm 0.41$	$25.12\pm 0.17$	$28.32\pm 0.16$	$59.45\pm 0.32$	$39.74\pm 0.14$
Recursive		Seq	157M	20B	$32.90\pm 0.40$	$50.03\pm 0.42$	$24.88\pm 0.34$	$27.51\pm 0.22$	$59.61\pm 0.31$	$38.98\pm 0.30$
		Cyc	157M	20B	$34.12\pm 0.50$	$50.37\pm 1.13$	$24.39\pm 0.26$	$28.37\pm 0.10$	$\textbf{59.90}\pm 0.43$	$39.43\pm 0.20$
	2	M-Seq	167M	20B	$32.96\pm 0.11$	$50.76\pm 0.34$	$25.21\pm 0.41$	$28.02\pm 0.34$	$59.17\pm 0.50$	$39.22\pm 0.23$
		M-Cyc	167M	20B	$34.73\pm 0.21$	$51.12\pm 0.63$	$25.12\pm 0.19$	$28.59\pm 0.12$	$59.23\pm 0.10$	$39.76\pm 0.12$

Table 1: Main results on SmolLM-360M under a fixed 20B-token training budget. We compare vanilla and recursive Transformers with different static sharing strategies.

N_{r}

denotes the number of recursion steps, and

N_{\text{tok}}

denotes the number of training tokens. The best average performance is highlighted in bold. Results are reported as mean and standard deviation across 3 seeds.

Figure 3: Validation NLL of SmolLM-360M variants under different training compute budgets. The x-axis shows estimated training FLOPs ( $\times 10^{18}$ ). NLL is computed on 2,000 FineWeb-Edu validation samples. All models are pretrained from scratch; sharing configurations are listed in Table 5 .

Figure 4: Comparison of parameter-sharing strategies under matched training-token budgets. We pretrain SmolLM-135M and SmolLM-360M variants with 1B, 5B, 10B, and 20B tokens under two and four recursion settings. We report mean accuracy over five few-shot benchmarks across three seeds.

Model	Recursion		Pretrain		Few-shot Accuracy $\uparrow$
Model	$N_{r}$	Share	Param	$N_{tok}$	ARC	WG	MMLU	HS	PIQA	Avg
Vanilla	-	-	1.61B	20B	$38.12\pm 0.22$	$\textbf{51.55}\pm 0.72$	$25.97\pm 0.26$	$\textbf{31.76}\pm 0.17$	$61.03\pm 0.16$	$\textbf{41.68}\pm 0.25$
Recursive	2	M-Cyc	870M	20B	$37.65\pm 0.26$	$50.93\pm 0.89$	$25.64\pm 0.21$	$31.31\pm 0.32$	$\textbf{61.13}\pm 0.12$	$41.33\pm 0.20$
Recursive	2	Ours	805M	20B	$\textbf{38.25}\pm 0.29$	$51.30\pm 0.61$	$\textbf{26.10}\pm 0.18$	$31.69\pm 0.22$	$60.82\pm 0.27$	$\underline{41.63}\pm 0.22$

Table 2: Comparison of parameter-sharing strategies in vanilla and recursive Transformers using SmolLM-1.7B. All models are trained for 20B tokens. We report task-wise and mean accuracy across three seeds. The best overall performance is highlighted in bold, while the best recursive strategy is underlined.

Sharing			Pretrain		Few-shot Accuracy $\uparrow$
$S$	Untied	Share	Param	$N_{\text{tok}}$	ARC	WG	MMLU	HS	PIQA	Avg
1	No	BRT	157M	10B	$31.35\pm 0.39$	$50.75\pm 0.59$	$24.93\pm 0.26$	$27.33\pm 0.13$	$58.01\pm 0.16$	$38.47\pm 0.24$
2	No	BRT	157M	10B	$\textbf{32.10}\pm 0.35$	$50.31\pm 0.26$	$24.74\pm 0.22$	$27.70\pm 0.11$	$\textbf{59.05}\pm 0.75$	$38.78\pm 0.24$
3	No	BRT	147M	10B	$32.08\pm 0.36$	$50.31\pm 0.36$	$24.53\pm 0.31$	$27.54\pm 0.13$	$57.92\pm 0.31$	$38.47\pm 0.04$
4	No	BRT	157M	10B	$32.07\pm 0.17$	$49.40\pm 0.53$	$24.35\pm 0.29$	$27.49\pm 0.11$	$58.48\pm 0.32$	$38.36\pm 0.07$
16	No	BRT	157M	10B	$30.42\pm 0.50$	$51.22\pm 1.17$	$24.71\pm 0.19$	$26.79\pm 0.20$	$57.20\pm 0.20$	$38.07\pm 0.32$

Table 3: Block-count ablation of BRT on SmolLM-360M.

S

denotes the number of blocks. The Untied column indicates whether the first and last layers are excluded from recursive sharing. The largest

S

differs by setting because the recursive core has a different number of layers when boundary layers are untied.

S=1

corresponds to cycle-style sharing, while the largest

S

values correspond to sequence-style sharing. All models are trained for 10B tokens and evaluated on five benchmarks.

Figure 5: Block-granularity analysis of BRT on SmolLM-360M architecture. Fully Recursive BRT denotes the setting without independent boundary layers ( $E=0$ ). All models are trained for 10B tokens. Points show mean accuracy over five benchmarks and three seeds; error bars show the standard deviation.

Figure 6: Comparison on encoder-decoder and encoder-only architectures under matched training tokens. (a) T5-220M with recursion applied to both stacks, decoder only, or encoder only. (b) Encoder-only ModernBERT-149M. All models are trained for 10B tokens, and NLL is computed on 2,000 FineWeb-Edu validation samples.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Description
Models	Architecture	Total Params	Layers
SmolLM	Decoder-only	135M	30
	Decoder-only	360M	32
	Decoder-only	1.7B	24
T5	Encoder-decoder	220M	12+12
ModernBERT	Encoder-only	149M	22

Appendix

Table 4: Description of the models used in our experiments. Total parameters include embedding and non-embedding parameters. The reported values correspond to the original model versions, considering both the embedding and non-embedding parameters.

Base Model	Setting	Share	$N_{\text{r}}$	$S$	Param	$D_{\text{exec}}$	$L_{\text{uniq}}$
SmolLM-135M	Vanilla	–	–	–	106M	30	30
	Recursive	Seq	2	15	53M	30	15
		Cyc		1	53M	30	15
		M-Seq		14	57M	30	16
		M-Cyc		1	57M	30	16
		BRT		2	57M	30	16

Appendix

Table 5: Comparison of parameter count, executed depth, and unique layers for different sharing strategies on SmolLM. Base Model denotes the original SmolLM scale, Setting indicates vanilla or recursive models, Param denotes the non-embedding parameters,

N_{\text{r}}

denotes the recursion number,

S

denotes the number of blocks,

D_{\text{exec}}

denotes executed depth, and

L_{\text{uniq}}

denotes unique layers.

Figure 7: Visualization of the representation similarity across layers in vanilla Transformers.

Figure 8: Visualization of the representation similarity across layers in sequence sharing.

Figure 9: Visualization of the training loss using SmolLM-360M for different sharing strategies. All models are trained for 10B tokens. We set the number of recursions to 2, and the exact executed depth and number of parameters can be found in Table 5 .

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 544 papers

Sep 18, 2026

Trading Depth for Time in Recurrent Transformers

Recurrent Transformers increase computational depth through temporal recurrence, feeding each token's high-level hidden state into the computation of the next. This raises a natural question: is additional computation better spent on more temporal steps or greater physical depth? We investigate this question using Latent Recurrent Transformers (LRT), which retain one backbone forward pass per vocabulary token during decoding and provide a controlled setting for comparing these two ways of adding computation. We insert a latent thought token between consecutive vocabulary tokens, adding a hidden-state refinement step through the shared backbone before predicting the next token. On 16- and 20-layer NanoChat MoE backbones, we compare this model with double-depth vanilla Transformers and LRT, two-loop Transformers, and PonderLM-2 with one thought token. All configurations execute 2L blocks per vocabulary token during decoding. Under matched training FLOPs, LRT with one thought token outperforms PonderLM-2 and overtakes the two-loop Transformer as compute increases. Its gap to vanilla double-depth narrows from 0.014 to 0.001 bits per byte, despite approximately 46% fewer parameters and approximately half as many unique training tokens. At the training budget used for our main comparison, one thought token recovers an average of 89% of the gains from doubling LRT depth across the two backbone sizes, with approximately 48% fewer parameters. These results highlight temporal thinking as a parameter-efficient way to extend computation in recurrent Transformers.

Zeyi Huang, Xuehai He, Yelong Shen +1

Microsoft · University of Wisconsin–Madison

Transformer Recurrent Transformers

Ask PDF

Oct 23, 2026

LSTM-Style Looped Transformers with Shared KV Caches

Recurrent transformers reuse weights to perform deep computation with fewer parameters. We study an LSTM-style transformer that also shares one key--value cache across its recurrent core. At width 1024, it uses 40.4% fewer parameters than an untied control at nearly matched forward FLOPs, and reduces persistent cache sets from fifteen to four. Parallel training supplies cache entries from a standard-attention pass, whereas continuation reads entries produced recursively using shared attention. We address this mismatch by aligning the two training passes' cache entries with a consistency loss during the final 10% of a fixed training-token budget. Across three model widths on FineWeb, this removes 81--89% of the continuation-loss penalty relative to full-cache evaluation. The improvement replicates on disjoint documents for two training seeds. Across four widths and 1.05--4.19B training tokens, the loss cost of weight tying is smallest at lower token budgets, suggesting an opportunity to use recurrent computation when data and model storage are limited.

Andreas Kirsch

Recurrent Transformers KV-Cache Management

Ask PDF

Apr 23, 2026

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

Transformers process tokens in parallel but are temporally shallow: at position

t

, each layer attends to key-value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrent models offer unbounded temporal depth but suffer from optimization instability and historically underutilize modern accelerators. We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations, yielding layerwise recurrent memory while preserving standard autoregressive decoding cost. We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-token recurrent updates under mild assumptions, while avoiding optimization instability. Naively, prefill/training appears bandwidth-bound with effective arithmetic intensity near

1

because keys and values are revealed sequentially; we give an exact tiling-based algorithm that preserves the mathematical computation while reducing HBM traffic from

Θ(N^2)

Θ(N\log N)

, increasing effective arithmetic intensity to

Θ(N/\log N)

for sequence length

N

. On 150M and 300M parameter C4 pretraining, Recurrent Transformers improve cross-entropy over a parameter-matched Transformer baseline and achieve the improvement with fewer layers (fixed parameters), suggesting that recurrence can trade depth for width, thus reducing KV cache memory footprint and inference latency.

Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3

Harvard University

Autoregressive Language Modeling Self-Attention Attention Mechanisms Efficient Transformer Inference Recurrent Transformers Efficient Attention+1

Ask PDF

Abstract

Transformer

Explore similar work

Trading Depth for Time in Recurrent Transformers

LSTM-Style Looped Transformers with Shared KV Caches

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding