Random Recursive Models

Jama Hussein Mohamud; Mirco Ravanelli

cs.LGSep 30, 2026

Random Recursive Models

Authors: Jama Hussein Mohamud, Mirco Ravanelli

Organizations: Mila – Quebec AI Institute · Université de Montréal · Concordia University

Abstract

Recursive models create computational depth through parameter reuse, offering a parameter-efficient alternative to increasing model size. However, most recursive models repeatedly apply one learned transformation or a prescribed sequence of transformations, restricting computation to a fixed layer order. We introduce the Random Recursive Model (RRM), which maintains a pool of $L$ learned layers and performs $T$ recursive steps by sampling one layer independently with replacement for each example and step. This enables flexible layer reuse while retaining the parameter efficiency of recurrence. We evaluate RRM on challenging reasoning tasks, where it matches or exceeds the baselines, often with 50-75 % fewer parameters. RRM can vary its depth at inference, including beyond that seen during training, without retraining or adding parameters, improving tasks that benefit from deeper iterative computation. RRM also supports Monte Carlo inference and probabilistic test-time scaling, both of which improve performance without retraining. These insights may open new directions in neural network architecture design.

Figures & tables

Panel of Figure 1 — Figure 1: Fixed-order depth, fixed-recursive depth, and RRM. Four applications under the three depth constructions. Fixed-order depth applies each layer once in its prescribed position. Fixed-recursive depth repeatedly applies a prescribed sequence of learned functions. Here, the repeated sequence contains one function ( $L=1$ ); more generally, it may contain multiple layers and be surrounded by fixed-order layers. RRM samples from the complete set of learned layers at every step, allowing both reordering and reuse. Colors identify shared parameters.

Figure 2: CIFAR-10 representations across recursive depth. We take the class-token representations from the RRM trained with $L=2$ and $T_{\mathrm{train}}=16$ and plot them at $T_{\mathrm{test}}\in\{8,12,16,20,24\}$ , using one sampled layer sequence per image. Each panel is a t-SNE projection of the same test images, colored by class. Both the $k$ -NN classification accuracy and the t-SNE projections show that the representations become more class-specific as the number of recursive applications increases.

Model	$L$	$T_{\mathrm{train}}$	Params.	$M$	Accuracy (%)
Dense	8	8	3.20M	1	$92.520\pm 0.000$
RRM	2	8	0.824M	1	$91.403\pm 0.078$
RRM	2	16	0.824M	1	$91.670\pm 0.155$
RRM	2	16	0.824M	16	$92.337\pm 0.049$
RRM	2	8	0.824M	16	$\mathbf{92.597\pm 0.110}$

Table 1: CIFAR-10 top-1 accuracy.

M

denotes Monte Carlo inference samples.

Table 4

Model	$L$	$T$	Params.	Accuracy (%)
HRM	8	24	27M	$55.00$
TRM	2	42	5M	$87.63$
RRM	2	21	5M	$87.22\pm 0.03$
TRM+RRM	2	42	5M	$89.17\pm 0.02$
RRM †	2	32	5M	$\mathbf{89.21\pm 0.03}$

Table 4: Sudoku-Extreme exact-grid accuracy. HRM, TRM, and TRM+RRM use

N_{\mathrm{sup}}=16

, while RRM uses

N_{\mathrm{sup}}=32

\dagger

evaluates the same RRM with

T_{\mathrm{test}}=32

without retraining. The TRM row is our reproduction; the published result is

87.40\%

( Jolicoeur-Martineau, 2025 ) .

Model	$L$	$T$	$T\!\times\!N_{\mathrm{sup}}$	Params.	Pass@2 (%)
HRM	8	24	384	27M	$40.30$
TRM-Att	2	30	480	7M	$44.60$
RRM †	2	15	30	7M	$45.79\pm 0.29$
RRM	2	15	480	7M	$\mathbf{47.38\pm 0.25}$

Table 5: ARC-AGI-1 Pass@2 accuracy. Baseline results are from Jolicoeur-Martineau (2025) .

\dagger

evaluates RRM with

N_{\mathrm{sup}}=2

instead of the 32 used during training.

Panel of Figure 3 — Figure 3: Top: Parameter efficiency across four tasks. RRM matches or outperforms the corresponding baselines with substantially fewer parameters in (a–c) and at a comparable parameter count in (d). Sudoku LM denotes the autoregressive Sudoku task in Section 5.2 . Bottom: Inference compute efficiency across four reasoning tasks. FLOPs are measured per evaluation example and include all applications across supervision steps when deep supervision is used. Overall, RRM consistently improves parameter efficiency and achieves competitive or better compute efficiency on three of the four reasoning tasks, with the largest gains on Maze-Hard and ARC-AGI-1.

Figure 4: Effect of the number of reusable layers. RRM remains effective with

L=1

, while

L=2

matches or exceeds the baselines on both tasks.

Panel of Figure 5 — Figure 5: Top: Test-time scaling of RRM without retraining or adding parameters. Vertical dotted lines mark $T_{\mathrm{train}}$ . Horizontal dashed lines show the corresponding baselines. Additional test-time applications leave performance in (a,b) largely unchanged but improve it in (c,d). Bottom: Train-test depth generalization. Rows indicate $T_{\mathrm{train}}$ and columns indicate $T_{\mathrm{test}}$ ; outlined cells mark $T_{\mathrm{train}}=T_{\mathrm{test}}$ . Across both tasks, performance decreases as $T_{\mathrm{test}}$ moves farther below $T_{\mathrm{train}}$ . When $T_{\mathrm{test}}>T_{\mathrm{train}}$ , Maze-Hard improves or plateaus, whereas CIFAR-10 plateaus or degrades depending on $T_{\mathrm{train}}$ .

Model	Standard (%)	PTRM (%)
TRM	$87.20$	$98.43\pm 0.12$
TRM+RRM	$88.57\pm 0.25$	$98.83\pm 0.06$
RRM	$87.20\pm 0.26$	$\mathbf{99.50\pm 0.10}$

Table 6: Sudoku-Extreme accuracy with standard inference and PTRM test-time scaling.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$L$	$T$	$M$	Accuracy (%)
Dense	8	$8$	1	$92.52$
Learned policy	8	$7.99$	1	$85.07$
RRM	2	$8$	1	$91.40$
RRM	2	$8$	16	$92.60$
RRM	8	$32$	1	$\mathbf{93.67}$

Appendix

Table 7: CIFAR-10 learned-ordering ablation. The learned policy’s reported

T

is the mean number of applications.

M

denotes Monte Carlo inference samples.

Panel of Figure 6 — Figure 6: CIFAR-10 training-depth and test-time-scaling experiments. All RRMs use $L=2$ . Vertical dotted lines mark $T_{\mathrm{train}}$ , and horizontal dashed lines show the dense eight-layer baseline.

Model	$T_{\mathrm{detach}}$	$T_{\mathrm{train}}$	$T_{\mathrm{test}}$	$N_{\mathrm{sup}}$	Total	Pass@2 (%)
TRM-Att	–	30	30	16	480	$44.60$
RRM	0	8	8	32	256	$42.67\pm 0.14$
RRM	0	8	15	32	480	$42.79\pm 0.26$
RRM	5	15	15	32	480	$43.79\pm 0.29$
RRM	5	15	21	32	672	$43.79\pm 0.14$
RRM	0	15	15	2	30	$45.79\pm 0.29$

Appendix

Table 8: ARC-AGI-1 ablations.

T_{\mathrm{detach}}

is the number of initial applications, out of

T_{\mathrm{train}}

, through which gradients are truncated. Total gives

T_{\mathrm{test}}\times N_{\mathrm{sup}}

. All models use

L=2

. The TRM-Att result is from Jolicoeur-Martineau (2025) .

Explore similar work

What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling
Sep 30, 2026Yuliana Shakhvalieva, Dmitrii Kharchev, Viacheslav Bezrukov +4Recursive Models Large Reasoning Models
RND NLP, DAIMLD, Russian Federation
Probabilistic Tiny Recursive Model
May 19, 2026Amin Sghaier, Ali Parviz, Alexia Jolicoeur-MartineauRecursive Models Probabilistic Model
Mila – Quebec AI Institute ILLS & ETS Montreal · Independent
Generative Recursive Reasoning
May 19, 2026Junyeob Baek, Mingyu Jo, Minsu Kim +3Recursive Models Latent Trajectories
KAIST · Mila – Québec AI Institute · New York University +1

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling

Yuliana Shakhvalieva, Dmitrii Kharchev, Viacheslav Bezrukov, Inessa Fedorova+3

RND NLP, DAIMLD, Russian Federation

Recursive Models Large Reasoning Models Recurrent State

Open paper AskPDF arXiv

Recursive reasoning models apply a small shared Transformer block many times to refine a latent state. This gives them large effective depth with few parameters and makes them strong on algorithmic tasks. Such compact solvers are natural candidates for tools that an LLM can call on narrow algorithmic subproblems. However, existing models such as HRM, TRM and URM differ in architecture, gradient propagation and training procedure simultaneously. This makes it hard to tell what drives their performance, and their optimization is still poorly understood and often unstable. In this work we address both of these gaps. First, we study these questions under a unified experimental pipeline spanning six algorithmic domains. Individual controlled ablations are performed on representative domains, while the resulting recipe is evaluated across the full suite. The study reveals a surprisingly simple recipe for stable and generalizable recursive reasoning: an intermediate gradient horizon, large physical batches and controlled updates of the recurrent state. An explicit hierarchical architecture is not needed. Second, we combine these findings into a stable 13.6M-parameter model that achieves the strongest overall performance among the evaluated recursive baselines, with particularly large gains on out-of-distribution generalization. It raises Arithmetic OOD accuracy to 71.2%, from 36.2% for the strongest baseline, while reaching 98.41% on Sudoku and 59.5% pass@2 on ARC-AGI-1. Our results show that, within the recursive architectures studied here, performance depends strongly on how recurrence is optimized and stabilized. More broadly, it shows how AI systems can be improved by optimizing their components one at a time.

Abstract

Explore similar work

What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling

Probabilistic Tiny Recursive Model

Generative Recursive Reasoning