Symbol-Equivariant Recurrent Reasoning Models

Richard Freinschlag; Timo Bertram; Erich Kobler; Andreas Mayr; Günter Klambauer

cs.LGMar 2, 2026

Symbol-Equivariant Recurrent Reasoning Models

Authors: Richard Freinschlag, Timo Bertram, Erich Kobler, Andreas Mayr, Günter Klambauer

Organizations: ELLIS Unit Linz, LIT AI Lab & Institute for Machine Learning, Johannes Kepler University Linz, Austria · Clinical Research Institute for Medical AI, Johannes Kepler University Linz, Austria

Abstract

Reasoning problems such as Sudoku and ARC-AGI remain challenging for neural networks. The structured problem solving architecture family of Recurrent Reasoning Models (RRMs), including Hierarchical Reasoning Model (HRM) and Tiny Recursive Model (TRM), offer a compact alternative to large language models, but currently handle symbol symmetries only implicitly via costly data augmentation. We introduce Symbol-Equivariant Recurrent Reasoning Models (SE-RRMs), which enforce permutation equivariance at the architectural level through symbol-equivariant layers, guaranteeing identical solutions under symbol or color permutations. SE-RRMs outperform prior RRMs on 9x9 Sudoku and generalize from just training on 9x9 to smaller 4x4 and larger 16x16 and 25x25 instances, to which existing RRMs cannot extrapolate. On ARC-AGI-1 and ARC-AGI-2, SE-RRMs achieve competitive performance with substantially less data augmentation and only 2 million parameters, demonstrating that explicitly encoding symmetry improves the robustness and scalability of neural reasoning. Code is available at https://github.com/ml-jku/SE-RRM.

Figures & tables

Figure 1: Comparison of a RRM, in particular TRM ( Jolicoeur-Martineau, 2025 ) , (left) and the proposed SE-RRM (right) for solving a

4\times 4

Sudoku. For RRM, the puzzle is encoded with a symbol-specific embedding at every position, resulting in a 2D matrix. In contrast, SE-RRM introduces a third dimension to link positions and symbols and uses the same embedding for all symbols. SE-RRM is symbol-equivariant, since both Self-Attention

T^{D,I}

and

T^{D,K}

layers in the transformer block operate subsequently in position and symbol dimension, and the norm, addition, and MLP layers are applied token-wise.

	4 $\times$ 4 Mini Sudoku		9 $\times$ 9 Sudoku		16 $\times$ 16 Maxi Sudoku		25 $\times$ 25 Ultra Sudoku
Model	FSR	GPA	FSR	GPA	FSR	GPA	FSR	GPA
HRM	$0^{(0-1.32)}$	$29.19^{(27.45-30.95)}$	$63.53^{(63.38-63.67)}$	$86.11^{(86.10-86.12)}$	– a	– a	– a	– a
TRM	$0^{(0-1.32)}$	$45.88^{(43.96-47.79)}$	$71.94^{(71.80-72.08)}$	$89.80^{(89.79-89.81)}$	– a	– a	– a	– a
SE-RRM (ours)	${\bf 95.46}^{(92.01-97.08)}$	${\bf 99.15}^{(98.67-99.41)}$	${\bf 93.73}^{(93.66-93.81)}$	${\bf 97.58}^{(97.54-97.63)}$	$0^{(0-1.75)}$	${\bf 51.95}^{(51.43-52.47)}$	$0^{(0-8.38)}$	${\bf 31.49}^{(30.22-32.75)}$
GPT-OSS-20B b	$100^{(98.68-100)}$	$100^{(99.85-100)}$	$21.67^{(13.12-33.62)}$	– c	– d	– d	– d	– d

Table 1: Sudoku performance of different methods across grid sizes. We report fully solved rate (FSR) , i.e., the percentage of puzzles solved exactly, and grid-point accuracy GPA , i.e., the percentage of correctly predicted unfilled Sudoku cells, for each mode with 95% Wilson score confidence intervals. All models except GPT-OSS ( OpenAI et al., 2025 ) are trained on

9\times 9

Sudoku grids. The best RRM is marked bold. SE-RRM generalizes to larger problem sizes, while the other RRMs are unable to accommodate unseen symbols.

	Number of inference steps
Model	1	2	4	8	16	32	64	128
HRM	$2.01$	$20.47$	$49.37$	$59.45$	$63.53$	$65.67$	$67.09$	$68.15$
TRM	$2.51$	$31.02$	$55.49$	$66.46$	$71.95$	$75.36$	$77.86$	$79.78$
SE-RRM (ours)	$\mathbf{16.05}$	$\mathbf{62.06}$	$\mathbf{77.31}$	$\mathbf{87.38}$	$\mathbf{93.73}$	$\mathbf{96.82}$	$\mathbf{98.22}$	$\mathbf{98.84}$

Table 2: Test-time scaling on

9\times 9

Sudoku. Fully solved rate (FSR in %) as a function of the number of deep supervision steps at inference.

	ARC-AGI-1	ARC-AGI-2	Maze
Model	pass@2		FSR
HRM	$40.3^{(35.56-45.13)}$	$5.0^{(2.31-10.48)}$	$74.5^{(71.66-77.15)}$
TRM	44.6 (39.71-49.40)	7.8 (4.0-13.64)	$85.3^{(82.92-87.41)}$
SE-RRM (ours)	$\textbf{45.3}^{(40.44-50.15)}$	7.1 (3.42-12.61)	88.8 (86.64-90.65)

Table 3: pass@2 on ARC-AGI benchmarks and fully solved rate (FSR) on Maze with 95% Wilson score confidence intervals. Reported values for HRM and TRM are taken from the respective papers.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Definition	Symbol	Type
Scalars and indices
feature dimension	$D$	$\mathbb{N}$
number of positions (grid points)	$I$	$\mathbb{N}$
number of symbols / colors	$K$	$\mathbb{N}$
recurrent iteration index	$t$	$\mathbb{N}$
number of layers per block	$L$	$\mathbb{N}$

Appendix

Table A1 : Notation used in the description of Recurrent Reasoning Models (RRMs), Hierarchical Reasoning Models (HRMs), Tiny Recursive Models (TRMs), and Symbol-Equivariant RRMs (SE-RRMs).

Figure A1 : Illustration of Model Equivariance under Symbol Permutation. The model f is equivariant if applying a permutation

\pi

to the input

X

results in an equivalent permutation of the output, such that

f(\pi(X))=\pi(f(X))

. Here, the permutation

\pi

swaps the symbols 1 and 4. In contrast to other reasoning models, SE-RRM guarantees permutation equivariance.

Figure A2 : Illustration of an exemplary ARC-AGI task where the colors of the squares in the input have to be exchanged. Vanilla RRMs would rely on extensive colour augmentations, while SE-RRMs can immediately learn the task type rule.

Hyperparameter	HRM	TRM	SE-RRM
Deep supervision steps	16	16	16
early stopping of deep supervision	q-learning	q-learning	random (5% chance per step)
Recursive Cycles	$H=2,L=2$	$H=3,L=6$	$H=3,L=6$
Batch Size	384	768	272
Number of epochs	50000	50000	10000
Learning Rate	0.0001	0.0001	0.0005

Appendix

Table A2 : Hyperparameter Comparison for Sudoku

Hyperparameter	HRM	TRM	SE-RRM
Deep supervision steps	16	16	16
early stopping of deep supervision	q-learning	q-learning	random (10% chance per step)
Recursive Cycles	$H=2,L=2$	$H=3,L=4$	$H=3,L=4$
Batch Size	768	768	272
Number of epochs	100000	100000	100000
Learning Rate	0.0001	0.0001	0.0005

Appendix

Table A3 : Hyperparameter Comparison for ARC-AGI. All models use the same hyperparameters for ARC-AGI-1 and ARC-AGI-2.

Hyperparameter	HRM	TRM	SE-RRM
Deep supervision steps	16	16	16
early stopping of deep supervision	q-learning	q-learning	random (10% chance per step)
Recursive Cycles	$H=2,L=2$	$H=3,L=4$	$H=3,L=4$
Batch Size	384	768	64
Number of epochs	20000	50000	10000
Learning Rate	0.0001	0.0001	0.0005

Appendix

Table A4 : Hyperparameter Comparison for Maze

Model	ARC-AGI-1 (%)
TRM	44.6 (39.71-49.40)
TRM + random halting	34.8 (30.25-39.54)
TRM + RoPE2d	38.4 (33.62-43.10)
TRM + random halting + RoPE2d	33.1 (28.57-37.75)
TRM + all hyperparameters matched	32.3 (27.86-36.98)
TRM + all hyperparameters matched + equivariance = SE-RRM (ours)	$\textbf{45.3}^{(40.44-50.15)}$

Appendix

Table A5 : pass@2 on ARC-AGI-1 benchmark with 95% Wilson score confidence intervals. The hyperaparameters and architecture of TRM is changed step by step into SE-RRM.

Figure A3 : Learning curves for HRM, TRM, and SE-RRM. Validation metrics HRM, TRM and SE-RRM on Sudoku, ARC-AGI-1 and Maze are shown on the y-axis. The number of samples processes during training is shown on the x-axis. SE-RRM is more data efficient since it reaches higher metrics with lower number of samples processed.

Model	Halting	PosEnc	Hyperparameters	Sudoku (%)	Maze (%)
TRM	q-learning	RoPE	TRM	$71.9^{(71.80-72.08)}$	$85.3^{(82.92-87.41)}$
TRM	random	RoPE	TRM	$79.4^{(79.31-79.55)}$	$84.9^{(82.55-86.99)}$
TRM	q-learning	RoPE2D	TRM	$76.5^{(76.34-76.59)}$	$85.4^{(83.08-87.45)}$
TRM	random	RoPE2D	TRM	$0^{(0.00-0.00)}$	$59.3^{(56.23-62.30)}$
TRM	random	RoPE2D	SE-RRM	$0^{(0.00-0.00)}$	$65.0^{(61.99-67.89)}$
SE-RRM	random	RoPE2D	SE-RRM	93.7 (93.66-93.81)	88.8 (86.64-90.65)

Appendix

Table A6 : Ablations of TRM and SE-RRM on Sudoku and Maze.

	Symbol	Sudoku	Maze
Model	Equivariance	FSR	FSR
TRM	False	$71.9^{(71.80-72.08)}$	$85.3^{(82.92-87.41)}$
HRM	False	$63.5^{(63.38-63.67)}$	$74.5^{(71.66-77.15)}$
SE-RRM	True	93.7 (93.66-93.81)	$0^{(0.00-0.38)}$
SE-RRM	False	89.7 (89.61-89.79)	88.8 (86.64-90.65)

Appendix

Table A7 : Results of SE-RRM on Sudoku and Maze including ablations. The second column indicates whether symbol equivariance is enforced through shared embeddings.

Explore similar work

Recursive Vision Language Models for General Symbolic Reasoning
Aug 2, 2026Omid Nejati Manzari, Guillaume Lajoie, Hassan RivazSymbolic Reasoning Implicit Reasoning in Language Models
Concordia University, Montreal, Canada · Mila — Quebec AI Institute · Université de Montréal
G-RRM: Guiding Symbolic Solvers with Recurrent Reasoning Models
Jul 2, 2026Timo Bertram, Sidhant Bhavnani, Richard Freinschlag +3Symbolic Reasoning Neuro-Symbolic Reasoning
ELLIS Unit Linz, LIT AI Lab & Institute for Machine Learning, Johannes Kepler University Linz, Austria · Institute for Symbolic Artificial Intelligence, Johannes Kepler University Linz, Austria · Clinical Research Institute for Medical AI, Johannes Kepler University Linz, Austria
Form Follows Function: Recursive Stem Model
Mar 3, 2026Navid HakimiNeural Algorithmic Reasoning Algorithmic Reasoning

Aug 2, 2026J/K move · Enter open · S save

Recursive Vision Language Models for General Symbolic Reasoning

Omid Nejati Manzari, Guillaume Lajoie, Hassan Rivaz

Concordia University, Montreal, Canada · Mila — Quebec AI Institute · Université de Montréal

Symbolic Reasoning Implicit Reasoning in Language Models Algorithmic Reasoning LLM Reasoning

Open paper AskPDF arXiv

Hard symbolic-reasoning tasks such as Sudoku, maze pathfinding, and ARC remain challenging for LLMs due to their fixed-depth autoregressive reasoning, which limits systematic search, refinement, and backtracking. While recursive models such as Hierarchical Reasoning Model (HRM) and Tiny Recursive Model (TRM) address this limitation through iterative latent-state refinement, they are typically task-specific and do not leverage pretrained language priors. We propose R-Qwen, a recursive reasoning framework built upon a pretrained Qwen backbone. R-Qwen repeatedly refines a candidate solution through programmatic self-recursion and deep supervision, combining the structured iterative computation of recursive models with the linguistic and reasoning priors of pretrained LLMs. We further adapt Hierarchical Supervision Weighting (HSW) to autoregressive models by exponentially weighting losses across recursive steps. HSW reduces gradient variance by at least 50%, improves the signal-to-noise ratio of stochastic gradients, and accelerates convergence. Across eight challenging benchmarks, R-Qwen consistently outperforms prior recursive reasoning models and substantially larger LLMs while using a comparable number of trainable parameters. Notably, on ARC-AGI dataset, our model achieves a 27.6% improvement over the baseline, highlighting the effectiveness of recursive refinement for general symbolic reasoning. These results suggest that recursive reasoning mechanisms and pretrained language model priors are complementary approaches for improving symbolic puzzle-solving. Code and models will be released after acceptance.

Abstract

Explore similar work

Recursive Vision Language Models for General Symbolic Reasoning

G-RRM: Guiding Symbolic Solvers with Recurrent Reasoning Models

Form Follows Function: Recursive Stem Model