Dissecting Hierarchical Reasoning Models: A Mechanistic Study

Leo Raphael Rodrigues; Jian Kang

cs.LGAug 30, 2026

Dissecting Hierarchical Reasoning Models: A Mechanistic Study

Authors: Leo Raphael Rodrigues, Jian Kang

Organizations: Mohamed bin Zayed University of Artificial Intelligence

Abstract

We study Hierarchical Reasoning Model (HRM), a representative hierarchical Transformer-based latent reasoning model with many variants, on Sudoku, Maze, and ARC-AGI-2. We mechanistically understand how HRM reasons and what information it encodes. Our analyses compare HRM against Transformer baselines with and without recurrent modules, apply causal interventions on recurrent states, and utilize linear probes against random-direction ablations, as well as sparse autoencoders with feature ablations. Our results reveal several key findings: recurrent models outperform one-pass baselines, while single-state recurrent Transformers are comparable to HRM. State interventions further show that the causal contributions of the high- and low-level states vary across task-specific checkpoints and inference stages. Selected task variables are linearly decodable from the recurrent states in HRM, yet ablating probe directions produce effects comparable to random controls. SAE ablations yield larger behavioral changes than probe-direction ablations. However, top-ranked SAE features show no stable advantage over size-matched random subsets at larger ablation sizes or across tasks; the same pattern persists in a Sudoku control with within-step BPTT. Together, we characterize that HRM is essentially implementing constraint-aware iterative refinement on a puzzle-specific solution state, in which the functional contributions of components at different levels vary without relying on a compact, causally important feature set. These results highlight the necessity of studying the different working mechanisms and the importance of developing mechanistic interpretability techniques better suited for latent-space, recursive reasoning models.

Figures & tables

Figure 1: Across all five architectures, recurrent models (HRM, Recurrent/Universal Transformer) (a) steadily reduce Hamming distance to the ground-truth solution and (b) improve cell accuracy over 16 ACT steps, while the non-recurrent baselines (Plain Transformer, GRU) remain flat at

{>}30

mismatches and substantially lower accuracy.

Figure 2: Causal evidence for

z_{\mathrm{H}}

as a dynamic plan. (a) Per-step ablation: zeroing

z_{\mathrm{H}}

is increasingly destructive at later steps (

-6.8

at step 0 vs.

-26.0

at step 15). (b) Freeze curve: preventing

z_{\mathrm{H}}

updates after step

k

shows progressive refinement, freezing at step 0 loses

-9.1

; by step 8, the plan is nearly complete. (c) Time-shift: transplanting later

z_{\mathrm{H}}

states into earlier positions slightly improves accuracy, confirming monotonic refinement.

Figure 3: Causal evidence for Maze’s reorganized division of labor; values are percentage-point changes. (a) All-step ablation: token accuracy moves little for either stream, but valid paths rate fall

-32.5\%

under

z_{\mathrm{H}}

and

-8.5\%

under

z_{\mathrm{L}}

, so token accuracy is a weak diagnostic. (b) Per-step ablation:

z_{\mathrm{H}}

matters the most at the final readout; no single-step

z_{\mathrm{L}}

ablation exceeds

-4.5

. (c) Donor patching: donor-

z_{\mathrm{L}}

destroys

43

–

56\%

of valid paths from step 4 onward, whereas donor-

z_{\mathrm{H}}

is inert until readout (

-26

at step 15).

z_{\mathrm{L}}

is therefore the strongest patchable carrier of the intermediate path state.

Task	State / step	Target	Probe acc.	Baseline / note
Sudoku	$z_{\mathrm{H}}$ , step 15	Row / col / box violation	88.3 / 87.8 / 87.5	per-cell binary
Sudoku	$z_{\mathrm{L}}$ , step 15	Row / col / box violation	88.2 / 87.6 / 87.1	per-cell binary
Sudoku	$z_{\mathrm{H}}$ , step 15	Per-cell correctness	82.5	per-cell binary
Sudoku	$z_{\mathrm{L}}$ , step 15	Per-cell correctness	83.3	per-cell binary
Maze	$z_{\mathrm{H}}$ , step 15	Exact solved / valid S–G path	77.5 / 95.7	global path features
Maze	$z_{\mathrm{L}}$ , step 15	Exact solved / valid S–G path	75.8 / 92.1	global path features

Table 1: Selected linear-probe validation accuracies across tasks. Sudoku and Maze results use final-step activations from both recurrent streams where available. ARC-AGI-2 is included as a representation-level replication using

z_{\mathrm{H}}

at step 15. Expanded probe sweeps, MLP comparisons and complete target definitions are reported in Appendix A.7 , A.8 .

Figure 4: Causal effects for a representative SAE configuration. Values are mean percentage-point changes relative to the no-intervention output; negative values indicate performance degradation. Sudoku reports

\Delta

cell accuracy, Maze reports

\Delta

valid path rate, and ARC-AGI-2 reports

\Delta

color-cell accuracy. Full SAE sweeps are reported in Appendix A.10 .

Figure 5: Probe readout does not imply causal relevance. (a) Task variables are linearly decodable from HRM states across Sudoku, Maze, and ARC-AGI-2. (b) Directed ablation removes the corresponding probe directions and compares the behavioral effect with rank-matched random directions. (c) Maze exhibits the mismatch most clearly: both

z_{\mathrm{H}}

and

z_{\mathrm{L}}

decode global path variables, whereas Section 4 identifies donor-

z_{\mathrm{L}}

patching as the stronger intervention on the intermediate path state.

Model	Top-50 SAE	Random-50 SAE
Stock HRM	$-1.59$	$-1.37$
BPTT HRM	$-29.64$	$-29.63$

Table 2: BPTT training-regime check. Values are mean changes in cell accuracy under SAE-feature ablation over 200 shared puzzles. Top-50 and random-50 ablations are statistically indistinguishable in both regimes (paired

t

-tests: stock,

p=0.39

d=-0.06

; BPTT,

p=0.93

d=-0.01

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Recur.	Hier.	Steps	Params
HRM	✓	✓	16	27M
Recurrent Trans.	✓		16	27M
Univ. Trans.	✓		16	28M
Plain Trans.			1	27M
GRU			1	25M

Appendix

Table 3: Architectures compared in this paper. Recur. denotes refinement-step recurrence; Hier. denotes hierarchical structure; Steps denotes max encoder updates per input.

Checkpoint	Cell%	Puzzle%	Hamming
Recurrent Trans. epoch 2	76.2	34.4	19.3
Recurrent Trans. epoch 4	81.4	51.4	15.1
Recurrent Trans. epoch 5	74.3	32.6	20.8
Recurrent Trans. epoch 6	71.7	25.4	22.9
Univ. Trans. epoch 3	77.0	35.4	18.6
Univ. Trans. epoch 4	80.7	48.4	15.6

Appendix

Table 4: Baseline metrics across training checkpoints on 500 held-out Sudoku puzzles. Recurrent models are evaluated at segment 15; non-recurrent models at their single forward pass. Bold marks the best checkpoint of each model.

Figure 6: Cell accuracy across segments for all five architectures on the held-out 500-puzzle test set. The recurrent models (HRM, Recurrent Transformer, Universal Transformer) improve monotonically; the non-recurrent baselines (Plain Transformer, GRU) are flat single-pass.

Figure 7: Hamming distance to the ground-truth solution across segments for the same five architectures.

	Sudoku	Maze
State	Cell acc.	Token acc.	Path-cell acc.	Valid S–G	Connected
$z_{\mathrm{H}}$	$-19.3$	$-0.8$	$-2.7$	$-32.5$	$-26.0$
$z_{\mathrm{L}}$	$-19.3$	$-0.4$	$-1.4$	$-8.5$	$-7.0$

Appendix

Table 5: Combined zero-ablation results. Panel A reports all-step ablation under the previously reported task metrics. Panel B reports the task-specific primary metric at selected steps

k

: cell accuracy for Sudoku (

1{,}000

puzzles; clean baseline

81.8\%

) and valid start–goal path rate for Maze (

200

puzzles; clean baseline

93.5\%

). All values are percentage-point changes from the corresponding clean rollout. “All” zeroes the indicated stream at every recurrent step.

Task	State	$\mathbf{1}$	$\mathbf{2}$	$\mathbf{4}$	$\mathbf{8}$	$\mathbf{12}$	$\mathbf{14}$	$\mathbf{15}$
Sudoku	$z_{\mathrm{H}}$	$-6.2$	$-3.9$	$-2.0$	$-0.4$	$-0.2$	$0.0$	$0.0$
Sudoku	$z_{\mathrm{L}}$	$+0.2$	$-0.2$	$+0.2$	$+0.2$	$-0.1$	$0.0$	$0.0$
Maze	$z_{\mathrm{H}}$	$-0.5$	$-0.4$	$+0.7$	$+0.9$	$+0.6$	$0.0$	$0.0$
Maze	$z_{\mathrm{L}}$	$+0.5$	$0.0$	$+0.1$	$0.0$	$+0.3$	$0.0$	$0.0$

Appendix

Table 6: Combined freezing results. Each stream is frozen from step

k

onward. Sudoku reports

\Delta

cell accuracy (

1{,}000

puzzles; clean baseline

81.8\%

), while Maze reports

\Delta

valid start–goal path rate (

1{,}000

puzzles; clean baseline

92.4\%

). Values are percentage points.

Task	$\mathbf{9{\to}2}$	$\mathbf{12{\to}2}$	$\mathbf{15{\to}2}$	$\mathbf{10{\to}13}$	$\mathbf{10{\to}14}$	$\mathbf{10{\to}15}$
Sudoku	$+0.8$	$+0.7$	$+0.6$	$-0.2$	$-0.5$	$-0.6$
Maze	$+0.3$	$+0.8$	$+0.3$	$+0.2$	$+0.6$	$+0.6$

Appendix

Table 7: Selected within-instance

z_{\mathrm{H}}

temporal transplants. Column labels give the donor and recipient steps as donor

{\to}

recipient. Sudoku reports

\Delta

cell accuracy and Maze reports

\Delta

valid start–goal path rate. Both experiments use

1{,}000

puzzles, with clean baselines of

81.8\%

for Sudoku and

92.4\%

for Maze. Values are percentage points.

Task	Donor state	$\mathbf{1}$	$\mathbf{2}$	$\mathbf{4}$	$\mathbf{8}$	$\mathbf{12}$	$\mathbf{14}$	$\mathbf{15}$
Sudoku	$z_{\mathrm{H}}$	$-55.8$	$-58.8$	$-60.6$	$-61.7$	$-62.7$	$-65.4$	$-66.7$
Sudoku	$z_{\mathrm{L}}$	$-0.3$	$-1.5$	$-2.3$	$-0.2$	$-0.0$	$+0.3$	$-0.1$
Maze	$z_{\mathrm{H}}$	$-1.6$	$+0.4$	$-0.8$	$+0.8$	$+0.4$	$-4.8$	$-34.0$
Maze	$z_{\mathrm{L}}$	$-40.8$	$-43.6$	$-43.2$	$-44.0$	$-44.4$	$-42.8$	$-56.0$

Appendix

Table 8: Combined cross-puzzle patching results on the common step grid. Both tasks are evaluated over

250

donor–target pairs. Sudoku reports

\Delta

cell accuracy (clean target baseline

78.3\%

). Maze reports

\Delta

valid start–goal path rate under full-grid patching (clean target baseline

92.0\%

). Values are percentage points.

Figure 8: Recipient puzzle used in Figure 9 : given clues (left) and ground-truth solution (right).

Figure 9: Sudoku cross-puzzle patching example: baseline prediction on the target puzzle (left) versus prediction after donor- $z_{\mathrm{H}}$ patching (right). The patched output is rewritten toward the constraints of the donor puzzle. Blue denotes a given cell, green denotes a constraint-satisfying prediction, red denotes a constraint-violating prediction, and yellow denotes a prediction changed from the previous recurrent step. A combination of yellow and red/green denotes a cell that changed value and is constraint-violating/satisfying.

Figure 10: PCA trajectories of $z_{\mathrm{H}}$ across 16 segments for 200 Sudoku puzzles. (a) Solved puzzles converge to a compact cluster. (b) Failed puzzles remain broadly dispersed, including at their endpoints. (c) Overlay of solved and failed trajectories, which occupy distinct regions of principal-component space. Green dots denote step 0 and red crosses denote step 15.

Figure 11: Population-level metrics for

z_{\mathrm{H}}

across 16 segments, separated into solved and failed Sudoku puzzles: (a) consecutive-state cosine similarity,

\cos(z_{\mathrm{H}}^{(t)},z_{\mathrm{H}}^{(t-1)})

; (b) state norm,

|z_{\mathrm{H}}^{(t)}|

; (c) alignment with the final state; and (d) update magnitude,

|\Delta z_{\mathrm{H}}^{(t)}|

Figure 12: Maze $z_{\mathrm{H}}$ trajectory analysis over 300 puzzles. (a) Trajectories projected onto the first two principal components. PC1 explains $70.6\%$ of the variance and PC2 explains $17.7\%$ ; the trajectories are predominantly linear and PC1-dominated, in contrast to the convergence-versus-wandering pattern observed on Sudoku. Dots denote step 0 and crosses denote step 15. (b) Per-step path metrics: valid start–goal paths increase from $24.7\%$ at step 0 to $88.7\%$ after the first recurrent update and subsequently plateau near $93\%$ , quantifying the early saturation in Maze.

Figure 13: Representative linear-probe sweeps for the three tasks. Rows correspond to

z_{\mathrm{H}}

and

z_{\mathrm{L}}

. Sudoku probes report given-cell identity, per-cell correctness, and the mean of row-, column-, and box-violation probes. Maze probes report validation accuracy for four global path targets. ARC-AGI-2 shows output-grid membership, per-cell correctness, and color change. Task variables are readable from both

z_{\mathrm{H}}

and

z_{\mathrm{L}}

, even when the causal interventions in Finding 4 assign different roles to the two states.

Target	$z_{\mathrm{H}}^{(0)}$	$z_{\mathrm{H}}^{(15)}$	$z_{\mathrm{L}}^{(0)}$	$z_{\mathrm{L}}^{(15)}$
cell_digit	$98.5\pm 0.1$	$98.9\pm 0.1$	$63.6\pm 0.4$	$90.9\pm 0.5$
is_given	$99.6\pm 0.1$	$97.2\pm 0.5$	$100.0\pm 0.0$	$99.9\pm 0.0$
per_cell_correct	$71.3\pm 0.3$	$82.5\pm 1.2$	$67.7\pm 0.6$	$83.3\pm 1.5$
violated_in_row	$74.1\pm 0.6$	$88.3\pm 0.6$	$71.4\pm 0.8$	$88.2\pm 0.7$
violated_in_col	$74.1\pm 0.6$	$87.8\pm 0.6$	$70.9\pm 0.6$	$87.6\pm 0.8$
violated_in_box	$74.0\pm 0.5$	$87.5\pm 0.7$	$70.3\pm 0.2$	$87.1\pm 0.8$

Appendix

Table 9: Complete Sudoku linear-probe endpoint results. Values are validation accuracy (%), reported as mean

\pm

standard deviation over five seeds. Each probe uses

32{,}400

training cells and

8{,}100

validation cells.

Global path targets
Target	$z_{\mathrm{H}}^{(0)}$	$z_{\mathrm{H}}^{(15)}$	$z_{\mathrm{L}}^{(0)}$	$z_{\mathrm{L}}^{(15)}$
exact_solved	$85.7\pm 3.3$	$77.5\pm 1.5$	$76.1\pm 3.1$	$75.8\pm 3.7$
connects_start_goal	$78.6\pm 3.9$	$94.5\pm 2.6$	$61.2\pm 3.2$	$92.3\pm 3.0$
valid_sg_path	$84.6\pm 2.2$	$95.7\pm 2.1$	$68.3\pm 4.6$	$92.1\pm 1.7$
valid_optimal_path	$83.9\pm 3.4$	$86.6\pm 1.8$	$72.5\pm 2.6$	$84.9\pm 1.7$
path_f1	n/r	n/r	n/r	n/r

Appendix

Table 10: Complete Maze linear-probe endpoint results. Categorical targets report validation accuracy (%) as mean

\pm

standard deviation over five puzzle-disjoint seeds. Global probes use

800/200

train/test mazes; local probes use

102{,}400/25{,}600

train/test cells. n/r denotes continuous targets whose

R^{2}

is explicitly marked unreliable because the evaluation target is nearly constant.

Per-cell targets
Target	$z_{\mathrm{H}}^{(0)}$	$z_{\mathrm{H}}^{(15)}$	$z_{\mathrm{L}}^{(0)}$	$z_{\mathrm{L}}^{(15)}$
per_cell_correct	$90.2\pm 0.6$	$91.6\pm 0.5$	$92.6\pm 0.4$	$97.7\pm 0.3$
input_inside_grid	$99.9\pm 0.1$	$99.7\pm 0.2$	$100.0\pm 0.0$	$99.9\pm 0.0$
output_inside_grid	$99.9\pm 0.1$	$99.8\pm 0.1$	$100.0\pm 0.0$	$99.9\pm 0.0$
colour_changed	$96.5\pm 0.2$	$97.2\pm 0.3$	$98.6\pm 0.1$	$99.7\pm 0.1$
same_as_input	$96.0\pm 0.2$	$97.6\pm 0.3$	$98.7\pm 0.0$	$99.7\pm 0.1$

Appendix

Table 11: Complete ARC-AGI-2 linear-probe endpoint results. Reportable categorical targets give validation accuracy (%) as mean

\pm

standard deviation over five seeds, except shape_correct at step 15 (

n=4

valid seeds). Dashes denote one-class step-0 targets for which a classifier cannot be fit. n/r denotes near-constant continuous targets with unreliable

R^{2}

Figure 14: MLP-minus-linear probe gains on Maze at step 15. Nonlinear probes provide modest gains for

z_{\mathrm{H}}

and larger gains for several

z_{\mathrm{L}}

path targets, particularly start–goal connectivity and valid-path status. These gains indicate nonlinear accessibility rather than causal relevance.

Figure 15: MLP-minus-linear probe gains on Sudoku at step 15. No target gains more than

1.9

points from a nonlinear readout, and the mean gain is

+0.68

for

z_{\mathrm{H}}

and

-0.08

for

z_{\mathrm{L}}

. The Sudoku constraint variables are therefore predominantly linearly accessible from both streams.

		Step
		$\mathbf{0}$	$\mathbf{4}$	$\mathbf{8}$	$\mathbf{12}$	$\mathbf{15}$
$z_{\mathrm{H}}$	V–V	$+0.921$	$+0.947$	$+0.953$	$+0.953$	$+0.953$
$z_{\mathrm{H}}$	C–V	$-0.485$	$-0.369$	$-0.346$	$-0.342$	$-0.330$
$z_{\mathrm{H}}$	PC1 (%)	$93.3$	$95.0$	$95.5$	$95.5$	$95.4$
$z_{\mathrm{H}}$	Rel.	$0.74$	$0.73$	$0.72$	$0.71$	$0.71$
$z_{\mathrm{L}}$	V–V	$+0.966$	$+0.964$	$+0.970$	$+0.967$	$+0.969$

Appendix

Table 12: Sudoku probe geometry across recurrent steps. V-V is the mean cosine between violation directions; C-V is the mean cosine between correctness and violation directions; PC1 is computed from the four centered, unit-normalized directions; Rel. is mean same-target agreement across seeds. Values are means over five seeds. Random-direction nulls:

|\cos|=0.035

and PC1

=35.9\%

	$z_{\mathrm{H}}$		$z_{\mathrm{L}}$
Target	Acc. (%)	Rel.	Acc. (%)	Rel.
is_wall	$100.0$	$0.79$	$100.0$	$0.84$
is_free	$100.0$	$0.76$	$100.0$	$0.86$
on_optimal_path	$98.6$	$0.77$	$98.9$	$0.84$
off_path_passable	$98.5$	$0.47$	$98.4$	$0.85$
is_corner_on_path	$81.0$	$0.19$	$88.2$	$0.81$

Appendix

Table 13: Maze probe geometry at step 15. Panel A reports validation accuracy from the geometry refits and same-target seed reliability. Panel B reports pairwise cosines among the four stable directions and PC1 for the stable set and all nine directions. Values are means over five seeds.

	$z_{\mathrm{H}}$		$z_{\mathrm{L}}$
Direction pair	step 0	step 15	step 0	step 15
changed–copied	$-0.873$	$-0.780$	$-0.916$	$-0.883$
correct–copied	$+0.573$	$+0.328$	$+0.274$	$+0.020$
correct–changed	$-0.551$	$-0.139$	$-0.234$	$+0.016$
changed–boundary	$+0.386$	$+0.256$	$+0.079$	$+0.057$
correct–in-grid	$+0.028$	$+0.257$	$+0.143$	$+0.257$

Appendix

Table 14: ARC-AGI-2 geometry for six probe directions. Changed, copied, correct, boundary, in-grid, and eos denote colour_changed , same_as_input , per_cell_correct , is_object_boundary , input_inside_grid , and is_eos .

Task	$n$	Top-50	Rand.-50	$p$
Sudoku	300	$-3.63$	$-3.42$	0.22
Maze	300	$-2.99$	$-3.35$	0.18
ARC-AGI-2	150	$+1.27$	$+1.28$	0.94

Appendix

Table 15: SAE causal effects used in Figure 5 (b). Mean metric change on the primary metric of each task (i.e., cell accuracy for Sudoku, valid path rate for Maze, color-cell accuracy for ARC-AGI-2).

p

tests top-50

=

random-50.

$k$	Top- $k$	Random- $k$	Top–Random	$p$
16	$-3.08$	$-2.13$	$-0.95$	$2.9\times 10^{-4}$
32	$-12.12$	$-12.37$	$+0.25$	0.45
50	$-3.63$	$-3.42$	$-0.21$	0.23
64	$-3.92$	$-4.44$	$+0.52$	0.085

Appendix

Table 16: Sudoku top-

k

SAE sensitivity. Values are mean

\Delta

cell accuracy (%).

Feat.	Best target	Peak step	$r_{\text{best}}$	$r_{\text{next}}$
263	box violation	0	$+0.41$	0.40
597	is-given	13	$+0.39$	0.33
1809	is-given	–	$+0.39$	0.21
1542	per-cell correct	0	$-0.37$	0.34
654	is-given	–	$+0.35$	0.31
211	is-given	2	$-0.35$	0.33

Appendix

Table 17: Most-specialized Sudoku SAE features (

D=2048

\lambda=0.01

r_{\text{best}}

is the Pearson correlation between the feature and the best target;

r_{\text{next}}

is the largest absolute correlation between the feature and any other target.

Model	Top-50	Random-50	paired $p$	Cohen’s $d$
Stock HRM	$-1.59$	$-1.37$	0.39	$-0.06$
BPTT HRM	$-29.64$	$-29.63$	0.93	$-0.01$

Appendix

Table 18: BPTT SAE causal-ablation details. Values are mean

\Delta

cell accuracy (%).

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 926 papers

Transformers Linearly Represent Highly Structured World Models
May 13, 2026Roman Kniazev, Nathanaël FijalkowCircuit Discovery World Models
LaBRI, CNRS · University of Bordeaux · France
One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
May 18, 2026Jucheng Shen, Barbara Su, Anastasios KyrillidisTransformer Interpretability Recurrent Transformers
Rice University
Form Follows Function: Recursive Stem Model
Mar 3, 2026Navid HakimiRecursive Reasoning Models Neural Algorithmic Reasoning

May 13, 2026J/K move · Enter open · S save

Transformers Linearly Represent Highly Structured World Models

Roman Kniazev, Nathanaël Fijalkow

LaBRI, CNRS · University of Bordeaux · France

Circuit Discovery World Models Mechanistic Interpretability Structured Reasoning Constraint Satisfaction Algorithmic Reasoning

Open paper AskPDF arXiv

Do transformers, when trained on sequential reasoning traces, build internal models of the underlying task? And if so, does the structure of those internal representations mirror the structure of the domain? We train an 8-layer transformer on Sudoku solving traces and perform a mechanistic analysis of its internal computation. We establish two results. First, the model builds a substructure world model: it does not represent the board state cell by cell, as a human analyst would expect, but organizes information around the rows, columns, and boxes that Sudoku's constraints act on. Second, we identify a naked-single circuit: a small set of dedicated neurons in the final MLP layer, each individually detecting when exactly one digit remains possible for a specific cell, and reliably promoting that digit. These findings show that the geometry of an emergent world model is shaped by the constraint algebra of the domain, not its surface presentation, and that the resulting decision circuit is sparse, monosemantic, and fully interpretable. More broadly, they demonstrate that mechanistic interpretability tools can recover an end-to-end algorithmic account of how a transformer solves a combinatorial reasoning task.

Abstract

LLM Reasoning

Explore similar work

Transformers Linearly Represent Highly Structured World Models

One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer

Form Follows Function: Recursive Stem Model