Structured Sparse Memory for Recurrent Reasoning

Zixuan Zhao; Samuel Wheeler; Neil Getty; Xiaotian Duan; Rick Stevens; Fangfang Xia

cs.AISep 27, 2026

Structured Sparse Memory for Recurrent Reasoning

Authors: Zixuan Zhao, Samuel Wheeler, Neil Getty, Xiaotian Duan, Rick Stevens, Fangfang Xia

Organizations: Department of Computer Science University of Chicago Chicago, IL 60637 · Argonne National Laboratory Lemont, IL 60439

Abstract

Recurrent models trained from scratch have recently become competitive on ARC-style reasoning tasks, but the usual framing around small recurrent backbones overlooks two important parts of the system: task-conditioned memory and synthetic augmentation data. We study this regime through CHARM, a compact hybrid ARC model that combines recurrent reasoning with structured task memory, synthetic data, and inference-time aggregation. In existing approaches, task-conditioned memory supplies a large hidden source of capacity, reaching more than 30x the size of the recurrent backbone. We introduce a compositional sparse embedding (CoSE) for task conditioning that reduces learned task-memory parameters by over 90% while improving pass@2 in controlled ARC ablations. For the recurrent backbone, recurrent depth helps only when balanced with learning horizon. Combining these ingredients, our system reaches 84% pass@2 on ARC-AGI-1 and 46.7% pass@2 on ARC-AGI-2 public evaluation. The benefits of structured memory also generalize to unseen puzzles and other domains. Our code, dataset, and model checkpoints are available at https://github.com/water-vapor/charm.

Figures & tables

Benchmark	Synthetic data	Full task table		CoSE
Benchmark	Synthetic data	pass@2	Params	pass@2	Params
ARC-AGI-1	N/A	$59.97\scriptstyle\pm 1.77$	462M	$64.91\scriptstyle\pm 2.25$	43M
ARC-AGI-1	Re-ARC	$78.50\scriptstyle\pm 1.03$	463M	$\mathbf{80.71}\scriptstyle\pm 0.51$	43M
ARC-AGI-2	N/A	$15.42\scriptstyle\pm 1.51$	623M	$18.61\scriptstyle\pm 1.63$	53M
ARC-AGI-2	Re-ARC2	$28.23\scriptstyle\pm 3.58$	674M	$\mathbf{37.22}\scriptstyle\pm 1.77$	57M

Table 1: Major contributors to CHARM performance. We report public-evaluation pass@2 accuracy and total learned parameters varying task memory and synthetic data under the default evaluator. Values are percentage means

\pm

sample standard deviations over 3–4 runs. Training budgets are 600k steps for ARC-AGI-1 and 1M for ARC-AGI-2.

Embedding setup	Color permutation	pass@2	pass@1000
Combinatorial task table	Unrestricted	$49.75\scriptstyle\pm 2.50$	$71.75\scriptstyle\pm 0.22$
Combinatorial task table	1000-pooled	52.08	71.25
3 separate tables	Unrestricted	52.33	71.00
3 separate tables	1000-pooled	$45.52\scriptstyle\pm 0.80$	$62.19\scriptstyle\pm 0.97$

Table 2: Color-permutation pooling isolates the capacity effect in task embeddings. A fixed pool of 1,000 color permutations leaves the performance of the full combinatorial table nearly unchanged but sharply hurts separated puzzle, dihedral, and color-permutation embeddings. 2 2 2 Models are trained on the ARC-AGI-1 training set for 170k steps. Reported standard deviations are specific to this short-training split setting. 600k-step public-evaluation runs are substantially more stable.

Embedding setup	pass@2	pass@1000
Stock TRM full task table ( $r=512$ )	$49.75\scriptstyle\pm 2.50$	$71.75\scriptstyle\pm 0.22$
Low-rank task table ( $r=128$ )	47.71	69.38
Low-rank task table ( $r=64$ )	46.04	66.00
Low-rank task table ( $r=32$ )	45.46	67.50
Low-rank task table ( $r=8$ )	$38.29\scriptstyle\pm 0.95$	$55.92\scriptstyle\pm 1.35$

Table 3: Standalone low-rank task tables lose accuracy as row width shrinks. The stock row uses the full-width TRM task table with

r=512

; the remaining rows replace it with a narrower per-instance table without the compositional branch. 2 2 footnotemark: 2

Setup	Embedding params	pass@1	pass@2	pass@1000
Full task table	448.9M	$73.25\scriptstyle\pm 0.74$	$78.50\scriptstyle\pm 1.03$	$91.38\scriptstyle\pm 0.65$
Rank-32 task table only	28.1M	70.62	76.62	88.88
Compositional branch only	1.5M	73.50	79.25	89.00
CoSE (rank-32 residual)	29.6M	$74.29\scriptstyle\pm 0.75$	$80.71\scriptstyle\pm 0.51$	$90.62\scriptstyle\pm 0.50$
CoSE (gated $r{=}32$ )	29.9M	$74.59\scriptstyle\pm 0.74$	$80.00\scriptstyle\pm 0.70$	$90.94\scriptstyle\pm 1.10$
CoSE (rank-512 residual)	450.4M	77.75	81.50	92.12

Table 4: Improving the task-memory accuracy–parameter tradeoff. CoSE uses

1/15

as many task-memory parameters as the full table while improving pass@1 and pass@2. All rows use the same recurrent backbone, training data, and optimization setup, varying only the task-memory module. While the low-rank residual and compositional branch are less effective, their combination yields superior performance. Repeated rows report mean

\pm

sample standard deviation; the remaining architectural variants are single runs.

Method	data	Backbone	ARC-1 all	pass@1	pass@2	pass@1	pass@2
	Synthetic	Parameters		ARC-AGI-1		ARC-AGI-2
HRM [ 1 ]	N/A	27M	475M	34.4	40.3	3.8	5.0
TRM [ 2 ]	N/A	7M	455M	40.0	44.6	4.6	7.8
URM [ 3 ]	N/A	14M	462M	53.8	61.8	16.0	18.2
VARC [ 4 ]	Re-ARC	73M	73M	55.1	60.4	9.4	11.1
CHARM	Re-ARC(2) 3 3 3 CHARM uses Re-ARC on ARC-AGI-1, as in VARC; and our synthetic Re-ARC2 dataset (Section 4.1 ) for ARC-AGI-2.	14M	44M	78.6	84.0	39.3	46.7

Table 5: Public-evaluation accuracy of ARC systems trained from scratch.

Synthetic data	Task embedding	pass@1	pass@2	pass@1000
None (baseline)	full table	$54.16\scriptstyle\pm 1.91$	$59.97\scriptstyle\pm 1.77$	$80.41\scriptstyle\pm 1.67$
Re-ARC (aug=10)	full table	68.0	72.5	84.63
Re-ARC (aug=10)	CoSE	70.88	76.38	86.25
Re-ARC (aug=100)	CoSE	$\mathbf{74.29}\scriptstyle\pm 0.75$	$\mathbf{80.71}\scriptstyle\pm 0.51$	$\mathbf{90.62}\scriptstyle\pm 0.50$

Table 6: Interaction between synthetic data and task memory on ARC-AGI-1. Rows use the same backbone, training schedule, and default evaluator. Re-ARC gives large gains over the no-synthetic baseline, and CoSE retains those gains with compact task memory.

Synthetic data	Task embedding	pass@1	pass@2	pass@1000
None (baseline)	full table	$13.16\scriptstyle\pm 0.65$	$15.42\scriptstyle\pm 1.51$	$30.00\scriptstyle\pm 2.91$
None	CoSE	$13.96\scriptstyle\pm 1.12$	$18.61\scriptstyle\pm 1.63$	$31.22\scriptstyle\pm 2.90$
Re-ARC (ARC-AGI-1) (aug=100)	CoSE	23.89	28.47	46.39
Synthetic Re-ARC2 (aug=10)	CoSE	30.69	35.97	45.97
Synthetic Re-ARC2 (aug=100)	full table	$23.61\scriptstyle\pm 3.52$	$28.23\scriptstyle\pm 3.58$	$49.58\scriptstyle\pm 2.59$
Synthetic Re-ARC2 (aug=100)	CoSE	$\mathbf{30.97}\scriptstyle\pm 1.52$	$\mathbf{37.22}\scriptstyle\pm 1.77$	$\mathbf{54.48}\scriptstyle\pm 1.89$

Table 7: Matched synthetic data is the largest ARC-AGI-2 driver. Rows use the same backbone, optimizer, training schedule, and default evaluator. ARC-AGI-1 Re-ARC tests cross-benchmark transfer; Re-ARC2 tests task-matched augmentation at increasing scale.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Model	Rows	Task memory	Total
ARC-AGI-1	URM, full table	0.876M	448.3M	462.0M
ARC-AGI-1 + Re-ARC	URM, full table	0.877M	448.9M	462.5M
ARC-AGI-1 + Re-ARC	CHARM ( $r{=}32$ + gate)	0.877M	29.9M	43.5M
ARC-AGI-2	URM, full table	1.191M	609.6M	623.3M
ARC-AGI-2	CHARM ( $r{=}32$ )	1.191M	39.8M	53.5M
ARC-AGI-2 + Re-ARC2	CHARM ( $r{=}32$ )	1.289M	43.0M	56.6M

Appendix

Table 8: Task-memory footprint in learnable parameters. CoSE reduces learned task-memory parameters by 94% across ARC-AGI dataset setups. Rows counts combinatorial puzzle-augmentation instances. Task memory is the full task table for stock models and the compositional branch plus low-rank residual table for CoSE .

Component	Count	Parameters
URM backbone	measured	$13{,}663{,}233$
Full instance table	$Md$	$448{,}872{,}960$
CoSE compositional branch	$(P+9)d+2(2d^{2}+d)$	$1{,}545{,}728$
CoSE residual table	$Mr$	$28{,}054{,}560$
CoSE residual up-projection	$rd$	$16{,}384$
CoSE optional gate	$d^{2}+d$	$262{,}656$

Appendix

Table 9: Component-level parameter accounting for ARC-AGI-1 with Re-ARC augmentation. The table reports the measured recurrent backbone and the analytic counts for the full task table, CoSE compositional branch, rank-32 residual table, up-projection, and optional gate.

Design	Data	Task table	Reasoning	Compositional	Table params	Total
TRM task table	ARC-AGI-1	$875{,}617{\times}512$	$6{,}830{,}082$	$0$	$448{,}315{,}904$	$455{,}145{,}986$
URM task table	ARC-AGI-1	$875{,}617{\times}512$	$13{,}663{,}746$	$0$	$448{,}315{,}904$	$461{,}979{,}650$
CoSE composition only	ARC-AGI-1 + Re-ARC	none	$13{,}663{,}233$	$1{,}545{,}728$	$0$	$15{,}208{,}961$
URM task table	ARC-AGI-1 + Re-ARC	$876{,}705{\times}512$	$13{,}663{,}233$	$0$	$448{,}872{,}960$	$462{,}536{,}193$
Rank-32 table only	ARC-AGI-1 + Re-ARC	$876{,}705{\times}32$	$13{,}663{,}233$	$16{,}384$	$28{,}054{,}560$	$41{,}734{,}177$
CoSE + rank-32 table	ARC-AGI-1 + Re-ARC	$876{,}705{\times}32$	$13{,}663{,}233$	$1{,}562{,}112$	$28{,}054{,}560$	$43{,}279{,}905$

Appendix

Table 10: Exact ARC-AGI-1 parameter counts across task-memory designs. The table reports recurrent-backbone, compositional, per-instance table, and total parameters for each design. The task-table column gives rows

\times

width when a learned per-instance table is present; width 512 is the stock task embedding, and width 32 is the low-rank residual used by CoSE .

Design	Data	Task table	Reasoning	Compositional	Table params	Total
URM task table	ARC-AGI-2	$1{,}190{,}624{\times}512$	$13{,}663{,}746$	$0$	$609{,}599{,}488$	$623{,}263{,}234$
CoSE + rank-32 table	ARC-AGI-2	$1{,}190{,}624{\times}32$	$13{,}663{,}233$	$1{,}725{,}952$	$38{,}099{,}968$	$53{,}489{,}153$
CoSE + rank-32 table	ARC-AGI-2 + Re-ARC2	$1{,}289{,}151{\times}32$	$13{,}663{,}233$	$1{,}725{,}952$	$41{,}252{,}832$	$56{,}642{,}017$

Appendix

Table 11: Exact ARC-AGI-2 parameter counts under the same accounting as Table 10 . The comparison shows how the full task table and CoSE scale with the larger ARC-AGI-2 task set and with synthetic Re-ARC2 augmentation.

Benchmark	Synthetic data	Task memory	$n$	pass@1	pass@2	pass@1000
ARC-AGI-1	None	Full task table	4	$54.16\scriptstyle\pm 1.91$	$59.97\scriptstyle\pm 1.77$	$80.41\scriptstyle\pm 1.67$
		CoSE (ungated)	3	$56.46\scriptstyle\pm 1.51$	$62.67\scriptstyle\pm 2.43$	$79.96\scriptstyle\pm 1.81$
		CoSE (gated)	4	$\mathbf{57.41}\scriptstyle\pm 1.82$	$\mathbf{64.91}\scriptstyle\pm 2.25$	$\mathbf{81.41}\scriptstyle\pm 1.65$
	Re-ARC	Full task table	4	$73.25\scriptstyle\pm 0.74$	$78.50\scriptstyle\pm 1.03$	$\mathbf{91.38}\scriptstyle\pm 0.65$
		CoSE (ungated)	3	$74.29\scriptstyle\pm 0.75$	$\mathbf{80.71}\scriptstyle\pm 0.51$	$90.62\scriptstyle\pm 0.50$
		CoSE (gated)	4	$\mathbf{74.59}\scriptstyle\pm 0.74$	$80.00\scriptstyle\pm 0.70$	$90.94\scriptstyle\pm 1.10$

Appendix

Table 12: Complete component ablation with repeated runs. The comparison studies benchmark, matched synthetic data, and task-memory design. Bold values are the highest means within each benchmark–data block.

Setup	Embedding params	pass@1	pass@2	pass@1000
Full task table	448.3M	$54.16\scriptstyle\pm 1.91$	$59.97\scriptstyle\pm 1.77$	$80.41\scriptstyle\pm 1.67$
Rank-32 task table only	28.0M	52.88	58.00	79.25
Compositional branch only	1.5M	47.38	51.50	70.12
CoSE (rank-32 residual)	29.6M	$56.46\scriptstyle\pm 1.51$	$62.67\scriptstyle\pm 2.43$	$79.96\scriptstyle\pm 1.81$
CoSE (gated $r{=}32$ )	29.8M	$\mathbf{57.41}\scriptstyle\pm 1.82$	$\mathbf{64.91}\scriptstyle\pm 2.25$	$\mathbf{81.41}\scriptstyle\pm 1.65$

Appendix

Table 13: Improving the task-memory accuracy–parameter tradeoff without Re-ARC dataset. This table mirrors Table 4 , but omits Re-ARC from the ARC-AGI-1 training data, matching the original XRM setups. The recurrent backbone, training data, and optimization setup are fixed, varying only the task-memory module. All models are trained for at most 600k steps.

Dihedral Embedding	Color Embedding	Combination method	pass@2	pass@1000
Full task table	Full task table	Lookup	$49.75\scriptstyle\pm 2.50$	$71.75\scriptstyle\pm 0.22$
Lookup table	Slotperm	FiLM-concat	$47.50\scriptstyle\pm 1.16$	$63.25\scriptstyle\pm 2.01$
Slotperm	Slotperm	FiLM-concat	$46.77\scriptstyle\pm 0.27$	$64.50\scriptstyle\pm 1.59$
Lookup table	Slotperm	Hypernetwork	46.46	59.38
Lookup table	Slotperm	More expressive FiLM	$46.09\scriptstyle\pm 2.30$	$62.00\scriptstyle\pm 0.88$
Lookup table	Slotperm	FiLM-concat, larger backbone	$45.58\scriptstyle\pm 0.71$	$64.13\scriptstyle\pm 0.35$

Appendix

Table 14: Pure compositional task conditioning does not fully recover the instance-specific capacity of the task table. On ARC-AGI-1 training tasks, several operators for combining puzzle, dihedral, and color information improve over simple addition but remain below the full lookup baseline. This persistent gap motivates adding a small per-instance residual in CoSE . Values are percentage accuracies, reported as mean

\pm

std when multiple recovered runs are available.

Dihedral Embedding	Color Embedding	Combination/residual method	pass@2	pass@1000
Full task table	Full task table	Lookup	$49.75\scriptstyle\pm 2.50$	$71.75\scriptstyle\pm 0.22$
Slotperm	Slotperm	FiLM-concat + gated residual ( $r=32$ )	$51.83\scriptstyle\pm 0.76$	$68.96\scriptstyle\pm 1.48$
Slotperm	Slotperm	FiLM-concat + residual ( $r=32$ )	$51.25\scriptstyle\pm 0.94$	$67.19\scriptstyle\pm 0.80$
Slotperm	Slotperm	FiLM-concat + Engram residual ( $r=32$ )	50.08	66.88

Appendix

Table 15: Residual-memory design ablation for compositional task conditioning on ARC-AGI-1 training tasks. Starting from the compositional setup in Table 14 , we compare gated, ungated, and Engram-style rank-32 residuals under the same TRM backbone and training augmentation set. The full task-table row is included as the lookup baseline.

Task memory	Total params	Memory params	pass@1	pass@2	pass@1	pass@2
			Seen		Unseen
Full table	4.783M	3.072M	$50.78\scriptstyle\pm 1.56$	$53.94\scriptstyle\pm 1.62$	$0.00\scriptstyle\pm 0.00$	$0.00\scriptstyle\pm 0.00$
Low-rank table	1.906M	0.196M	$51.43\scriptstyle\pm 1.81$	$53.49\scriptstyle\pm 1.76$	$0.05\scriptstyle\pm 0.12$	$0.07\scriptstyle\pm 0.19$
Three-factor composition	2.229M	0.519M	$56.48\scriptstyle\pm 2.16$	$57.89\scriptstyle\pm 2.03$	$\mathbf{2.54}\scriptstyle\pm 0.52$	$\mathbf{3.69}\scriptstyle\pm 0.56$
Three-factor CoSE	2.425M	0.715M	$\mathbf{57.40}\scriptstyle\pm 0.97$	$\mathbf{58.86}\scriptstyle\pm 0.90$	$1.80\scriptstyle\pm 0.37$	$2.73\scriptstyle\pm 0.48$

Appendix

Table 16: CAR single-horizon generalization at 500k steps. Models train on one horizon for each of 1,500 rules and evaluate both those seen combinations and the other 4,500 rule–horizon combinations. Parameters include the 1.710M shared backbone. Experiments use 7 repeats.

	Three-factor composition		Three-factor CoSE
$T_{\rm eval}$	pass@1	pass@2	pass@1	pass@2
1	$\mathbf{1.78}\scriptstyle\pm 0.75$	$\mathbf{2.87}\scriptstyle\pm 1.06$	$1.00\scriptstyle\pm 0.37$	$1.61\scriptstyle\pm 0.52$
2	$\mathbf{1.92}\scriptstyle\pm 0.83$	$\mathbf{2.95}\scriptstyle\pm 0.93$	$1.44\scriptstyle\pm 0.46$	$2.20\scriptstyle\pm 0.37$
4	$\mathbf{3.49}\scriptstyle\pm 0.64$	$\mathbf{4.79}\scriptstyle\pm 0.54$	$2.43\scriptstyle\pm 0.50$	$3.71\scriptstyle\pm 0.76$
8	$\mathbf{2.96}\scriptstyle\pm 0.57$	$\mathbf{4.15}\scriptstyle\pm 0.40$	$2.35\scriptstyle\pm 0.43$	$3.41\scriptstyle\pm 0.56$

Appendix

Table 17: CAR transfer by unseen evaluation horizon. Results are over held-out rule–horizon combinations; both experiments use 7 repeats.

Task memory	1.5k	6k	pass@1	pass@2	pass@1	pass@2
	Memory params		Single-horizon		All-horizons
Full table	3.072M	12.288M	$45.20\scriptstyle\pm 1.78$	$48.24\scriptstyle\pm 2.07$	$43.66\scriptstyle\pm 1.32$	$47.21\scriptstyle\pm 1.72$
Low-rank table	0.196M	0.772M	$45.09\scriptstyle\pm 1.46$	$47.19\scriptstyle\pm 1.52$	$43.09\scriptstyle\pm 0.31$	$45.61\scriptstyle\pm 0.33$
Two-factor composition	0.518M	1.670M	$52.41\scriptstyle\pm 1.65$	$53.98\scriptstyle\pm 1.56$	$48.20\scriptstyle\pm 1.04$	$50.13\scriptstyle\pm 1.06$
Two-factor CoSE	0.714M	2.442M	$51.97\scriptstyle\pm 2.36$	$53.56\scriptstyle\pm 2.21$	$47.53\scriptstyle\pm 1.81$	$49.38\scriptstyle\pm 1.81$
Three-factor composition	0.519M	0.519M	$50.26\scriptstyle\pm 2.38$	$51.83\scriptstyle\pm 2.33$	$49.43\scriptstyle\pm 1.82$	$51.39\scriptstyle\pm 2.02$

Appendix

Table 18: Matched-budget CAR comparison at 280k steps. The single-horizon dataset has 1,500 combinations; the all-horizons dataset has 6,000. Experiments use 7 repeats.

Method	pass@1	pass@2	pass@1	pass@2	pass@1	pass@2	pass@1	pass@2
	$T{=}1$		$T{=}2$		$T{=}4$		$T{=}8$
CAR, three-factor CoSE	$\mathbf{98.70}$	$\mathbf{99.25}$	$\mathbf{88.84}$	$\mathbf{90.27}$	31.56	34.15	10.50	11.77
GPT-5.6, no tools	48.0	66.0	8.5	11.0	3.0	5.0	5.0	8.0
GPT-5.6, Python	78.5	90.0	73.5	78.0	$\mathbf{66.5}$	$\mathbf{72.0}$	$\mathbf{66.0}$	$\mathbf{76.0}$

Appendix

Table 19: CAR accuracy by rollout horizon: three-factor CoSE vs. GPT-5.6. CAR values are means over seven replicas of the terminal cumulative evaluation on seen combinations; GPT-5.6 entries have no across-run error bars.

Capacity	Engram	CoSE	Engram	CoSE
	Memory params		Validation bpb
0	0	3.5M	$0.85365\scriptstyle\pm 0.00111$	$0.85363\scriptstyle\pm 0.00051$
$\times 5$	254M	258M	$0.84638\scriptstyle\pm 0.00050$	$\mathbf{0.84436}\scriptstyle\pm 0.00083$
$\times 10$	506M	510M	$0.84482\scriptstyle\pm 0.00057$	$\mathbf{0.84277}\scriptstyle\pm 0.00081$
$\times 20$	1.01B	1.01B	$0.84333\scriptstyle\pm 0.00046$	$\mathbf{0.84103}\scriptstyle\pm 0.00064$
$\times 40$	2.02B	2.02B	$0.84222\scriptstyle\pm 0.00062$	$\mathbf{0.83946}\scriptstyle\pm 0.00058$

Appendix

Table 20: CoSE improves n-gram memory across row capacities on nanochat d12. All nonzero rows use five seeds.

Width	Engram	CoSE	CoSE -Mix	Engram	CoSE	CoSE -Mix
	Memory params			Validation bpb
24	15.9M	19.4M	19.3M	$0.85067\scriptstyle\pm 0.00097$	$0.84937\scriptstyle\pm 0.00090$	$\mathbf{0.84854}\scriptstyle\pm 0.00042$
48	31.8M	35.3M	35.1M	$0.84878\scriptstyle\pm 0.00099$	$0.84773\scriptstyle\pm 0.00108$	$\mathbf{0.84695}\scriptstyle\pm 0.00047$
96	63.6M	67.1M	66.7M	$0.84685\scriptstyle\pm 0.00063$	$0.84569\scriptstyle\pm 0.00109$	$\mathbf{0.84504}\scriptstyle\pm 0.00032$

Appendix

Table 21: CoSE -Mixture performs best across lookup widths on nanochat d12. The row-capacity multiplier is fixed at

\times 5

; all cells use five seeds. Width is the lookup dimension per n-gram order.

Capacity	Engram	CoSE	Engram	CoSE
	Memory params		Validation bpb
$\times 5$	426M	436M	$0.73002\scriptstyle\pm 0.00031$	$\mathbf{0.72939}\scriptstyle\pm 0.00082$
$\times 10$	846M	856M	$0.72901\scriptstyle\pm 0.00045$	$\mathbf{0.72800}\scriptstyle\pm 0.00091$
$\times 20$	1.68B	1.69B	$0.72730\scriptstyle\pm 0.00019$	$\mathbf{0.72664}\scriptstyle\pm 0.00058$

Appendix

Table 22: Matched-capacity language-model results at d20. All experiments use five seeds.

Backbone	1.19B tokens	4.64B tokens
d12	$-0.0020\scriptstyle\pm 0.0005$ ( $n=5$ )	$-0.0002\scriptstyle\pm 0.0003$ ( $n=2$ )
d20	$-0.0018\scriptstyle\pm 0.0003$ ( $n=3$ )	$-0.0006\scriptstyle\pm 0.0008$ ( $n=5$ )

Appendix

Table 23: Token-budget controls at fixed

\times 5

memory capacity. Cells report paired CoSE minus Engram validation bpb; lower is better.

	ARC-1 accuracy	ARC-2 train zero-shot		ARC-2 eval zero-shot
Base task memory	pass@2	pass@1	pass@2	pass@1	pass@2
CoSE $(r=0,t=32)$	77.50	0.86	0.86	0.00	0.00
CoSE $(r=0)$	78.33	0.86	0.86	0.00	0.00
CoSE $(r=32,t=32)$	77.50	0.43	0.43	0.00	0.00
CoSE $(r=32)$	80.00	0.43	0.86	0.00	0.00
CoSE $(r=512)$	76.67	0.43	0.86	0.00	0.00

Appendix

Table 24: Base checkpoints and zero-shot starting points. ARC-1 uses the fixed 60-puzzle evaluation; stream columns report mean per-puzzle pass@1 and pass@2 before adaptation.

Base task memory	Task embedding row	Residual rows	Train params/task	Eval params/task
CoSE $(r=0,t=32)$	32	—	32	32
CoSE $(r=0)$	512	—	512	512
CoSE $(r=32,t=32)$	32	$32m$	31.28k	31.91k
CoSE $(r=32)$	512	$32m$	31.76k	32.39k
CoSE $(r=512)$	512	$512m$	500.44k	510.61k
Table $(r=32)$	—	$32m$	31.25k	31.88k

Appendix

Table 25: Parameters updated by memory-only adaptation. Each puzzle has

m

augmentation instances, averaging 976.42 on the training stream and 996.28 on public evaluation. Shared weights are fixed. Composition-only CoSE (

r=0

) trains a task embedding but no residual rows.

	Memory-only CL			Per-puzzle FT Oracle
Base task memory	Params/task	pass@1	pass@2	Params/task	pass@1	pass@2
ARC-AGI-2 training stream
CoSE $(r=0,t=32)$	32	$17.94\scriptstyle\pm 1.08$	$20.60\scriptstyle\pm 0.86$	14.733M	$42.93\scriptstyle\pm 1.22$	$48.70\scriptstyle\pm 1.85$
CoSE $(r=0)$	512	$36.05\scriptstyle\pm 1.15$	$39.73\scriptstyle\pm 1.48$	14.717M	$42.23\scriptstyle\pm 1.28$	$47.02\scriptstyle\pm 2.15$
CoSE $(r=32,t=32)$	31.28k	$22.07\scriptstyle\pm 1.22$	$27.42\scriptstyle\pm 0.48$	14.781M	$\mathbf{45.23}\scriptstyle\pm 0.83$	$49.25\scriptstyle\pm 1.20$
CoSE $(r=32)$	31.76k	$\mathbf{38.56}\scriptstyle\pm 0.70$	$\mathbf{43.58}\scriptstyle\pm 0.58$	14.765M	$41.93\scriptstyle\pm 0.44$	$46.27\scriptstyle\pm 1.32$

Appendix

Table 26: Held-out-task adaptation from ARC-AGI-1 base checkpoints. Each row uses the better halting-loss setting by mean final pass@2 within each protocol.

r=0

denotes composition only, and

t=512

unless specified. Counts are effective updated parameters per puzzle.

Base task memory	Params/task	$w_{h}$	pass@1	pass@2
ARC-AGI-2 training stream
CoSE $(r=0,t=32)$	63.27k	0.5	$30.58\scriptstyle\pm 1.61$	$36.78\scriptstyle\pm 0.70$
CoSE $(r=0)$	63.67k	0.5	$30.44\scriptstyle\pm 2.07$	$35.32\scriptstyle\pm 1.85$
CoSE $(r=32,t=32)$	94.58k	0.5	$33.95\scriptstyle\pm 1.20$	$39.94\scriptstyle\pm 0.34$
CoSE $(r=32)$	94.99k	0.5	$30.32\scriptstyle\pm 2.00$	$36.85\scriptstyle\pm 2.57$
CoSE $(r=512)$	563.60k	0.5	$33.87\scriptstyle\pm 2.07$	$39.77\scriptstyle\pm 0.71$

Appendix

Table 27: Joint fine-tuning has simultaneous access to all stream puzzles. Each row uses the auxiliary halting-loss weight

w_{h}\in\{0,0.5\}

that yields the higher mean final pass@2. Parameter counts include shared weights and all puzzle-private rows, divided by the number of puzzles.

Method	Scope	Params/task	Immediate	Final p@1	Final p@2	ARC-1 after
Per-puzzle FT Oracle	full, per-task	14.765M	$46.27\scriptstyle\pm 1.32$	$41.93\scriptstyle\pm 0.44$	$46.27\scriptstyle\pm 1.32$	—
Memory-only	private rows	31.76k	$42.94\scriptstyle\pm 1.47$	$38.65\scriptstyle\pm 0.98$	$43.37\scriptstyle\pm 0.48$	$80.00\scriptstyle\pm 0.00$
L2-SP	composition	1.102M	$45.40\scriptstyle\pm 1.68$	$38.87\scriptstyle\pm 0.86$	$44.13\scriptstyle\pm 0.70$	$78.33\scriptstyle\pm 0.00$
Rehearsal	composition	1.102M	$41.94\scriptstyle\pm 1.25$	$31.76\scriptstyle\pm 1.67$	$37.20\scriptstyle\pm 1.25$	$78.06\scriptstyle\pm 3.76$
EWC	composition	1.102M	$41.99\scriptstyle\pm 1.99$	$21.22\scriptstyle\pm 8.31$	$25.94\scriptstyle\pm 8.46$	$30.28\scriptstyle\pm 37.33$
Naive	composition	1.102M	$39.41\scriptstyle\pm 1.51$	$8.44\scriptstyle\pm 2.79$	$10.59\scriptstyle\pm 2.83$	$0.00\scriptstyle\pm 0.00$

Appendix

Table 28: Stability–plasticity comparison on the ARC-AGI-2 training stream. All methods use the CoSE

(r=32)

checkpoint. We select the better halting-loss setting by final pass@2.

		Training stream		Public-evaluation stream
Base task memory	Adaptation	$w_{h}=0.5$	$w_{h}=0$	$w_{h}=0.5$	$w_{h}=0$
CoSE $(r=0)$	Memory-only	0.1990	0.1650	0.2791	0.2451
CoSE $(r=0)$	Per-puzzle FT	0.0525	0.0259	0.0597	0.0281
CoSE $(r=32)$	Memory-only	0.1613	0.1266	0.2412	0.2021
CoSE $(r=32)$	Per-puzzle FT	0.0470	0.0237	0.0552	0.0261

Appendix

Table 29: Removing the auxiliary halting loss improves token fitting during adaptation. Entries report mean logged token losses over matched completed runs: five for memory-only adaptation and three for per-puzzle fine-tuning per setting. Token losses exclude the auxiliary halting loss.

	With halting loss ( $w_{h}=0.5$ )		Without halting loss ( $w_{h}=0$ )
Base task memory	pass@1	pass@2	pass@1	pass@2
CoSE $(r=0,t=32)$	$17.94\scriptstyle\pm 1.08$	$20.60\scriptstyle\pm 0.86$	$16.78\scriptstyle\pm 0.44$	$20.30\scriptstyle\pm 0.44$
CoSE $(r=0)$	$34.76\scriptstyle\pm 1.10$	$38.24\scriptstyle\pm 0.63$	$36.05\scriptstyle\pm 1.15$	$39.73\scriptstyle\pm 1.48$
CoSE $(r=32,t=32)$	$22.07\scriptstyle\pm 1.22$	$27.42\scriptstyle\pm 0.48$	$19.40\scriptstyle\pm 0.89$	$23.76\scriptstyle\pm 0.26$
CoSE $(r=32)$	$35.26\scriptstyle\pm 1.44$	$41.03\scriptstyle\pm 1.92$	$38.56\scriptstyle\pm 0.70$	$43.58\scriptstyle\pm 0.58$
CoSE $(r=512)$	$37.28\scriptstyle\pm 1.93$	$42.70\scriptstyle\pm 0.91$	$37.81\scriptstyle\pm 0.77$	$42.26\scriptstyle\pm 1.31$

Appendix

Table 30: Memory-only adaptation with and without halting loss on ARC-AGI-2 training puzzles. Scores are means

\pm

sample standard deviations over completed runs.

	With halting loss ( $w_{h}=0.5$ )		Without halting loss ( $w_{h}=0$ )
Base task memory	pass@1	pass@2	pass@1	pass@2
CoSE $(r=0,t=32)$	$37.09\scriptstyle\pm 2.03$	$40.78\scriptstyle\pm 2.70$	$42.93\scriptstyle\pm 1.22$	$48.70\scriptstyle\pm 1.85$
CoSE $(r=0)$	$34.80\scriptstyle\pm 0.95$	$39.18\scriptstyle\pm 2.02$	$42.23\scriptstyle\pm 1.28$	$47.02\scriptstyle\pm 2.15$
CoSE $(r=32,t=32)$	$40.50\scriptstyle\pm 1.18$	$45.65\scriptstyle\pm 0.30$	$45.23\scriptstyle\pm 0.83$	$49.25\scriptstyle\pm 1.20$
CoSE $(r=32)$	$36.21\scriptstyle\pm 1.91$	$42.54\scriptstyle\pm 2.15$	$41.93\scriptstyle\pm 0.44$	$46.27\scriptstyle\pm 1.32$
CoSE $(r=512)$	$37.71\scriptstyle\pm 0.53$	$41.36\scriptstyle\pm 0.53$	$43.35\scriptstyle\pm 1.59$	$48.51\scriptstyle\pm 2.02$

Appendix

Table 31: Per-puzzle FT Oracle with and without halting loss on ARC-AGI-2 training puzzles. Scores are means

\pm

sample standard deviations over completed runs.

	With halting loss ( $w_{h}=0.5$ )		Without halting loss ( $w_{h}=0$ )
Base task memory	pass@1	pass@2	pass@1	pass@2
CoSE $(r=0,t=32)$	$30.58\scriptstyle\pm 1.61$	$36.78\scriptstyle\pm 0.70$	$26.02\scriptstyle\pm 2.00$	$31.88\scriptstyle\pm 0.58$
CoSE $(r=0)$	$30.44\scriptstyle\pm 2.07$	$35.32\scriptstyle\pm 1.85$	$26.31\scriptstyle\pm 1.85$	$31.35\scriptstyle\pm 2.60$
CoSE $(r=32,t=32)$	$33.95\scriptstyle\pm 1.20$	$39.94\scriptstyle\pm 0.34$	$29.08\scriptstyle\pm 0.23$	$32.83\scriptstyle\pm 0.45$
CoSE $(r=32)$	$30.32\scriptstyle\pm 2.00$	$36.85\scriptstyle\pm 2.57$	$28.06\scriptstyle\pm 3.98$	$33.76\scriptstyle\pm 2.86$
CoSE $(r=512)$	$33.87\scriptstyle\pm 2.07$	$39.77\scriptstyle\pm 0.71$	$25.83\scriptstyle\pm 3.43$	$31.59\scriptstyle\pm 4.89$

Appendix

Table 32: Joint fine-tuning with and without halting loss on ARC-AGI-2 training puzzles. Scores are means

\pm

sample standard deviations.

	With halting loss ( $w_{h}=0.5$ )		Without halting loss ( $w_{h}=0$ )
Base task memory	pass@1	pass@2	pass@1	pass@2
CoSE $(r=0,t=32)$	$1.75\scriptstyle\pm 0.54$	$2.54\scriptstyle\pm 0.57$	$3.86\scriptstyle\pm 0.84$	$5.18\scriptstyle\pm 0.78$
CoSE $(r=0)$	$9.30\scriptstyle\pm 0.48$	$9.65\scriptstyle\pm 0.54$	$9.21\scriptstyle\pm 1.12$	$10.76\scriptstyle\pm 0.96$
CoSE $(r=32,t=32)$	$5.88\scriptstyle\pm 0.39$	$6.58\scriptstyle\pm 0.31$	$5.88\scriptstyle\pm 0.59$	$6.14\scriptstyle\pm 0.54$
CoSE $(r=32)$	$8.74\scriptstyle\pm 1.97$	$10.03\scriptstyle\pm 2.57$	$11.05\scriptstyle\pm 1.56$	$12.19\scriptstyle\pm 0.90$
CoSE $(r=512)$	$9.71\scriptstyle\pm 1.02$	$11.20\scriptstyle\pm 0.90$	$11.46\scriptstyle\pm 1.08$	$11.96\scriptstyle\pm 1.52$

Appendix

Table 33: Memory-only adaptation with and without halting loss on ARC-AGI-2 public-evaluation puzzles. Scores are means

\pm

sample standard deviations over completed runs.

	With halting loss ( $w_{h}=0.5$ )		Without halting loss ( $w_{h}=0$ )
Base task memory	pass@1	pass@2	pass@1	pass@2
CoSE $(r=0,t=32)$	$11.01\scriptstyle\pm 0.66$	$12.33\scriptstyle\pm 1.71$	$10.58\scriptstyle\pm 1.18$	$13.74\scriptstyle\pm 0.91$
CoSE $(r=0)$	$8.77\scriptstyle\pm 1.16$	$10.77\scriptstyle\pm 1.69$	$12.33\scriptstyle\pm 0.61$	$14.04\scriptstyle\pm 1.14$
CoSE $(r=32,t=32)$	$8.92\scriptstyle\pm 0.25$	$11.35\scriptstyle\pm 0.45$	$13.74\scriptstyle\pm 1.01$	$16.47\scriptstyle\pm 1.69$
CoSE $(r=32)$	$10.72\scriptstyle\pm 1.06$	$13.01\scriptstyle\pm 2.44$	$13.40\scriptstyle\pm 0.97$	$15.20\scriptstyle\pm 0.88$
CoSE $(r=512)$	$8.70\scriptstyle\pm 2.38$	$9.36\scriptstyle\pm 2.07$	$13.16\scriptstyle\pm 0.25$	$15.06\scriptstyle\pm 0.51$

Appendix

Table 34: Per-puzzle FT Oracle with and without halting loss on ARC-AGI-2 public-evaluation puzzles. Scores are means

\pm

sample standard deviations over completed runs.

	With halting loss ( $w_{h}=0.5$ )		Without halting loss ( $w_{h}=0$ )
Base task memory	pass@1	pass@2	pass@1	pass@2
CoSE $(r=0,t=32)$	$5.95\scriptstyle\pm 2.47$	$8.38\scriptstyle\pm 3.22$	$6.09\scriptstyle\pm 1.14$	$6.77\scriptstyle\pm 1.85$
CoSE $(r=0)$	$5.26\scriptstyle\pm 1.24$	$6.14\scriptstyle\pm 0.00$	$6.07\scriptstyle\pm 0.10$	$7.60\scriptstyle\pm 0.83$
CoSE $(r=32,t=32)$	$7.36\scriptstyle\pm 1.24$	$11.16\scriptstyle\pm 0.59$	$5.95\scriptstyle\pm 1.04$	$8.63\scriptstyle\pm 0.15$
CoSE $(r=32)$	$6.19\scriptstyle\pm 0.55$	$8.24\scriptstyle\pm 1.19$	$5.75\scriptstyle\pm 0.94$	$8.63\scriptstyle\pm 1.16$
CoSE $(r=512)$	$6.87\scriptstyle\pm 1.24$	$9.72\scriptstyle\pm 1.55$	$5.31\scriptstyle\pm 1.49$	$7.80\scriptstyle\pm 1.39$

Appendix

Table 35: Joint fine-tuning with and without halting loss on ARC-AGI-2 public-evaluation puzzles. Scores are means

\pm

sample standard deviations.

Method	$w_{h}$	Immediate p@2	Final p@1	Final p@2	ARC-1 after
L2-SP	0.5	$41.58\scriptstyle\pm 1.51$	$35.48\scriptstyle\pm 1.80$	$40.68\scriptstyle\pm 2.70$	$79.44\scriptstyle\pm 0.96$
L2-SP	0	$45.40\scriptstyle\pm 1.68$	$38.87\scriptstyle\pm 0.86$	$44.13\scriptstyle\pm 0.70$	$78.33\scriptstyle\pm 0.00$
Rehearsal	0.5	$39.94\scriptstyle\pm 1.77$	$31.43\scriptstyle\pm 0.75$	$35.50\scriptstyle\pm 1.04$	$78.33\scriptstyle\pm 1.67$
Rehearsal	0	$41.94\scriptstyle\pm 1.25$	$31.76\scriptstyle\pm 1.67$	$37.20\scriptstyle\pm 1.25$	$78.06\scriptstyle\pm 3.76$
EWC	0.5	$40.13\scriptstyle\pm 1.31$	$14.45\scriptstyle\pm 6.15$	$18.10\scriptstyle\pm 6.58$	$6.11\scriptstyle\pm 7.88$
EWC	0	$41.99\scriptstyle\pm 1.99$	$21.22\scriptstyle\pm 8.31$	$25.94\scriptstyle\pm 8.46$	$30.28\scriptstyle\pm 37.33$

Appendix

Table 36: Halting-loss effects on sequential adaptation of the composition module. All runs use the CoSE

(r=32)

checkpoint and ARC-AGI-2 training stream, updating shared composition parameters and the current puzzle’s private rows. Scores are means

\pm

sample standard deviations.

Method	$w_{h}$	Immediate p@2	Final p@1	Final p@2	ARC-1 after
L2-SP	0.5	$48.20\scriptstyle\pm 1.86$	$1.50\scriptstyle\pm 2.24$	$2.07\scriptstyle\pm 3.23$	$30.00\scriptstyle\pm 39.05$
L2-SP	0	$52.37\scriptstyle\pm 2.38$	$16.50\scriptstyle\pm 1.33$	$20.86\scriptstyle\pm 2.34$	$77.78\scriptstyle\pm 0.96$
Rehearsal	0.5	$47.90\scriptstyle\pm 0.26$	$1.65\scriptstyle\pm 0.87$	$1.93\scriptstyle\pm 1.19$	$51.67\scriptstyle\pm 3.33$
Rehearsal	0	$45.51\scriptstyle\pm 0.98$	$4.58\scriptstyle\pm 1.18$	$5.58\scriptstyle\pm 0.94$	$56.94\scriptstyle\pm 4.28$
EWC	0.5	$38.40\scriptstyle\pm 2.02$	$0.14\scriptstyle\pm 0.25$	$0.14\scriptstyle\pm 0.25$	$0.00\scriptstyle\pm 0.00$
EWC	0	$44.11\scriptstyle\pm 0.62$	$0.79\scriptstyle\pm 1.36$	$0.93\scriptstyle\pm 1.26$	$6.67\scriptstyle\pm 11.55$

Appendix

Table 37: Halting-loss effects on sequential full-model adaptation. All runs use the CoSE

(r=32)

checkpoint and ARC-AGI-2 training stream, updating all shared weights and the current puzzle’s private rows. Scores are means

\pm

sample standard deviations; single-run results omit the standard deviation.

Review round	Error source	Count	Puzzle IDs
Round 1	Generator	19	15113be4 , 17cae0c1 , 2c0b0aff , 342ae2ed , 3ad05f52 , 423a55dc , 4e45f183 , 4ff4c9da , 538b439f , 705a3229 , 80214e03 , 8719f442 , 902510d5 , a416fc5b , b7955b3c , c6141b15 , c920a713 , e6de6e8f , e9fc42f2
Round 1	Official data	3	b74ca5d1 , d931c21c , f18ec8cc
Round 2	Generator	7	1b8318e3 , 689c358e , 79369cc6 , aee291af , c61be7dc , e048c9ed , e5c44e8f
Round 2	Official data	9	17829a00 , 85fa5666 , 963c33f8 , a3f84088 , ac0c5833 , b942fd60 , ba1aa698 , d753a70b , e1d2900e

Appendix

Table 38: Corrections identified during the two review rounds, grouped by error source.

Re-ARC aug per puzzle	Evaluation repeats	pass@2	pass@1000
10	1	76.38	86.25
10	2	77.38	86.62
10	5	62.12	79.25
10	10	54.62	70.50
100	1	79.25	88.62
100	2	79.50	89.50

Appendix

Table 39: ARC-AGI-1 runs over Re-ARC augmentation scale and evaluation-example repetition. The first column is the number of Re-ARC augmentations per Re-ARC puzzle. More augmentations generally help, but it makes evaluation pairs a tiny fraction of each epoch. Repeating evaluation examples partially corrects this imbalance, but too much repetition reduces accuracy.

$L$	$U$	$R$	$H_{\ell}$	Params (M)	Step	Loss	pass@1	pass@2	pass@1000
12	2	6	6	6.8	67588	0.012	47.50	52.08	71.38
12	3	4	6	10.2	67588	0.009	45.88	50.21	69.75
12	4	3	6	13.6	67588	0.006	40.83	46.08	66.62
12	6	2	6	20.5	67588	0.005	34.62	39.21	54.62
24	2	12	12	6.8	33794	0.034	47.00	52.96	69.75
24	3	8	12	10.2	33794	0.029	48.12	52.08	70.12

Appendix

Table 40: Iso-FLOP recurrent backbone runs on ARC-AGI-1 training tasks. Within each iso-FLOP family, the total forward depth

L=UR

and layer-level TBPTT horizon

H_{\ell}

are fixed, while the number of unique parameter layers

U

and recurrent applications

R

vary. This compares less parameter sharing against deeper recurrence at matched forward compute and learning horizon. Pass@

k

scores are percentages.

$U$	$R$	$L$	$H_{\ell}$	Params (M)	Step	Loss	pass@1	pass@2	pass@1000
2	12	24	6	6.8	67588	0.014	46.25	51.58	70.62
2	12	24	12	6.8	67588	0.009	49.75	56.08	71.38
2	12	24	18	6.8	67588	0.007	51.88	56.83	71.12
2	12	24	24	6.8	67588	0.017	47.00	52.04	69.75
3	8	24	3	10.2	67588	0.015	39.25	45.83	66.75
3	8	24	6	10.2	67588	0.012	40.88	47.83	68.25

Appendix

Table 41: Learning horizon sweep at fixed forward configuration on ARC-AGI-1 training tasks. All rows use total depth

L=24

; within each backbone shape,

U

and

R

are fixed while the layer-level learning horizon

H_{\ell}

varies. This tests how much of the recurrent computation should receive gradient updates during training, separate from the forward compute used at evaluation. Pass@

k

scores are percentages.

Protocol	Augmentation view voting	Checkpoint vote window	Exponential vote decay	Outer-loop steps	Relative compute
HRM/TRM [ 1 , 2 ]	Yes	All	No	16	$\sim 1\times 10^{4}$
URM [ 3 ]	Yes	All	No	16, 24	$\sim 1\times 10^{5}$
No TTC	No	Current only	No	16	$1$
Standard protocol (ours)	Yes	10 most recent	No	16	$\sim 1\times 10^{4}$
Additional TTC (ours)	Yes	recent half	Yes	16, 24	$\sim 6\times 10^{4}$
Continual learning (ours)	Yes	Current only	No	16	$\sim 1\times 10^{3}$

Appendix

Table 42: Evaluation protocols differ in view aggregation, checkpoint aggregation, and refinement depth. Relative evaluation compute estimates the inference cost from the number of views, evaluated checkpoints, and refinement steps, normalized to No TTC (

=1

). Exponential vote decay uses a half-life of 10 checkpoints.

Benchmark	Evaluator	pass@1	pass@2	pass@1000
ARC-AGI-1	CoSE , Re-ARC (aug=100)	74.75	79.62	89.25
ARC-AGI-1	+ loops 16/24, replay $h_{1/2}{=}10$	75.75	81.50	91.25
ARC-AGI-2	CoSE , synthetic Re-ARC2 (aug=100)	34.72	38.47	55.0
ARC-AGI-2	+ loops 16/24, replay $h_{1/2}{=}10$	33.50	39.30	58.3

Appendix

Table 43: Checkpoint replay and deeper refinement improve pass@2. The replay evaluator aggregates votes from recent checkpoints with exponential decay and ensembles 16 and 24-step outer refinement loops, compared with the 16-step default evaluator.

Re-ARC aug count	Eval repeats	Optimizer	pass@1	pass@2	pass@1000
10	1	AdamW	70.88	76.38	86.25
10	1	Muon	71.38	78.38	90.25
100	1	AdamW	72.62	79.25	88.62
100	1	Muon	72.50	77.38	91.50
100	2	AdamW	74.12	79.50	89.50
100	2	Muon	$74.29\scriptstyle\pm 0.75$	$80.71\scriptstyle\pm 0.51$	$90.62\scriptstyle\pm 0.50$

Appendix

Table 44: Optimizer ablation on ARC-AGI-1. AdamW and Muon reach comparable saturated accuracy across Re-ARC data mixes, while Muon often reaches its best checkpoint in fewer training steps. The 100/2 Muon row reports mean

\pm

sample standard deviation over three runs; all other rows are single runs.

			Hardware (1 node, 8 GPUs)
Benchmark	Model	Metric	RTX PRO 6000	H100	B200
ARC-AGI-1	TRM	Train (s/step)	0.195	0.110	0.061
		Eval (s/iter)	1.81	1.74	0.525
		Total (h)	47.6	32.9	14.6
	URM	Train (s/step)	0.578	0.385	0.257
		Eval (s/iter)	4.09	2.70	1.78

Appendix

Table 45: Single-node compute estimates for ARC training runs. Train and eval rows report seconds per training step and per evaluation iteration, total rows report projected end-to-end wall-clock hours for the full training.

		ARC-AGI-1 TRM			ARC-AGI-1 CHARM			ARC-AGI-2 CHARM
System	Nodes	Train (s)	Eval (s)	Total (h)	Train (s)	Eval (s)	Total (h)	Train (s)	Eval (s)	Total (h)
Polaris	1	0.535	5.00	131.0	1.740	12.50	394.6	1.750	12.60	561.7
	4	0.188	1.26	41.9	0.550	3.15	118.0	0.550	3.15	171.7
	8	0.107	0.69	23.6	0.320	1.65	67.1	0.316	1.64	97.8
Aurora	1	1.630	15.36	400.2	3.720	25.46	833.0	3.720	25.46	1186.1
	4	0.413	3.79	100.5	1.040	6.28	225.9	1.040	6.28	326.6

Appendix

Table 46: ALCF compute estimates for ARC training runs. Rows report measured seconds per training update and evaluation iteration on Polaris and Aurora, followed by projected end-to-end hours for the full training run. Polaris uses 4 NVIDIA A100 GPUs per node, Aurora uses 6 Intel Data Center GPU Max Series GPUs per node. Aurora 24 and 32-node CHARM experiments failed deterministically with driver-level segmentation faults, reported directly as “segfault.”

Explore similar work

Aug 10, 2026cs.NE

BDH-CQ: In-Context Learning with Recurrent Latent Reasoning

We introduce BDH-CQ, a reasoning model that combines in-context learning with recurrent latent reasoning. Inputs presented at inference time continuously update the model's recurrent memory; the model then solves a query through iterative computation in a high-dimensional latent space, without verbalizing its intermediate reasoning. We evaluate the model on the public ARC-AGI-1 evaluation set and use controlled ARC-like interventions to study what it learns from demonstrations, how consistently it applies an inferred transformation, and which concepts remain difficult. A 150M-parameter configuration reaches 29.5% pass@2 at a computed inference cost of $0.0007 per task. This operating point breaks through the previously reported ARC-AGI-1 cost-accuracy Pareto frontier, establishing a new state of the art in benchmark cost efficiency.

Björn Engdahl, Adrian Kosowski, Jan Chorowski +6

New York University

Efficient Latent Reasoning Large Reasoning Models Inference Cost In-Context Transform Concepts

Ask PDF

Sep 30, 2026cs.LG

What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling

Recursive reasoning models apply a small shared Transformer block many times to refine a latent state. This gives them large effective depth with few parameters and makes them strong on algorithmic tasks. Such compact solvers are natural candidates for tools that an LLM can call on narrow algorithmic subproblems. However, existing models such as HRM, TRM and URM differ in architecture, gradient propagation and training procedure simultaneously. This makes it hard to tell what drives their performance, and their optimization is still poorly understood and often unstable. In this work we address both of these gaps. First, we study these questions under a unified experimental pipeline spanning six algorithmic domains. Individual controlled ablations are performed on representative domains, while the resulting recipe is evaluated across the full suite. The study reveals a surprisingly simple recipe for stable and generalizable recursive reasoning: an intermediate gradient horizon, large physical batches and controlled updates of the recurrent state. An explicit hierarchical architecture is not needed. Second, we combine these findings into a stable 13.6M-parameter model that achieves the strongest overall performance among the evaluated recursive baselines, with particularly large gains on out-of-distribution generalization. It raises Arithmetic OOD accuracy to 71.2%, from 36.2% for the strongest baseline, while reaching 98.41% on Sudoku and 59.5% pass@2 on ARC-AGI-1. Our results show that, within the recursive architectures studied here, performance depends strongly on how recurrence is optimized and stabilized. More broadly, it shows how AI systems can be improved by optimizing their components one at a time.

Yuliana Shakhvalieva, Dmitrii Kharchev, Viacheslav Bezrukov +4

RND NLP, DAIMLD, Russian Federation

Recursive Models Large Reasoning Models Recurrent State

Ask PDF

May 19, 2026cs.AI

Generative Recursive Reasoning

How should future neural reasoning systems implement extended computation? Recursive Reasoning Models (RRMs) offer a promising alternative to autoregressive sequence extension by performing iterative latent-state refinement with shared transition functions. Yet existing RRMs are largely deterministic, following a single latent trajectory and converging to a single prediction. We introduce Generative Recursive reAsoning Models (GRAM), a framework that turns recursive latent reasoning into probabilistic multi-trajectory computation. GRAM models reasoning as a stochastic latent trajectory, enabling multiple hypotheses, alternative solution strategies, and inference-time scaling through both recursive depth and parallel trajectory sampling. This yields a latent-variable generative model supporting conditional reasoning via

p_θ(y \mid x)

and, with fixed or absent inputs, unconditional generation via

p_θ(x)

. Trained with amortized variational inference, GRAM improves over deterministic recurrent and recursive baselines on structured reasoning and multi-solution constraint satisfaction tasks, while demonstrating an unconditional generation capability. https://ahn-ml.github.io/gram-website

Junyeob Baek, Mingyu Jo, Minsu Kim +3

KAIST · Mila – Québec AI Institute · New York University +1

Recursive Models Latent Trajectories Autoregressive Model Variational Inference

Ask PDF