Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM

Adrien Deliege; Marc Van Droogenbroeck

cs.LGOct 9, 2026

Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM

Authors: Adrien Deliege, Marc Van Droogenbroeck

Organizations: Montefiore Institute of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium

Abstract

The Abstraction and Reasoning Corpus (ARC) is increasingly used to evaluate whether neural networks can infer task-specific transformations from few demonstrations. Recent ARC models differ in architecture, but also in broader training and evaluation protocols, including pretraining data, data augmentation, test-time setup, and optimization budgets, making conclusions about architecture benefits debatable. In this work, we study how the performance of the vision-inspired VARC and recursion-based TRM models is affected by pretraining and test-time training (TTT) budgets under a common inductive protocol on ARC-AGI-1. We cross five pretraining budgets with four TTT budgets for each test task. Each budget specifies a task-variant count and total sample presentations, defining the controlled training budgets. We observe that increasing TTT sample presentations improves the standard ARC pass@2 metric in all settings for both models. Increasing the TTT task-variant count consistently helps only TRM, whereas VARC often performs better with fewer variants. VARC also remains competitive without pretraining, while TRM benefits much more from it. These results show that pretraining and TTT budgets affect ARC evaluation and motivate reporting at comparable budgets.

Figures & tables

Figure 1: Example of an ARC task. Evaluation task a680ac02 provides three demonstration input-output pairs specifying a task-specific transformation. The evaluation query exposes only its input; the target needs to be predicted.

Figure 2: Pretraining and TTT pipelines and budgets. Pretraining on ARC tasks incurs a choice of task variants

V

and of sample-presentation budget

B

, then it produces one pretrained model. Then, for each new evaluation task, a potentially different choice of variants

V

can be made, as well as budget

B

during the TTT process. The pretrained model is adapted for the task at hand, predicts the query, inverts the transformations, and aggregates predictions. Green boxes are fixed, orange boxes contain parameters that we vary, based on original VARC and TRM values.

Figure 3: Performance under fixed pretraining and TTT budgets. We report pass@2 results for VARC and TRM. Rows vary the pretraining task-variant count $V$ and sample-presentation budget $B$ ; columns vary them for TTT. Both tables use a shared linear 26–40% scale, with values below 26% shown in dark grey.

Figure 4: Performance changes under budget increases. We show the changes in pass@2 across our budget configurations when one budget is increased, indicated on the left. Diamonds show medians. Positive changes favor the budget-intensive setting.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

(a) VARC
Variants/Samples	TTT 51/15.3k	TTT 51/300.3k	TTT 1001/15.3k	TTT 1001/300.3k
No pretraining	23.75/26.12/34.75	30.75/34.38/43.13	14.50/16.13/23.13	30.63/35.50/46.25
Pretrain 51/40M	31.75/34.88/43.00	35.88/39.37/50.00	29.13/32.37/47.63	33.87/39.25/52.62
Pretrain 51/130M	31.50/34.25/42.25	35.00/40.00/47.75	29.38/33.12/44.75	33.00/36.75/49.38
Pretrain 1001/40M	26.50/28.75/39.25	33.12/35.75/46.25	24.87/28.63/44.00	30.63/35.13/48.88
Pretrain 1001/130M	26.62/29.75/38.25	28.88/32.37/39.75	26.12/30.13/41.50	30.25/33.75/43.63

Appendix

Table 1: Complete ARC-AGI-1 tables. Each cell is pass@1 / pass@2 / oracle in percentage over 400 evaluation tasks. Rows are pretraining task-variant count/sample-presentation budget; columns are the corresponding TTT budgets. All pretrained rows use ARC training demonstrations without RE-ARC.

Model	$V$	$B$	Wall-h	GPU-h	pass@2 range
VARC-ViT	51	40M	10.81	86.5	32.37–39.37
VARC-ViT	51	130M	35.08	280.6	33.12–40.00
VARC-ViT	1001	40M	12.12	96.9	28.63–35.75
VARC-ViT	1001	130M	38.61	308.9	29.75–33.75
TRM	51	40M	5.3	42.4	28.00–33.37
TRM	51	130M	16.77	134.2	29.00–37.50

Appendix

Table 2: ARC pretraining compute on 8-A100 GPUs. GPU-hours are wall time multiplied by eight. The pass@2 range covers the four downstream TTT cells.

Model	$V$	$B$	Time/task	GPU-h	Scratch pass@2
VARC	51	15k	112.1s	12.46	26.12
VARC	51	300k	1716.8s	190.75	34.38
VARC	1001	15k	164.4s	18.27	16.13
VARC	1001	300k	1785.3s	198.37	35.50
TRM	51	15k	88.0s	9.78	2.38
TRM	51	300k	1338.1s	148.67	10.37

Appendix

Table 3: TTT runtime and scratch performance for all TTT configurations. Runtime values are means across complete 400-task TTT runs on A100 GPUs, with one GPU per running task. Scratch pass@2 is reported as a common performance reference.

Recursive Reasoning Models

See all 12 papers

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks
Sep 18, 2026Adrien Deliège, Claas Beger, Marc Van Droogenbroeck +1Test-Time Training Visual Reasoning
University of Liège · Santa Fe Institute
DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models
Jun 25, 2026Yuxuan Yang, Feiyang Li, Yile Wang
College of Computer Science and Software Engineering, Shenzhen University
Tiny Autoregressive Recursive Models
Mar 9, 2026Paulius Rauba, Claudio Fanconi, Mihaela van der SchaarRecursive Language Models Iterative Refinement
University of Cambridge

Sep 18, 2026J/K move · Enter open · S save

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

Adrien Deliège, Claas Beger, Marc Van Droogenbroeck, Melanie Mitchell

University of Liège · Santa Fe Institute

Test-Time Training Visual Reasoning ARC-AGI

Open paper AskPDF arXiv

The Abstraction and Reasoning Corpus and related benchmarks evaluate whether AI models can solve novel reasoning tasks, but often leave unclear whether success reflects inference of the intended underlying rule or reliance on shortcuts. We address this gap by studying test-time task embeddings in Vision ARC (VARC), a model in which a pre-trained backbone is complemented by a trainable embedding representing the transformation rule. In the original VARC, test-time training (TTT) is jointly applied to the backbone and task embedding. Here we introduce a novel two-step TTT protocol: first finetune only the task embedding (Embed-TTT), then freeze it and finetune the backbone. Across ARC-AGI-1, ConceptARC, and two controlled datasets with known rules, Embed-TTT consistently yields improved task embeddings, ones that align better with underlying task rules, improve embedding-based retrieval, and enable accurate linear probing of known rules. Qualitatively, Embed-TTT identifies more semantically meaningful relations between test and train tasks on ARC-AGI-1. We also show that optimizing only task embeddings (less than 0.01% of model parameters) already solves a non-trivial fraction of ARC-AGI-1, ConceptARC, and Mini-ARC tasks, while the full two-step pipeline improves final performance. Finally, we show that Embed-TTT recovers the underlying geometric structure of parametric rules and learns compositional capabilities that enable rule-wise interpolation, but not extrapolation. These findings support a clearer separation between rule induction and rule execution in ARC-like evaluations, motivating benchmarks that better distinguish in-distribution from out-of-distribution rules.

Abstract

Recursive Reasoning Models

Explore similar work

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models

Tiny Autoregressive Recursive Models