Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

Adrien Deliège; Claas Beger; Marc Van Droogenbroeck; Melanie Mitchell

cs.AISep 18, 2026

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

Authors: Adrien Deliège, Claas Beger, Marc Van Droogenbroeck, Melanie Mitchell

Organizations: University of Liège · Santa Fe Institute

Abstract

The Abstraction and Reasoning Corpus and related benchmarks evaluate whether AI models can solve novel reasoning tasks, but often leave unclear whether success reflects inference of the intended underlying rule or reliance on shortcuts. We address this gap by studying test-time task embeddings in Vision ARC (VARC), a model in which a pre-trained backbone is complemented by a trainable embedding representing the transformation rule. In the original VARC, test-time training (TTT) is jointly applied to the backbone and task embedding. Here we introduce a novel two-step TTT protocol: first finetune only the task embedding (Embed-TTT), then freeze it and finetune the backbone. Across ARC-AGI-1, ConceptARC, and two controlled datasets with known rules, Embed-TTT consistently yields improved task embeddings, ones that align better with underlying task rules, improve embedding-based retrieval, and enable accurate linear probing of known rules. Qualitatively, Embed-TTT identifies more semantically meaningful relations between test and train tasks on ARC-AGI-1. We also show that optimizing only task embeddings (less than 0.01% of model parameters) already solves a non-trivial fraction of ARC-AGI-1, ConceptARC, and Mini-ARC tasks, while the full two-step pipeline improves final performance. Finally, we show that Embed-TTT recovers the underlying geometric structure of parametric rules and learns compositional capabilities that enable rule-wise interpolation, but not extrapolation. These findings support a clearer separation between rule induction and rule execution in ARC-like evaluations, motivating benchmarks that better distinguish in-distribution from out-of-distribution rules.

Figures & tables

Figure 1: Method. After training a model that learns task embeddings that condition the backbone (left), we elicit implicit rule induction at test time: we first freeze the backbone and learn a task embedding (Embed-TTT, center), then freeze the embedding and finetune the backbone (right).

Train dataset	Num. tasks	Evaluation	Metric	Metric score
Train dataset	train/probe	Evaluation	Metric	Chance	Full-TTT	Embed-TTT
ARC-1 train	400/same	Emb. retrieval (train)	Top-5 match $\uparrow$	1.3%	6.5%	86.25%
ConceptARC	160/same	Emb. retrieval (train)	Top-5 match $\uparrow$	3.1%	69.4%	98.8%
ConceptARC	160/same	Lin. classif. (train)	Accuracy $\uparrow$	6.2%	21.3%	60.6%
Custom LUCD	15/150	Lin. classif. (test)	Accuracy $\uparrow$	6.7%	11.3%	100%
Custom Moves	121/1210	Lin. regress. (test)	RMSE $\downarrow$	-	2.94	2.12

Table 1: Quantitative implicit rule induction. We evaluate Embed-TTT and Full-TTT on ARC-like datasets in their ability to produce embeddings that potentially elicit rule induction. It appears that Embed-TTT excels, while Full-TTT generally loses any semantically meaningful information.

Panel of Figure 2 — Figure 2: UMAP projections for ARC-1. Full-TTT embeddings (left) are projected far from the original embedding distribution, while Embed-TTT embeddings (right) are projected adequately onto it. The original embedding usually ranks among the closest train embeddings to the fine-tuned one with Embed-TTT (most points are dark green, indicating a good rank).

Panel of Figure 3 — Figure 3: Qualitative evaluation through test-train task similarity. For Full-TTT (left part) and Embed-TTT (right part), we report the closest associations between test (left input-output pairs per part) and train (right input-output pairs) tasks. Full-TTT does not associate tasks that seem to share common rules, while Embed-TTT associates tasks that are clearly similar rule-wise, the top one being a task duplicate with different IDs (070dd51e and 40853293) in test and train sets.

Train dataset	Test dataset	Pass@1 per TTT method			Pass@2 per TTT method
Train dataset	Test dataset	Full	Embed	Embed-Full	Full	Embed	Embed-Full
ARC-1 train	ARC-1 test	48.8%	13.0%	49.5%	53.8%	17.2%	55.2%
ARC-1 train	ConceptARC	23.8%	8.8%	28.1%	32.5%	14.4%	37.5%
ARC-1 train	Mini-ARC	59.7%	22.2%	63.8%	62.4%	34.9%	71.1%
ARC-1 train	ConceptARC (*)	47.1%	24.0%	49.8%	53.8%	33.3%	57.1%
Concept retrieval in ARC		Top-1: Test 36.3% (29/80), Train 35.0% (28/80), from Embed-TTT

Table 2: Cross-dataset results. Our Embed-Full-TTT allows to reach better final per-task performances than Full-TTT on ARC-1, ConceptARC and Mini-ARC. In addition, Embed-TTT, solves a significant amount of tasks by just finetuning the embeddings, indicating that rules underlying test tasks might not always differ much from train tasks. Pair-level (instead of task-level) evaluation on ConceptARC(*) shows a large gap, indicating potential difficulties in solving all input-output pairs of a given task consistently. Finally, ConceptARC concepts inferred on ARC-1 tasks by task embeddings similarity decently match manual annotations despite the fact that the model was never trained for concept retrieval.

Panel of Figure 4 — Figure 4: Embeddings structure. In our Moves dataset, the task geometry (left) is reflected by the UMAP of train embeddings (center-left), which forms an almost regular grid. We then learn a linear map from train embeddings finetuned with Embed-TTT (resp. Full-TTT) to their ground-truth move coordinates (gray squares) (center-right, resp. right), and apply it to test embeddings (points colored by projection error). Embed-TTT preserves a regular grid with low error, unlike Full-TTT.

Panel of Figure 5 — Figure 5: Task inter/extrapolation. (Left) Move tasks are placed in the plane by their displacement. Gray squares denote train tasks; circles denote test tasks, colored by success under Embed-TTT finetuning, across several train/test splits. Test tasks are mostly solved when at least $2$ train moves lie on both their row and column, suggesting interpolation is feasible while extrapolation is limited. (Right) Violin plots aggregate test successes by the minimum number of train tasks on the same row/column. A clear transition appears at $2$ , akin to a “grokking” effect. Split violins show “inside” test tasks (left half) are slightly better solved than “outside” tasks (right half).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Our LUCD tasks . Examples of input-output pairs per LUCD task of our dataset. Tasks are encoded as ‘left’ (L), ‘up’ (U), ’cross’ (C), ’dilate’ (D), ’neutral’ (N) transforms, composed with each other.

Panel of Figure 7 — Figure 7: Our Moves tasks. Examples of input-output pairs per Moves task of our dataset. Tasks are encoded as left (L) or right (R) moves and up (U) and down (D) moves.

Panel of Figure 8 — Figure 8: Similarity retrieval. (Left) Full-TTT retrieves test-train pairs of tasks that do not seem semantically similar. (Right) Embed-TTT retrieves test-train pairs of tasks that share some common rules.

Panel of Figure 9 — Figure 9: Embed-TTT PCA on our Moves tasks. (Top) PCA visualization of the Embed-TTT embeddings of our Moves tasks, colored according to ground truth vertical (left) and horizontal (right) moves. (Bottom) Heatmap showing, for each ground-truth move (represented as a square at (x,y) coordinates), its two main PCA values. These two equivalent plots show that the first component (PCA1) captures vertical moves, the second component (PCA2) captures horizontal moves.

Panel of Figure 10 — Figure 10: Full-TTT PCA on our Moves tasks. (Top) PCA visualization of the Full-TTT embeddings of our Moves tasks, colored according to ground truth vertical (left) and horizontal (right) moves. (Bottom) Heatmap showing, for each ground-truth move (represented as a square at (x,y) coordinates), its two main PCA values. Full-TTT seems to distinguish left moves from right moves, and up moves from down moves, but the exact move ordering seems much less recaptured than with Embed-TTT.

TTT method	$k=2$	$k=3$	$k=4$	$k=5$	$k=10$	$k=30$	$k=50$	$k=56$
Embed-TTT	82.81%	91.41%	95.31%	98.18%	100.00%	100.00%	100.00%	100.00%
Full-TTT	0.00%	0.00%	0.00%	0.00%	0.00%	1.30%	35.94%	93.49%

Appendix

Table 3: Top-

k

retrieval of both correct constituents (train move and train switch) for composed test tasks. Note that the entire dataset consists of k=57 tasks.

Test-Time Training

Test-time training adapts model parameters or memory during deployment using each input or stream to improve performance under changing tasks and data.

See all 99 papers

Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM
Oct 9, 2026Adrien Deliege, Marc Van DroogenbroeckRecursive Reasoning Models Abstract Reasoning
Montefiore Institute of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium
ARC-TGI: Human-Validated Task Generators with Reasoning Chain Templates for ARC-AGI
Mar 5, 2026Jens Lehmann, Syeda Khushbakht, Nikoo Salehfard +4ARC-AGI Synthetic Data Generation
Dresden University of Technology Dresden, Germany · Amazon · TIB - Leibniz Information Centre Hanover, Germany +1
PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks
Sep 23, 2026Claas Beger, Ryan Yi, Melanie MitchellAbstract Reasoning LLM Evaluation
Santa Fe Institute

Oct 9, 2026J/K move · Enter open · S save

Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM

Adrien Deliege, Marc Van Droogenbroeck

Montefiore Institute of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium

Recursive Reasoning Models Abstract Reasoning Test-Time Training Visual Autoregressive Models

Open paper AskPDF arXiv

The Abstraction and Reasoning Corpus (ARC) is increasingly used to evaluate whether neural networks can infer task-specific transformations from few demonstrations. Recent ARC models differ in architecture, but also in broader training and evaluation protocols, including pretraining data, data augmentation, test-time setup, and optimization budgets, making conclusions about architecture benefits debatable. In this work, we study how the performance of the vision-inspired VARC and recursion-based TRM models is affected by pretraining and test-time training (TTT) budgets under a common inductive protocol on ARC-AGI-1. We cross five pretraining budgets with four TTT budgets for each test task. Each budget specifies a task-variant count and total sample presentations, defining the controlled training budgets. We observe that increasing TTT sample presentations improves the standard ARC pass@2 metric in all settings for both models. Increasing the TTT task-variant count consistently helps only TRM, whereas VARC often performs better with fewer variants. VARC also remains competitive without pretraining, while TRM benefits much more from it. These results show that pretraining and TTT budgets affect ARC evaluation and motivate reporting at comparable budgets.

Abstract

Test-Time Training

Explore similar work

Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM

ARC-TGI: Human-Validated Task Generators with Reasoning Chain Templates for ARC-AGI

PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks