The Abstraction and Reasoning Corpus and related benchmarks evaluate whether AI models can solve novel reasoning tasks, but often leave unclear whether success reflects inference of the intended underlying rule or reliance on shortcuts. We address this gap by studying test-time task embeddings in Vision ARC (VARC), a model in which a pre-trained backbone is complemented by a trainable embedding representing the transformation rule. In the original VARC, test-time training (TTT) is jointly applied to the backbone and task embedding. Here we introduce a novel two-step TTT protocol: first finetune only the task embedding (Embed-TTT), then freeze it and finetune the backbone. Across ARC-AGI-1, ConceptARC, and two controlled datasets with known rules, Embed-TTT consistently yields improved task embeddings, ones that align better with underlying task rules, improve embedding-based retrieval, and enable accurate linear probing of known rules. Qualitatively, Embed-TTT identifies more semantically meaningful relations between test and train tasks on ARC-AGI-1. We also show that optimizing only task embeddings (less than 0.01% of model parameters) already solves a non-trivial fraction of ARC-AGI-1, ConceptARC, and Mini-ARC tasks, while the full two-step pipeline improves final performance. Finally, we show that Embed-TTT recovers the underlying geometric structure of parametric rules and learns compositional capabilities that enable rule-wise interpolation, but not extrapolation. These findings support a clearer separation between rule induction and rule execution in ARC-like evaluations, motivating benchmarks that better distinguish in-distribution from out-of-distribution rules.
Figures & tables
Figure 1: Method. After training a model that learns task embeddings that condition the backbone (left), we elicit implicit rule induction at test time: we first freeze the backbone and learn a task embedding (Embed-TTT, center), then freeze the embedding and finetune the backbone (right).
Train dataset
Num. tasks
Evaluation
Metric
Metric score
train/probe
Chance
Full-TTT
Embed-TTT
ARC-1 train
400/same
Emb. retrieval (train)
Top-5 match ↑
1.3%
6.5%
86.25%
ConceptARC
160/same
Emb. retrieval (train)
Top-5 match ↑
3.1%
69.4%
98.8%
ConceptARC
160/same
Lin. classif. (train)
Accuracy ↑
6.2%
21.3%
60.6%
Custom LUCD
15/150
Lin. classif. (test)
Accuracy ↑
6.7%
11.3%
100%
Custom Moves
121/1210
Lin. regress. (test)
RMSE ↓
-
2.94
2.12
Table 1: Quantitative implicit rule induction. We evaluate Embed-TTT and Full-TTT on ARC-like datasets in their ability to produce embeddings that potentially elicit rule induction. It appears that Embed-TTT excels, while Full-TTT generally loses any semantically meaningful information.
Figure 2: UMAP projections for ARC-1. Full-TTT embeddings (left) are projected far from the original embedding distribution, while Embed-TTT embeddings (right) are projected adequately onto it. The original embedding usually ranks among the closest train embeddings to the fine-tuned one with Embed-TTT (most points are dark green, indicating a good rank).
Figure 3: Qualitative evaluation through test-train task similarity. For Full-TTT (left part) and Embed-TTT (right part), we report the closest associations between test (left input-output pairs per part) and train (right input-output pairs) tasks. Full-TTT does not associate tasks that seem to share common rules, while Embed-TTT associates tasks that are clearly similar rule-wise, the top one being a task duplicate with different IDs (070dd51e and 40853293) in test and train sets.
Train dataset
Test dataset
Pass@1 per TTT method
Pass@2 per TTT method
Full
Embed
Embed-Full
Full
Embed
Embed-Full
ARC-1 train
ARC-1 test
48.8%
13.0%
49.5%
53.8%
17.2%
55.2%
ARC-1 train
ConceptARC
23.8%
8.8%
28.1%
32.5%
14.4%
37.5%
ARC-1 train
Mini-ARC
59.7%
22.2%
63.8%
62.4%
34.9%
71.1%
ARC-1 train
ConceptARC (*)
47.1%
24.0%
49.8%
53.8%
33.3%
57.1%
Concept retrieval in ARC
Top-1: Test 36.3% (29/80), Train 35.0% (28/80), from Embed-TTT
Table 2: Cross-dataset results. Our Embed-Full-TTT allows to reach better final per-task performances than Full-TTT on ARC-1, ConceptARC and Mini-ARC. In addition, Embed-TTT, solves a significant amount of tasks by just finetuning the embeddings, indicating that rules underlying test tasks might not always differ much from train tasks. Pair-level (instead of task-level) evaluation on ConceptARC(*) shows a large gap, indicating potential difficulties in solving all input-output pairs of a given task consistently. Finally, ConceptARC concepts inferred on ARC-1 tasks by task embeddings similarity decently match manual annotations despite the fact that the model was never trained for concept retrieval.
Figure 4: Embeddings structure. In our Moves dataset, the task geometry (left) is reflected by the UMAP of train embeddings (center-left), which forms an almost regular grid. We then learn a linear map from train embeddings finetuned with Embed-TTT (resp. Full-TTT) to their ground-truth move coordinates (gray squares) (center-right, resp. right), and apply it to test embeddings (points colored by projection error). Embed-TTT preserves a regular grid with low error, unlike Full-TTT.
Figure 5: Task inter/extrapolation. (Left) Move tasks are placed in the plane by their displacement. Gray squares denote train tasks; circles denote test tasks, colored by success under Embed-TTT finetuning, across several train/test splits. Test tasks are mostly solved when at least 2 train moves lie on both their row and column, suggesting interpolation is feasible while extrapolation is limited. (Right) Violin plots aggregate test successes by the minimum number of train tasks on the same row/column. A clear transition appears at 2 , akin to a “grokking” effect. Split violins show “inside” test tasks (left half) are slightly better solved than “outside” tasks (right half).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Our LUCD tasks . Examples of input-output pairs per LUCD task of our dataset. Tasks are encoded as ‘left’ (L), ‘up’ (U), ’cross’ (C), ’dilate’ (D), ’neutral’ (N) transforms, composed with each other.
Figure 7: Our Moves tasks. Examples of input-output pairs per Moves task of our dataset. Tasks are encoded as left (L) or right (R) moves and up (U) and down (D) moves.
Figure 8: Similarity retrieval. (Left) Full-TTT retrieves test-train pairs of tasks that do not seem semantically similar. (Right) Embed-TTT retrieves test-train pairs of tasks that share some common rules.
Figure 9: Embed-TTT PCA on our Moves tasks. (Top) PCA visualization of the Embed-TTT embeddings of our Moves tasks, colored according to ground truth vertical (left) and horizontal (right) moves. (Bottom) Heatmap showing, for each ground-truth move (represented as a square at (x,y) coordinates), its two main PCA values. These two equivalent plots show that the first component (PCA1) captures vertical moves, the second component (PCA2) captures horizontal moves.
Figure 10: Full-TTT PCA on our Moves tasks. (Top) PCA visualization of the Full-TTT embeddings of our Moves tasks, colored according to ground truth vertical (left) and horizontal (right) moves. (Bottom) Heatmap showing, for each ground-truth move (represented as a square at (x,y) coordinates), its two main PCA values. Full-TTT seems to distinguish left moves from right moves, and up moves from down moves, but the exact move ordering seems much less recaptured than with Embed-TTT.
TTT method
k=2
k=3
k=4
k=5
k=10
k=30
k=50
k=56
Embed-TTT
82.81%
91.41%
95.31%
98.18%
100.00%
100.00%
100.00%
100.00%
Full-TTT
0.00%
0.00%
0.00%
0.00%
0.00%
1.30%
35.94%
93.49%
Appendix
Table 3: Top- k retrieval of both correct constituents (train move and train switch) for composed test tasks. Note that the entire dataset consists of k=57 tasks.
Test-time training adapts model parameters or memory during deployment using each input or stream to improve performance under changing tasks and data.
The Abstraction and Reasoning Corpus (ARC) is increasingly used to evaluate whether neural networks can infer task-specific transformations from few demonstrations. Recent ARC models differ in architecture, but also in broader training and evaluation protocols, including pretraining data, data augmentation, test-time setup, and optimization budgets, making conclusions about architecture benefits debatable. In this work, we study how the performance of the vision-inspired VARC and recursion-based TRM models is affected by pretraining and test-time training (TTT) budgets under a common inductive protocol on ARC-AGI-1. We cross five pretraining budgets with four TTT budgets for each test task. Each budget specifies a task-variant count and total sample presentations, defining the controlled training budgets. We observe that increasing TTT sample presentations improves the standard ARC pass@2 metric in all settings for both models. Increasing the TTT task-variant count consistently helps only TRM, whereas VARC often performs better with fewer variants. VARC also remains competitive without pretraining, while TRM benefits much more from it. These results show that pretraining and TTT budgets affect ARC evaluation and motivate reporting at comparable budgets.
Adrien Deliege, Marc Van Droogenbroeck
Montefiore Institute of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium
The Abstraction and Reasoning Corpus (ARC-AGI) probes few-shot abstraction and rule induction on small visual grids, but progress is difficult to measure on static collections of hand-authored puzzles due to overfitting, dataset leakage, and memorisation. We introduce ARC-TGI (ARC Task Generators Inventory), an open-source framework for task-family generators: compact Python programs that sample diverse ARC-AGI tasks while preserving a latent rule. ARC-TGI is built around a solver-facing representation: each generated task is paired with natural-language input and transformation reasoning chains and partially evaluated Python code implementing sampling, transformation, and episode construction. Crucially, ARC-TGI supports task-level constraints so that training examples collectively expose the variations needed to infer the underlying rule, a requirement for human-solvable ARC tasks that independent per-example sampling often fails to guarantee. All generators undergo human refinement and local verification to keep both grids and reasoning traces natural and consistent under variation. We release 461 generators covering 180 ARC-Mini tasks, 215 ARC-AGI-1 tasks (200 train, 15 test), and 66 ARC-AGI-2 tasks (55 train, 11 test), enabling scalable dataset sampling and controlled benchmarking.
The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format fails to evaluate the diversity of abilities that genuine abstract skill acquisition should enable. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task's underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin employs programmatic methods to generate new task instances and transform given inputs for a given ARC task, enabling dynamic generative sampling beyond fixed input-output pairs. Across five state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 25-52 percentage-point performance gap between standard ARC evaluation and evaluation on PotARCin, and find that multi-dimensional evaluation reorders models that standard accuracy ranks alike. We further investigate effects of generative sampling, difficulty of corruption types, and questions of self-consistency, showing that models frequently contradict their own formalized rule even where they have stated it correctly. We also introduce P-ARC, a held-out hand-crafted test set, on which models achieve 1-8% accuracy across all five dimensions, underscoring the importance of more holistic evaluations of abstract reasoning capabilities.