ARC-TGI: Human-Validated Task Generators with Reasoning Chain Templates for ARC-AGI
Authors: Jens Lehmann, Syeda Khushbakht, Nikoo Salehfard, Nur A Zarin Nishat, Dhananjay Bhandiwad, Andrei Aioanei, Sahar Vahdati
Organizations: Dresden University of Technology Dresden, Germany · Amazon · TIB - Leibniz Information Centre Hanover, Germany · TIB - Leibniz Information Centre Leibniz University of Hanover Hanover, Germany
The Abstraction and Reasoning Corpus (ARC-AGI) probes few-shot abstraction and rule induction on small visual grids, but progress is difficult to measure on static collections of hand-authored puzzles due to overfitting, dataset leakage, and memorisation. We introduce ARC-TGI (ARC Task Generators Inventory), an open-source framework for task-family generators: compact Python programs that sample diverse ARC-AGI tasks while preserving a latent rule. ARC-TGI is built around a solver-facing representation: each generated task is paired with natural-language input and transformation reasoning chains and partially evaluated Python code implementing sampling, transformation, and episode construction. Crucially, ARC-TGI supports task-level constraints so that training examples collectively expose the variations needed to infer the underlying rule, a requirement for human-solvable ARC tasks that independent per-example sampling often fails to guarantee. All generators undergo human refinement and local verification to keep both grids and reasoning traces natural and consistent under variation. We release 461 generators covering 180 ARC-Mini tasks, 215 ARC-AGI-1 tasks (200 train, 15 test), and 66 ARC-AGI-2 tasks (55 train, 11 test), enabling scalable dataset sampling and controlled benchmarking.
Figures & tables
Method
Role (fixed signature)
create_input
(self, taskvars: dict, gridvars: dict) -> np.ndarray : sample an input grid while randomizing nuisance factors.
transform_input
(self, grid: np.ndarray, taskvars: dict) -> np.ndarray : deterministically apply the latent rule to produce the output grid.
create_grids
(self) -> Tuple[Dict[str, Any], TrainTestData] : construct a complete episode (train/test pairs) and return sampled variables plus train/test data.
Table 1. ARC-TGI generator API. taskvars are per-episode variables used to instantiate templates; gridvars are per-grid auxiliaries used during sampling.
Figure 1. Input grid-size heatmaps. Upper row: original tasks. Lower row: ARC-TGI samples (50 per generator).
Figure 3. Per-grid diversity visualization for one ARC-TGI generator ( taskcmBhVbGzL8ZgWXDE5CUS6B ), showing variation in spatial position, size, and color across sampled grids while preserving the latent rule.
Figure 4. Few-shot performance on ARC-TGI-50N . (A) ARC-style episode: infer a latent rule from training input/output pairs and apply it to a test input. (B) Exact-match accuracy across models on ARC-TGI-50N .
Figure 5. Per-generator performance across models (200 generators). Heatmap of model accuracy by generator. Models are ordered by mean accuracy (y-axis) and generators are ordered by Qwen3-30B accuracy (x-axis), revealing shared easy/hard regions and sparse success on the hardest generators.
Figure 6. Per-generator accuracy for Qwen3-30B. Accuracy over generators (50 samples per generator), highlighting broad coverage with a long-tail of difficult task families.
Figure 7. Per-generator comparison: Qwen3-30B vs. Claude Sonnet 4.5. Each point summarizes accuracy on a generator (50 samples), showing largely consistent generator difficulty with occasional reversals.
Figure 8. Core-knowledge priors vs. accuracy. Distribution of dominant priors across accuracy bins for Qwen3-30B and Sonnet 4.5 (manual annotation on stratified samples).
Figure 9. Post-fine-tuning task coverage across 200 generators. Stacked bars summarize task-level accuracy across generators after fine-tuning on ARC-TGI-50N .
Figure 10. Fine-tuning effects on ID vs. OOD evaluation. Accuracy for pre-trained models on ARC-TGI (PT-ARC-TGI), after ARC-TGI fine-tuning evaluated ID (FT-ARC-TGI ID) and on ARC-AGI-1 eval (FT-ARC-TGI OOD), and after fine-tuning on ARC-AGI-1 train evaluated on ARC-AGI-1 eval (FT-ARC-AGI-1 OOD).
Figure 11. Per-generator change after ARC-TGI fine-tuning. Number of generators whose accuracy improves, declines, or remains unchanged (per model).
ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) benchmarks the ability to infer and apply abstract rules from sparse examples.
ARC-AGI-2 is a benchmark of human-intuitive visual puzzles that measures a machine's ability to generalize from limited examples, interpret symbolic meaning, and flexibly apply rules in varying contexts. In this paper, we discuss our approach to solving the ARC-AGI-2 puzzles with TinyLM, with additional fine-tuning at test time, including Test-Time-Training (TTT) and Products of Experts (POE). Our model achieves 96.1% accuracy on the training set and 21.7% accuracy on the evaluation set.
We present ARCANA, a collaborative multi agent framework for solving ARC AGI 2 tasks under strict test time and hardware constraints. ARCANA decomposes each task into iterative perception, hypothesis generation, symbolic execution, and reflective refinement. A perceptual grounding agent builds object centric scene graphs from raw grids, a latent program policy proposes diverse DSL programs, a symbolic executor verifies candidates on demonstrations, and a reflective agent synthesizes failure driven feedback for the next turn. These agents communicate through a shared differentiable blackboard and are scheduled by a learned meta controller. The design combines structured program search with adaptive multi turn correction, improving reasoning efficiency and solution quality on challenging abstract transformation tasks.
Kunbo Zhang, Lei Fu, Zeyu Wang +2
Columbia University New York, USA · Independent Researcher San Jose, USA · University of California, Los Angeles Los Angeles, USA +2
The Abstraction and Reasoning Corpus (ARC) contains tasks that require summarizing patterns from limited grid samples and predicting output grids. Recently, many large language model based approaches have attempted to transform it into a text-based reasoning task. However, methods based on open-source models have generally yielded unsatisfactory results, while those relying on closed-source models are too costly. Current efforts mainly focus on data augmentation, constructing ARC-like data for more comprehensive supervised fine-tuning. In this work, we argue that solving ARC-like problems requires not only positive sample supervision but also the ability to improve model reasoning by distinguishing negative samples. To this end, we draw on the idea of preference alignment and propose DiARC, a method that constructs preference pairs to enable the model to distinguish between them. Specifically, we propose three ways to construct negative samples, including output-level visual transformations, DSL-level rule inversion, and task-specific rule editing. The resulting negative samples provide informative near-miss alternatives while keeping the observed demonstrations unchanged. Experimental results across multiple ARC-like benchmarks show that DiARC consistently improves performance over baseline models. The code is released at https://github.com/szu-tera/DiARC.
Yuxuan Yang, Feiyang Li, Yile Wang
College of Computer Science and Software Engineering, Shenzhen University