Ceci n'est pas une pipe -- Toward a Symbolic Solver for the Abstraction and Reasoning Corpus Using Evolutionary Computation
Authors: Luca Favole, Giorgio Mongardi, Giovanni Squillero, Alberto Tonda
Organizations: Department of Control and Computer Engineering Politecnico di Torino Corso Duca degli Abruzzi 24, 10129, Torino, Italy · Applied Mathematics and Computer Science Universit´e Paris-Saclay, INRAE 22 place de l’Agronomie, 91120, Palaiseau, France · Complex Systems Institute of Paris CNRS 113 rue Nationale, 75013, Paris, France
The Abstraction and Reasoning Corpus (ARC) is a benchmark designed to assess the capability of Artificial Intelligence (AI) systems to generalize starting from a few samples, a task humans excel at. The ARC-AGI contests held since 2020 have been the focus of intense competition, with participants proposing approaches ranging from traditional symbolic AI to Transformer architectures. Despite a long history of successes, evolutionary algorithms (EAs) were notably absent from the approaches appearing in the main contest. As score-and-search program synthesis approaches, EAs suffer from issues related to choosing the proper representation in the Domain-Specific Language applied to the problem (for example, pixel-level or object-level), and distinguishing between two candidate solutions producing the same outputs. In this work, we present the first fully symbolic, evolutionary-based program synthesis approach submitted to the ARC-AGI-2 main contest. The main contributions are: evolving the grid parser and the output projector together with the program that transforms the grid, so that a task-specific language emerges during program synthesis; grounding the symbols the parser can extract in core knowledge theory from cognitive psychology; and an object-aware fitness that grades partially correct outputs beyond pixel-level discrepancies.
Figures & tables
Figure 1: Example tasks: Each arrow leads from an input grid to its output grid; in each row, the first pairs are the demonstration pairs and the last is the test pair. Sources: github.com/fchollet/ARC-AGI and github.com/arcprize/ARC-AGI-2 .
Figure 2: Different graph representations of the same grid. Parser A uses four-neighbor connectivity and explicitly detects the cavity enclosed by the blue object. Parser B uses eight-neighbor connectivity and omits cavity detection. Consequently, the two green objects merge into one and the cavity node disappears. Only the nodes and relations relevant to this example are shown.
Benchmark
Best run
All runs
ARC-AGI-1 training
197/400
219/400
ARC-AGI-2 evaluation
3/120
3/120
Table 1: Comparison between the best single run and cumulative task coverage across all available runs at pass@2 .
840s
3840s
Benchmark
Tasks
Fit.
@1
@2
Fit.
@1
@2
ARC-AGI-1 training
400
237
182
193
258
183
197
ARC-AGI-1 evaluation
400
146
96
103
164
103
110
ARC-AGI-2 training
1000
405
273
294
455
283
304
ARC-AGI-2 evaluation
120
7
3
3
10
3
3
ARC-AGI-2 tr., new
233
42
13
20
55
19
20
Table 2: Fitted tasks (Fit.) and tasks solved at pass@1 and pass@2 with 840 and 3840s per task: the best of three seeds at 840s , one seed at 3840s . Of the 1000 ARC-AGI-2 training tasks, 767 come from ARC-AGI-1; the last row covers the 233 that do not.
Figure 3: Comparison of the percentage of tasks solved on ARC-AGI with published systems. Our system, FairlySymbolic, was limited to 3840s per task on ARC-AGI-1; on ARC-AGI-2 the figure reports the result of the Kaggle entry. All systems were allowed two attempts per task, except on ARC-AGI-1 training, where icecuber, GPT-4, and DreamCoder used three attempts; and on ARC-AGI-1 evaluation, where GPT-4o used one attempt. It should also be noted that, on ARC-AGI-1 evaluation, the results for o3 refer to a pre-release version using six samples per task, and those for the ARChitects to the 2025 method .
Figure 4: Percentage of tasks solved at pass@1 on the 160-task ARC-AGI-1 subset used by ARGA. Our system used 1 800 s per task.
Rules
0
1
2
3
≥ 4
Fitted
90
296
103
80
62
Solved
100%
95%
77%
64%
47%
Table 3: Fitted ARC-AGI-1 training tasks that are also solved at pass@2 , by the number of rules in the program ( 120s per task, summed over three seeds).
Figure 5: For each of the 400 ARC-AGI-1 training tasks, the number of runs that solve it at pass@2 , over the 55 runs of the parameter study (earlier version of the solver, 120s per task) that allow programs of at least three rules.
Search
Fitted
Solved
Generations
Full fitness
212
184
2600
Coarse fitness
158
153
3900
Random search
98
107
3300
Random, no refinement
74
92
3500
Table 4: Weaker searches on the ARC-AGI-1 training tasks at 120s per task: fitted and solved ( pass@2 ) tasks and median generations per task, best of three seeds (the text compares seed means).
Figure 6: Fitted and solved ARC-AGI-1 training tasks against the number of restarts (earlier version of the solver). Solid: 120s per task, three seeds (dots) for 1 and 5 restarts, one otherwise. Dashed: 480s per task, one seed.
Concept
Before
After
TopBottom2D
70.0
100.0
SameDifferent
66.7
96.7
ExtractObjects
56.7
100.0
⋮
CompleteShape
13.3
96.7
Order
6.7
100.0
Table 5: Percentage of ConceptARC test inputs solved at pass@2 , before and after the extensions.
Figure 7: Percentage of ConceptARC test inputs solved. FairlySymbolic used two attempts with 16 voted searches (600 s per task in total), icecuber used three attempts, and the remaining systems used one.
Figure 8: Qualitative examples from ARC-AGI-1: (a) a successfully solved task; (b) a program fitting all demonstrations but failing on an unseen configuration; (c) a task requiring interpretation of a spatial color legend.
ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) benchmarks the ability to infer and apply abstract rules from sparse examples.
TIB - Leibniz Information Centre for Science and Technology · L3S Research Center, Leibniz University of Hannover · Dresden University of Technology +1