PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

Claas Beger; Ryan Yi; Melanie Mitchell

cs.AISep 23, 2026

PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

Authors: Claas Beger, Ryan Yi, Melanie Mitchell

Organizations: Santa Fe Institute

Abstract

The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format fails to evaluate the diversity of abilities that genuine abstract skill acquisition should enable. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task's underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin employs programmatic methods to generate new task instances and transform given inputs for a given ARC task, enabling dynamic generative sampling beyond fixed input-output pairs. Across five state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 25-52 percentage-point performance gap between standard ARC evaluation and evaluation on PotARCin, and find that multi-dimensional evaluation reorders models that standard accuracy ranks alike. We further investigate effects of generative sampling, difficulty of corruption types, and questions of self-consistency, showing that models frequently contradict their own formalized rule even where they have stated it correctly. We also introduce P-ARC, a held-out hand-crafted test set, on which models achieve 1-8% accuracy across all five dimensions, underscoring the importance of more holistic evaluations of abstract reasoning capabilities.

Figures & tables

Figure 1: Consistency and mimetic similarity boundaries for ARC-AGI-1 task a85d4709 .

Figure 2: Overview of the PotARCin benchmark. The shown ARC task is 890034e9 , for which GPT-5.4 solved the standard test-grid task but failed all additional dimensions.

Figure 3: Performance across all seven measures for 400 ARC-AGI-1 training tasks and 50 P-ARC tasks, pooled over two runs (

n{=}800

and

n{=}100

task evaluations); whiskers are Wilson

95\%

intervals. Both panels share a vertical scale. Exact values are in Table 3 .

Figure 4: Results of an increased sampling budget across 20 ARC-AGI-1 training tasks that GPT-5.4 initially solved across all five PotARCin dimensions. “Hard full-task accuracy” is the percentage of tasks passing all sampled configurations up to that budget.

	ARC-AGI-1						P-ARC
Model	Human	Corrupted verifier	Correct	Color flip	Retrieval	Instance mismatch	Human	Corrupted verifier	Correct	Color flip	Retrieval	Instance mismatch
Claude 4.6 Opus	20.74	11.14	1.84	8.76	4.52	4.59	65.03	12.50	4.93	28.57	7.14	12.22
GPT-5.4	8.21	2.30	5.06	6.53	3.15	1.85	47.22	0.00	16.67	0.00	0.00	3.26
Gemini 3.1 Pro	16.37	8.99	5.48	2.82	2.17	6.81	60.42	9.38	22.92	14.29	2.38	7.61
Kimi K2.5	18.54	11.24	5.98	25.78	10.79	7.75	61.11	14.06	12.59	42.86	14.29	11.83
MiniMax M2.5	31.32	26.62	3.66	40.23	40.43	18.12	70.83	23.44	12.50	50.00	30.95	23.91

Table 1: Failure rates by corruption type and dataset , pooled over both runs. Entries are percentages: the fraction of candidates of each type the model labelled incorrectly. For corrupted candidates, failure means accepting an invalid pair; for valid candidates (“Correct”) it means rejecting a valid pair. Bold marks the lowest rate in each column. The last row gives the median number of candidates of that type per model; counts vary by

\leq 1\%

across models. Wilson

95\%

intervals are in Table 8 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Examples of corruption types generated on the ARC-AGI-1 training set. Correct output grids are shown to the right of each corrupted pair, outlined in green. Left: H-ARC corruptions are errors made by human participants in the H-ARC dataset. Middle: Instance mismatch corruptions replace the correct output grid with a similar output from the generator dataset, selected using an edit-distance metric. Right: Corrupted verifiers produce outputs from perturbed verifier programs, either by deleting AST branches or reassigning intermediate variables.

Panel of Figure 6 — Figure 6: Examples of color flip and retrieval corruptions. Top: Color flip corruptions change the color of a sampled cell or small connected group in either the input or output grid. Bottom: Retrieval corruptions use embedding-based lookup to retrieve tasks with similar underlying rules, from which an input-output pair is sampled.

Candidate type	ARC-AGI-1	P-ARC
Valid (dynamic correct)	0.250	0.250
Human error (H-ARC / P-ARC)	0.235	0.265
Instance mismatch	0.215	0.215
Corrupted verifier	0.130	0.160
Retrieval	0.103	0.073
Color flip	0.067	0.037

Appendix

Table 2: Sampling weights over valid pairs and corruption types used to construct Classification candidates. The weights differ between datasets to reflect differences in corruption quality: Retrieval is less suitable for the smaller, distribution-shifted P-ARC set (lower embedding-similarity scores), and Color Flip is less challenging because P-ARC contains fewer color-transformation rules, whereas manually validated P-ARC verifiers and human errors are comparatively reliable. The overall split between valid and invalid candidates is held fixed.

Model	Output grid	Definition	Classification	Generation	Editing	Inversion	Full-task	Indep.
ARC-AGI-1 Train
GPT-5.4 Low	84.2 [81.6, 86.6]	73.6 [70.5, 76.6]	83.8 [81.0, 86.1]	77.2 [74.2, 80.0]	84.0 [81.3, 86.4]	85.1 [82.5, 87.4]	46.6 [43.2, 50.1]	34.0
Gemini 3.1 Pro Low	89.1 [86.8, 91.1]	79.2 [76.3, 81.9]	75.9 [72.8, 78.7]	88.0 [85.6, 90.1]	85.8 [83.2, 88.0]	88.2 [85.8, 90.3]	58.2 [54.8, 61.6]	40.0
Claude 4.6 Opus Low (120K)	87.6 [85.2, 89.7]	66.8 [63.4, 69.9]	71.5 [68.3, 74.5]	78.9 [75.9, 81.6]	88.2 [85.8, 90.3]	91.0 [88.8, 92.8]	37.6 [34.3, 41.0]	30.2
Kimi K2.5	63.4 [60.0, 66.6]	64.5 [61.1, 67.7]	63.5 [60.1, 66.8]	71.8 [68.5, 74.8]	79.6 [76.7, 82.3]	85.8 [83.2, 88.0]	38.6 [35.3, 42.0]	20.1
MiniMax M2.5	72.9 [69.7, 75.8]	56.6 [53.2, 60.0]	42.4 [39.0, 45.8]	64.6 [61.2, 67.9]	68.6 [65.3, 71.7]	76.6 [73.6, 79.4]	20.6 [18.0, 23.6]	8.1

Appendix

Table 3: Model performance on ARC-AGI-1 Train and P-ARC, pooled over two independent runs (

n{=}800

and

n{=}100

task evaluations). Entries are percentages; brackets are pooled Wilson

95\%

intervals. Indep. is the product of the five dimension accuracies, i.e. the full-task rate expected if per-dimension pass events were independent. Bold marks the best full-task mean within each dataset block. These are the exact values underlying Figure 3 .

	Classification		Constrained generation		Inversion
Model	answer $\checkmark$	answer $\times$	answer $\checkmark$	answer $\times$	answer $\checkmark$	answer $\times$
ARC-AGI-1 Train
GPT-5.4	99.3 (3095)	14.4 (90)	97.0 (526)	31.9 (113)	98.2 (559)	65.0 (80)
Gemini 3.1 Pro	99.3 (3211)	3.2 (154)	99.4 (622)	48.0 (50)	99.2 (608)	73.0 (63)
Claude 4.6 Opus	99.2 (2759)	4.1 (196)	98.4 (491)	28.0 (100)	98.5 (545)	21.7 (46)
Kimi K2.5	99.6 (2503)	8.9 (202)	98.4 (446)	43.8 (96)	99.4 (506)	35.1 (37)

Appendix

Table 4: Self-consistency conditioned on a correct Definition. Percentage of cases in which the model’s Definition program agrees with its own answer in the given dimension, split by whether that answer is correct (

\checkmark

) or incorrect (

\times

). Denominators in parentheses. Pooled over both runs.

(a) All cases
	ARC-AGI1			P-ARC
Model	Classification	Constrained generation	Inversion	Classification	Constrained generation	Inversion
Claude 4.6 Opus	88.23	74.56	78.61	66.67	55.00	62.00
GPT-5.4	93.62	77.01	83.17	68.37	37.76	31.63
Gemini 3.1 Pro	90.48	87.17	87.15	63.43	47.47	30.30
Kimi K2.5	83.50	66.67	71.05	55.80	22.22	19.19

Appendix

Table 5: Unconditional self-consistency by dataset and model. Panel (a) reports the percentage of cases in which the Definition program agrees with the model response in the corresponding dimension. Panel (b) restricts to model error cases.

Dimension / Measure	Sample 1	Sample 2	Sample 5	Sample 10
Avg. full-task acc.	65.0 $\pm$ 24.7	65.0 $\pm$ 20.2	71.0 $\pm$ 13.2	70.0 $\pm$ 11.0
Hard full-task acc.	65.0	45.0	35.0	25.0
Definition	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0
Classification	90.0 $\pm$ 7.1	90.0 $\pm$ 5.8	92.0 $\pm$ 5.2	92.0 $\pm$ 6.1
Generation	80.0 $\pm$ 14.1	87.5 $\pm$ 10.4	85.0 $\pm$ 8.8	83.5 $\pm$ 7.4
Editing	95.0 $\pm$ 3.5	90.0 $\pm$ 7.6	95.0 $\pm$ 5.8	94.5 $\pm$ 5.9

Appendix

Table 6: GPT-5.4 sampling budget trends over 20 ARC-AGI-1 tasks, each evaluated under 10 sampled configurations across all five dimensions (200 complete five-dimension evaluations). Entries are percentages, reported as cumulative mean

\pm

sample standard deviation across sampled configurations.

Model	Run 1	Run 2	Shared	Jaccard	Task agreement
GPT-5.4 Low	185	188	146	0.64	79.8%
Gemini 3.1 Pro Low	239	227	205	0.79	86.0%
Claude 4.6 Opus Low (120K)	160	141	107	0.55	78.2%
Kimi K2.5	162	147	102	0.49	73.8%
MiniMax M2.5	76	89	57	0.53	87.2%

Appendix

Table 7: Run-to-run stability of full-task pass sets. For each model, the number of ARC-AGI-1 tasks passing all five dimensions in each run, their intersection and union, the Jaccard overlap, and the percentage of tasks on which the two runs agree (both pass or both fail). P-ARC is omitted: with 0–4 passing tasks per run, the overlap statistic is not meaningful.

Model	Human	Corrupted verifier	Correct	Color flip	Retrieval	Instance mismatch
ARC-AGI-1
Claude 4.6 Opus	20.74 [17.6, 24.3]	11.14 [8.5, 14.5]	1.84 [1.2, 2.8]	8.76 [6.2, 12.2]	4.52 [2.9, 7.0]	4.59 [3.5, 6.0]
GPT-5.4	8.21 [6.2, 10.8]	2.30 [1.3, 4.2]	5.06 [3.9, 6.5]	6.53 [4.4, 9.6]	3.15 [1.8, 5.3]	1.85 [1.2, 2.8]
Gemini 3.1 Pro	16.37 [13.5, 19.7]	8.99 [6.6, 12.0]	5.48 [4.3, 6.9]	2.82 [1.5, 5.1]	2.17 [1.1, 4.1]	6.81 [5.5, 8.5]
Kimi K2.5	18.54 [15.5, 22.0]	11.24 [8.6, 14.6]	5.98 [4.7, 7.5]	25.78 [21.5, 30.6]	10.79 [8.2, 14.1]	7.75 [6.3, 9.5]
MiniMax M2.5	31.32 [27.6, 35.3]	26.62 [22.7, 31.0]	3.66 [2.7, 4.9]	40.23 [35.2, 45.4]	40.43 [35.8, 45.2]	18.12 [15.9, 20.5]

Appendix

Table 8: Failure rates by corruption type with pooled Wilson

95\%

confidence intervals, over both runs. “ items per model ” is the median candidate count for that type; counts vary by at most

1\%

across models. Several P-ARC cells rest on few candidates (14 for color flip, 42 for retrieval) and the corresponding intervals are correspondingly wide.

Model	Suite	Programs	Failed to load	%
GPT-5.4	ARC-AGI-1	800	4	0.5
Gemini 3.1 Pro	ARC-AGI-1	800	4	0.5
Claude 4.6 Opus	ARC-AGI-1	800	10	1.2
Kimi K2.5	ARC-AGI-1	800	8	1.0
MiniMax M2.5	ARC-AGI-1	800	29	3.6
GPT-5.4	P-ARC	100	2	2.0

Appendix

Table 9: Share of collected Definition programs that fail to load.

Abstract Reasoning

See all 12 papers

Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning
Oct 2, 2025Claas Beger, Ryan Yi, Shuhao Fu +5Language Model Reasoning Evaluation Explainability Evaluation
Santa Fe Institute, New Mexico, USA · Sandia National Laboratories, New Mexico, USA · Advanced Micro Devices Inc., Texas, USA
The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning
Mar 9, 2026Sahar Vahdati, Andrei Aioanei, Haridhra Suresh +1Abstract Reasoning ARC-AGI
TIB - Leibniz Information Centre for Science and Technology · L3S Research Center, Leibniz University of Hannover · Dresden University of Technology +1
A Systematic Study of Small Language Models on Abstract Reasoning Tasks
Oct 6, 2026Nur A Zarin Nishat, Jens Lehmann, Andrei Aioanei +1OOD Generalization Language Model Robustness
Leibniz University of Hanover TIB – Leibniz Information Centre for Science and Technology and University Library · Amazon

Oct 2, 2025J/K move · Enter open · S save

Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning

Claas Beger, Ryan Yi, Shuhao Fu, Kaleda Denton+4

Santa Fe Institute, New Mexico, USA · Sandia National Laboratories, New Mexico, USA · Advanced Micro Devices Inc., Texas, USA

Language Model Reasoning Evaluation Explainability Evaluation

Open paper AskPDF arXiv

AI reasoning models have exceeded human performance on the ARC-AGI-1 benchmark, but does that mean state-of-the-art models have the underlying competence (humanlike abstract reasoning) that the benchmark was designed to test? Here we investigate the abstraction abilities of AI models using the closely related but simpler ConceptARC benchmark. Our evaluations vary input modality (textual vs. visual), use of external Python tools, and reasoning effort. Beyond output accuracy, we evaluate the natural-language rules that models generate to explain their solutions, enabling us to assess whether models recognize the abstractions that ConceptARC was designed to elicit. We show that the best models' rules are frequently based on less abstract, more domain-specific concepts, capturing intended abstractions considerably less often than humans. In the visual modality, AI models' output accuracy drops sharply; however, our rule-level analysis reveals that a substantial share of their rules capture intended abstractions, even as the models struggle to apply these concepts to generate correct solutions. In short, we show that using performance (accuracy) alone can substantially overestimate AI competence in textual modalities while underestimating it in visual modalities, an illustration of the risk of mistaking performance for competence.

Abstract

Abstract Reasoning

Explore similar work

Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning

The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning

A Systematic Study of Small Language Models on Abstract Reasoning Tasks