cs.AIOct 2, 2025
SaveDistinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning
Organizations: Santa Fe Institute, New Mexico, USA · Sandia National Laboratories, New Mexico, USA · Advanced Micro Devices Inc., Texas, USA
Abstract
AI reasoning models have exceeded human performance on the ARC-AGI-1 benchmark, but does that mean state-of-the-art models have the underlying competence (humanlike abstract reasoning) that the benchmark was designed to test? Here we investigate the abstraction abilities of AI models using the closely related but simpler ConceptARC benchmark. Our evaluations vary input modality (textual vs. visual), use of external Python tools, and reasoning effort. Beyond output accuracy, we evaluate the natural-language rules that models generate to explain their solutions, enabling us to assess whether models recognize the abstractions that ConceptARC was designed to elicit. We show that the best models' rules are frequently based on less abstract, more domain-specific concepts, capturing intended abstractions considerably less often than humans. In the visual modality, AI models' output accuracy drops sharply; however, our rule-level analysis reveals that a substantial share of their rules capture intended abstractions, even as the models struggle to apply these concepts to generate correct solutions. In short, we show that using performance (accuracy) alone can substantially overestimate AI competence in textual modalities while underestimating it in visual modalities, an illustration of the risk of mistaking performance for competence.
Figures & tables
Figure 1: Two tasks from the ConceptARC benchmark. A task consists of a small number of demonstrations of a transformation and one test grid. In our study, the solver is asked to generate a rule that describes the transformations and apply that rule to the test grid to generate an output grid.
| Model | low | medium | low +T | medium +T | |
| o3 | t | 68.3 | 77.1 | 67.9 | 75.6 |
| v | 6.7 | 5.6 | 18.1 | 29.2 | |
| Claude | t | – | 60.2 | – | 55.0 |
| v | – | 5.2 | – | 6.9 | |
| Gemini | t | – | 66.0 | – | 60.4 |
| v | – | 4.2 | – | 5.8 |
Table 1: Reasoning models: Output-grid accuracy (pass@1) for ConceptARC for humans and models. Accuracy is shown in %. Textual (t) and Visual (v) results are reported on separate rows, and settings with Python tools enabled are marked with +T. Settings with “–” were not tested. See section 3 for details of experiments. Bold numbers correspond to the highest visual and textual scores in each column.
Figure 2: Results of rule evaluations. For each model, as well as for humans, two bars are shown, representing the percentages of correct and incorrect grid outputs across the 480 ConceptARC tasks. Model evaluations are reported for both textual and visual settings. Each bar shows the fraction of tasks for which the rule is correct-intended, correct-unintended, and incorrect. The gray regions represent rules that could not be classified, further described in the section on Rule Evaluation. Detailed percentages for each bar are given in Appendix K .
Figure 3: Proportion of rules containing at least one “objectness" term (left) or grid-specific term (right), for humans and AI models. Objectness terms: object, shape, square, rectangle, rectangular, line, triangle, triangular, pyramid, diamond, cross, polygon. Grid-Specific terms: cell, pixel, row, column, value, number, neighbor, neighborhood, index, subgrid, coordinate. The number of rules analyzed for humans, Claude, o3, and Gemini was 2456, 261, 363, and 290, respectively.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Non-Reasoning model | No Python Tools | With Python Tools |
|---|---|---|
| Textual / Visual | Textual / Visual | |
| o4-mini | 70.8 / 8.1 | 77.7 / 25.0 |
| GPT-4o | 14.6 / 0.0 | 8.3 / 0.2 |
| Llama 4 Scout | 6.7 / 0.0 | - |
| Qwen 2.5 VL 72B | 9.2 / 0.0 | - |
Table 2: o4-mini (medium effort) and non-reasoning models: Output-grid accuracy (pass@1) on Concept-ARC across models and experimental settings. Accuracy is shown in %. Each cell shows accuracy in the textual / visual modality. Temperature is set to 1.0 for o4-mini and 0.0 for non-reasoning models. The Llama and Qwen interfaces did not provide options for Python tool use.
| Model | Setting | Textual | Visual |
| Original / Re-assessed | Original / Re-assessed | ||
|---|---|---|---|
| o3 | low effort | 68.3 / 69.4 | 6.5 / 6.5 |
| o3 | medium effort | 77.1 / 77.1 | 5.6 / 5.6 |
| o3 | low effort + tools | 67.9 / 68.1 | 18.2 / 18.2 |
| o3 | medium effort + tools | 75.6 / 75.6 | 29.2 / 29.2 |
| o4-mini | low effort | 52.1 / 59.6 | 3.8 / 3.8 |
Table 3: Output grid accuracies with incorrect grid formats included. For each model and setting, we give original accuracy / re-assessed accuracy . Original accuracies are from Table 2 and Table 1 .
| Model | Textual | Visual | |
| (Medium + Tools) | |||
| RGA | RGA | VE | |
| o3 | 98.1 | 97.4 | 49.1 |
| Claude | 92.4 | 96.6 | 59.7 |
| Gemini | 94.5 | 93.6 | 77.3 |
Table 4: Percentage of tasks in which models exhibited either “rule–grid alignment” (RGA)—the generated rule accurately described the generated output grid—or, in visual settings, a “visual error” (VE). The medium-effort + tools setting was used for all models. Tasks with missing or invalidly formatted output grids and/or non-responsive or unclear rules are excluded from these percentages.
Figure 4: Overview of different error types for o3 in different experimental settings. For each setting, the left bar shows textual modality results and the right bar shows visual modality results. The most common error type is a simple mismatch error, in which the output grid and the ground truth grid are not identical, including cases of incorrect grid dimensions and single-cell mismatches. We also encountered parsing errors, most often originating from incorrect formatting (see Appendix E )as well as from grids with rows of differing lengths.
Figure 5: Correct-unintended rule exploiting numerical properties of the integer color encoding. o3, textual, low effort with tools, Above vs. Below (AboveBelow5, Test 1).
Figure 6: Rule using local neighborhood connectivity as a proxy for containment. Gemini 2.5 Pro, textual, medium effort, Inside vs. Outside (InsideOutside4, Test 1).
Figure 7: Correct-unintended rule using local neighborhood connectivity as a proxy for objectness. o3, textual, low effort with tools, CleanUp (CleanUp3, Test 2).
Figure 8: Correct-unintended algorithmic rule. o3, visual, medium effort with tools, ExtractObjects (ExtractObjects10, Test 1).
Figure 9: Correct-unintended rule using a density heuristic as a proxy for 3D spatial position. Claude Sonnet 4, textual, medium effort with tools, Top vs. Bottom 3D (TopBottom3D6, Test 1).
Figure 10: Correct-unintended, demonstration-specific rule. o3, medium effort with tools, Complete Shape (CompleteShape8, Test 1).
Figure 11: Correct-unintended rule exploiting constrained solution space. o3, textual, low effort, Horizontal vs. Vertical (HorizontalVertical10, Test 1.
Figure 12: Results of rule evaluations for o3 across all settings. As in Figure 2 , two bars showing the percentage of correct and incorrect output grids are included for each setting, with each bar showing the fraction of tasks for which the generated rule is correct-intended, correct-unintended, and incorrect. The actual percentages corresponding to regions on the bars are given in Appendix K .
| o3 | Claude | Gemini | |
| Aggregate rates across three passes | |||
| Output-grid accuracy | 74.3 [70.8, 77.8] | 61.1 [55.6, 66.7] | 68.8 [64.6, 72.9] |
| Correct-intended rule | 59.9 [54.9, 64.8] | 59.0 [54.2, 63.9] | 56.1 [51.5, 60.6] |
| Correct-unintended rule | 31.7 [26.8, 36.6] | 19.4 [15.3, 23.6] | 36.4 [31.8, 40.9] |
| Incorrect rule | 8.5 [4.9, 12.0] | 18.8 [15.3, 22.9] | 7.6 [3.8, 11.4] |
| Unclear rule | 0.0 [0.0, 0.0] | 2.8 [0.7, 4.9] | 0.0 [0.0, 0.0] |
Table 5: Repeated-run results. All entries are percentages; bracketed values are 95% percentile-bootstrap confidence intervals. Rule-category rates are computed excluding nonresponses.
| Model (Modality) | Correct Grid | Incorrect Grid | ||||||
| Correct- Intended | Correct- Unint. | Incorrect | Not- Classif. | Correct- Intended | Correct- Unint. | Incorrect | Not- Classif. | |
|---|---|---|---|---|---|---|---|---|
| o3 (Textual) | 72.7 | 20.9 | 6.3 | 0.0 | 9.4 | 52.1 | 35.9 | 2.6 |
| o3 (Visual) | 70.0 | 19.3 | 10.7 | 0.0 | 27.6 | 18.2 | 45.9 | 8.2 |
| Claude Sonnet 4 (Textual) | 80.3 | 9.5 | 9.1 | 1.1 | 29.6 | 20.8 | 36.1 | 13.4 |
| Claude Sonnet 4 (Visual) | 57.6 | 6.1 | 33.3 | 3.0 | 11.2 | 2.7 | 71.4 | 14.8 |
| Gemini 2.5 Pro (Textual) | 72.4 | 19.7 | 7.9 | 0.0 | 5.8 | 25.3 | 65.3 | 3.7 |
Table 6: Data used to create Figure 2
| o3 Setting (Modality) | Correct Grid | Incorrect Grid | ||||||
| Correct- Intended | Correct- Unint. | Incorrect | Not- Classif. | Correct- Intended | Correct- Unint. | Incorrect | Not- Classif. | |
|---|---|---|---|---|---|---|---|---|
| Low effort (Textual) | 72.3 | 19.2 | 8.5 | 0.0 | 13.2 | 23.7 | 59.2 | 3.9 |
| Low effort (Visual) | 81.3 | 3.1 | 15.6 | 0.0 | 28.1 | 4.7 | 60.7 | 6.5 |
| Medium effort (Textual) | 68.4 | 22.4 | 9.2 | 0.0 | 8.2 | 48.2 | 42.7 | 0.9 |
| Medium effort (Visual) | 55.6 | 11.1 | 33.3 | 0.0 | 28.9 | 5.5 | 57.4 | 8.2 |
| Low effort + tools (Textual) | 67.5 | 24.2 | 8.3 | 0.0 | 6.5 | 35.1 | 47.4 | 11.0 |
Table 7: Data used to create Figure 12 .
| Concept | Gemini 2.5 Pro | o3 | o4-mini | Claude Sonnet 4 | Human |
| AboveBelow | 60 | 90 | 83.3 | 63.3 | 69 |
| Center | 70 | 93.3 | 96.7 | 83.3 | 84 |
| CleanUp | 23.3 | 46.7 | 60 | 46.7 | 89 |
| CompleteShape | 56.7 | 70 | 66.7 | 50 | 71 |
| Copy | 66.7 | 70 | 90 | 56.7 | 78 |
| Count | 86.7 | 80 | 80 | 76.7 | 61 |
Table 8: Concept performance (Textual): Per-concept accuracy (%) on Concept-ARC using medium effort + tools. Highest accuracy per concept is shown in bold.
| Concept | Gemini 2.5 Pro | o3 | o4-mini | Claude Sonnet 4 | Human |
| AboveBelow | 0 | 20 | 10 | 0 | 69 |
| Center | 6.7 | 43.3 | 26.7 | 6.7 | 84 |
| CleanUp | 10 | 23.3 | 26.7 | 13.3 | 89 |
| CompleteShape | 3.3 | 30 | 23.3 | 16.7 | 71 |
| Copy | 3.3 | 20 | 23.3 | 3.3 | 78 |
| Count | 16.7 | 53.3 | 50 | 0 | 61 |
Table 9: Concept performance (Visual): Per-concept accuracy (%) on Concept-ARC using medium effort + tools. Highest accuracy per concept is shown in bold.
Figure 13: Top row: Two example demonstrations from the concepts with the highest and lowest gaps between human and model performance, Count and CleanUp. Bottom row: Concept-wise output grid accuracy across three reasoning models in a medium with tools setting (note that we compare against the strongest settings in subsection L.1 ).
Figure 14: For each classification (correct-intended, correct-unintended, incorrect), proportion of human- and LLM-generated rules that used objectness-related and grid-specific terms. For humans, there were 2212 correct-intended rules, 122 correct-unintended rules, and 122 incorrect rules in total. For AI models, there were 686 correct-intended rules, 158 correct-unintended rules, and 70 incorrect rules in total.
Figure 15: For each concept group, proportion of rules containing at least one concept-associated term, for humans and for AI models (o3, Claude, and Gemini combined). The concept-associated terms are: AboveBelow: above, below, over, under, top, bottom, up, down, upward, downward, beneath, underneath, lower, upper, higher, topmost, bottommost, uppermost, high, low. Center: center, middle, midpoint, median, central, between, equidistant, halfway, centermost, midway. CleanUp: clean, remove, complete, noise, noisy, random, randomly, scatter, disparate, outlier, extra, anomalous, nonconforming, deviation, isolate, interrupt, mismatch, clutter, delete, replace, eliminate, declutter, miss, break, unbroken, solid. CompleteShape: complete, shape, square, miss, entire, gap, fill, finish, repair, replace, fix, break, incomplete, unbroken, full, pattern, hole, connect, unfinished. Copy: copy, second, double, recreate, duplicate, match, repeat, same, identical, replicate. Count: count, amount, total, for each, for every, how many, however many, number of, that many. ExtendToBoundary: extend, boundary, reach, touch, edge, hit, meet, toward, towards, top, bottom, end, side, far, furthest, wall, stop, stretch, contact, border, barrier, extension, topmost, bottommost, leftmost, rightmost. ExtractObjects: extract, object, shape, rectangle, rectangular, copy, recreate, replicate, reproduce, duplicate, crop. FilledNotFilled: fill, outline, inside, middle, interior, border, enclose, surround, hollow, contain, hole, solid, solidly, inner, interior, exterior, empty, miss, gap, open, frame, internal, external, space, within, outer, full. HorizontalVertical: horizontal, vertical, orient, orientation, left to right, top to bottom, up and down. InsideOutside: inside, outside, in, out, within, border, frame, enclose, contain, inner, outer, interior, exterior, innermost, outermost, internal, external. MoveToBoundary: move, boundary, touch, slide, toward, border, contact, edge, side, wall, reach, shift, bottom, top, barrier, topmost, bottommost, hit, meet, top, bottom, end, far, furthest, stop, contact, border, leftmost, rightmost, adjacent. Order: order, small, large, bottom, top, ascend, count, sort, arrange, rearrange, reorder, reorganize, sequence, big, descend, higher, lower, high, low, increase, decrease, least, great, few, most. SameDifferent: same, different, copy, match, identical, unique, repeat, differ, duplicate, unique, replicate. TopBottom2D: top, bottom, on, topmost, bottommost, under, over, lower, low, higher, high, upper, uppermost, lowermost, above, below, underneath, beneath. TopBottom3D: top, bottom, behind, front, back, forward, cover, under, over, underneath, above, below, topmost, bottommost, uppermost, lowermost, beneath, overlay, foreground, background.
| Category | Modality | Covered | Percentage |
| Humans | Overall | 476 | 99.17 |
| o3 | Textual | 412 | 85.83 |
| o3 | Visual | 281 | 58.54 |
| Claude | Textual | 346 | 72.08 |
| Claude | Visual | 83 | 17.29 |
| Gemini | Textual | 326 | 67.92 |
Table 10: Correct-intended task coverage: Number of tasks covered correctly by category and modality, with coverage rates listed as a percentage of the 480 total ConceptARC tasks. Here, a task is considered “covered” if the model in question produced a correct-intended rule in any of its solutions for that task in the given modality. The “AnyModel” rows show task coverage aggregated across all three reasoning models, and the entry for humans shows the coverage of tasks for which at least one human subject produced a correct-intended rule.
| Model | low | medium | low +T | medium +T | |
| o3 | t | 68.6 | 77.6 | 68.3 | 75.9 |
| v | 6.7 | 5.7 | 18.2 | 29.4 | |
| Claude | t | N/A | 60.6 | N/A | 55.3 |
| v | N/A | 5.2 | N/A | 6.9 | |
| Gemini | t | N/A | 66.5 | N/A | 60.4 |
| v | N/A | 4.2 | N/A | 5.9 |
Table 11: This table reports the same data as Table 1 , with Order5 removed from the analysis.
| Dataset | Numerical | Symbol |
| ARC-AGI-1 (pass 1) | 77.2% | 63.2% |
| ARC-AGI-1 (pass 2) | 80.8% | 67.1% |
| ARC-AGI-2 (pass 1) | 22.5% | 16.81% |
| ARC-AGI-2 (pass 2) | 23.47% | 13.47% |
Table 12: Accuracies of GPT-5.2 (with medium reasoning effort) on numeric and symbol encodings for ARC-AGI public evaluation sets. For each setting, accuracies of two independent passes are given.