ConceptARC at Scale: Studying Conceptual Understanding in Reasoning Models
Organizations: LMU Munich & MCML · University of Li`ege · New York University · Santa Fe Institute ·
Abstract
Large reasoning models (LRMs) perform well on challenging benchmarks, yet it remains unclear whether they form generalizable abstractions across conceptually related reasoning tasks. The Abstraction and Reasoning Corpus (ARC) provides a controlled testbed for studying abstraction, requiring solvers to infer transformation rules over two-dimensional grids from a small set of examples. While ARC evaluates rule abstraction within individual tasks, different tasks may share higher-level structure despite requiring different transformationsfor example, several tasks may involve reasoning about objects contained within other objects. Benchmarks such as ConceptARC capture this structure by grouping ARC-style tasks into recurring spatial and semantic concepts, but they often contain only a few manually constructed tasks per concept. We introduce ConceptARC at Scale, a generator for creating thousands of ARC-style tasks targeting a specific concept. Using this generator, we construct ConceptARC-100, a dataset of 1,600 new, manually reviewed tasks, with 100 tasks for each of the 16 concepts defined by ConceptARCten times the size of the original benchmark. We evaluate a range of recent LRMs and models trained from scratch on ConceptARC-100 to characterize which concepts they solve reliably and where they struggle. We further analyze the models' internal representations and find clear semantic structure: task representations cluster according to the operations applied to objects, such as recoloring, removal, or translation, even when realized through substantially different rules and scenarios. Together, our results suggest that models form stable abstractions of transformation operations that generalize across diverse tasks.
Figures & tables
| Concept | Gemma4-31B | Qwen3.5-27B | Qwen3.5-9B | GPT-OSS-120B | GPT-OSS-20B | Qwen3-32B | Qwen3-8B | Phi4-Reas. | VARC | TRM |
|---|---|---|---|---|---|---|---|---|---|---|
| AboveBelow | 75.30 | 73.40 | 43.90 | 56.97 | 37.17 | 13.97 | 6.90 | 13.17 | 37.07 | 14.63 |
| Center | 80.03 | 83.27 | 61.70 | 79.93 | 60.77 | 24.80 | 5.77 | 19.17 | 16.73 | 3.97 |
| FilledNotFilled | 60.53 | 71.13 | 32.43 | 48.20 | 23.37 | 9.83 | 2.40 | 4.33 | 16.90 | 7.27 |
| HorizontalVertical | 64.23 | 73.17 | 34.87 | 47.83 | 27.50 | 11.77 | 5.20 | 7.10 | 9.33 | 10.57 |
| InsideOutside | 61.30 | 58.03 | 31.07 | 33.77 | 26.37 | 9.17 | 2.80 | 3.93 | 38.53 | 13.60 |
| SameDifferent | 44.10 | 36.27 | 10.97 | 17.70 | 10.17 | 5.23 | 3.27 | 2.70 | 12.60 | 7.13 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Temperature | Top- | Top- | Presence penalty | Total output budget | Reasoning budget |
|---|---|---|---|---|---|---|
| Gemma4-31B | 1.0 | 0.95 | 64 | 0.0 | 65,536 | 49,152 |
| Qwen3.5-27B / Qwen3.5-9B | 1.0 | 0.95 | 20 | 1.5 | 65,536 | 49,152 |
| GPT-OSS-120B / GPT-OSS-20B | 1.0 | 1.00 | 0 | 0.0 | 65,536 | 49,152 |
| Qwen3-32B / Qwen3-8B | 0.6 | 0.95 | 20 | 1.5 | 32,768 | 20,480 |
| Phi4-Reasoning-Plus | 0.8 | 0.95 | 50 | 0.0 | 18,432 | 15,360 |
| Concept | Gemma4-31B | Qwen3.5-27B | Qwen3.5-9B | GPT-OSS-120B | GPT-OSS-20B | Qwen3-32B | Qwen3-8B | Phi4-Reas.+ | VARC | TRM |
|---|---|---|---|---|---|---|---|---|---|---|
| AboveBelow | ||||||||||
| Center | ||||||||||
| FilledNotFilled | ||||||||||
| HorizontalVertical | ||||||||||
| InsideOutside | ||||||||||
| SameDifferent |
| Concept | Gemma4-31B | Qwen3.5-27B | Qwen3.5-9B | GPT-OSS-120B | GPT-OSS-20B | Qwen3-32B | Qwen3-8B | Phi4-Reas.+ |
|---|---|---|---|---|---|---|---|---|
| AboveBelow | ||||||||
| Center | ||||||||
| FilledNotFilled | ||||||||
| HorizontalVertical | ||||||||
| InsideOutside | ||||||||
| SameDifferent |
| Type | Concept | Description |
|---|---|---|
| Predicate | Above and Below | The transformation depends on cells or objects lying above or below a visible reference, such as a line, band, diagonal, or distinguished object. Content on a specified side may be selected, removed, recolored, extended, or moved. Unlike Top and Bottom 2D , the relation is defined by a reference rather than solely by vertical rank. |
| Center | The transformation uses the geometric center or midpoint of a grid, object, line, gap, container, or pair of objects. It may mark, extract, remove, or fill the center, move an object relative to it, or otherwise perform an operation determined by the center. | |
| Filled and Not Filled | The defining distinction is whether a region or object is solid or hollow. The transformation may select, label, fill, empty, or invert objects according to their interior state. | |
| Horizontal and Vertical | The operation depends on horizontal versus vertical orientation or on an object’s dominant axis. Horizontal and vertical objects, internal stripes, marker alignments, or directions may be selected, removed, recolored, or extended differently. | |
| Inside and Outside | Containment determines which cells or objects are relevant: for example, whether they lie inside, outside, or between containers, outlines, or boundary lines. The selected content may then be retained, removed, recolored, counted, extracted, or otherwise transformed. | |
| Top and Bottom 2D | The vertical position or rank of whole objects in a two-dimensional grid determines the operation, such as selecting, extracting, marking, rearranging, or otherwise manipulating the topmost, bottommost, higher, lower, or intermediate object. Unlike Above and Below , this concept normally compares objects by vertical rank without an explicit separating reference. |
Spatial Reasoning Benchmarks
Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.
See all 176 papers