Exploring Human Behavior During Abstract Rule Inference and Problem Solving with the Cognitive Abstraction and Reasoning Corpus
Authors: Caroline Ahn, Quan Do, Leah Bakst, Michael P. Pascale, Joseph T. McGuire, Michael E. Hasselmo, Chantal E. Stern
Organizations: Graduate Program for Neuroscience, Boston University, Boston, MA, USA · Cognitive Neuroimaging Center, Boston University, Boston, MA, USA · Center for Systems Neuroscience, Boston University, Boston, MA, USA · Department of Psychological and Brain Sciences, Boston University, Boston, MA, USA
Humans exhibit remarkable flexibility in abstract reasoning, and can rapidly learn and apply rules from sparse examples. To investigate the cognitive strategies underlying this ability, we introduce the Cognitive Abstraction and Reasoning Corpus (CogARC), a diverse human-adapted subset of the Abstraction and Reasoning Corpus (ARC) which was originally developed to benchmark abstract reasoning in artificial intelligence. Across two experiments, CogARC was administered to a total of 260 human participants who freely generated solutions to 75 abstract visual reasoning problems. Success required inferring input-output rules from a small number of examples to transform the test input into one correct test output. Participants' behavior was recorded at high temporal resolution, including example viewing, edit sequences, and multi-attempt submissions. Participants were generally successful (mean accuracy ~90% for experiment 1 (n=40), ~80% for experiment 2 (n=220) across problems), but performance varied widely across problems and participants. Harder problems elicited longer deliberation times and greater divergence in solution strategies. Over the course of the task, participants initiated responses more quickly but showed a slight decline in accuracy, suggesting increased familiarity with the task structure rather than improved rule-learning ability. Importantly, even incorrect solutions were often highly convergent, even when the problem-solving trajectories differed in length and smoothness. Some trajectories progressed directly and efficiently toward a stable outcome, whereas others involved extended exploration or partial restarts before converging. Together, these findings highlight CogARC as a rich behavioral environment for studying human abstract reasoning, providing insight into how people generalize, misgeneralize, and adapt their strategies under uncertainty.
Figures & tables
Figure 1: Task interface for experiment 1. The study participants solved 75 visuospatial abstract reasoning problems on the web browser-based interface shown above. The screen was divided into three parts: input-output examples to the left, test input grid in the middle, and a test output editor that participants could manipulate to the right.
Problem features
Minimum
Maximum
Grid size (tiles)
3x3
21x21
Number of input-output examples
2
6
Number of colors used (excluding black)
1
9
Table 1: Range of task properties.
Types of rules
Number of problems
Core knowledge categories
Objectness
20
Geometry and pattern
31
Number and counting
15
Goal-directedness
9
Level of complexity
Table 2: Problems by rule type.
Demographics
Categories
n
Percentage (%)
Mean
Median
SD
Age
220
29.55
31
4.06
Gender
Male
115
52.3
Female
104
47.3
Other
1
0.5
Native language
English
215
97.7
Other
4
1.8
Table 3: Basic demographics.
Figure 2: Two problems from the CogARC task and their task properties. Problem A is solved by drawing a blue outline around the grey tiles in the input. The core knowledge prior required to learn this rule is ‘objectness’. Problem B has a conditional rule in which any instances of fewer than 3 connected tiles of the same color in the input get re-colored to green in the output. Therefore, we can say this problem involves the core knowledge priors of ‘numbers and counting’ and ‘objectness’, and is of higher complexity than Problem A. For visualization purposes, problems are shown here in a simplified schematic format rather than the full interactive task interface used in the experiment.
Figure 3: Task schematic. A) Experiment 2 task interface Participants could freely switch back and forth between edit view and example view. They were given feedback upon submission. B) Problem solving flow and difficulty scoring Participants got up to three attempts to reach the correct solution for a problem. Each participant was given a difficulty score of 1-4 for each problem based on how many submissions it took and what the outcome was.
Figure 4: Exploratory results from Experiment 1. A) Distribution of participant accuracy across all problems and attempts ( mean=89.5% , SD=10.2% ). Dashed line indicates the median. B) Distribution of deliberation times (latency from trial onset to first edit; mean=22.3s , SD=13.5s ). Values are plotted on a log10 scale for visualization. Dashed line indicates the median. C) Distribution of mean difficulty scores across problems ( mean=1.50 , SD=0.55 ). Dashed line indicates the median. D) Positive correlation between mean deliberation time and mean difficulty score per problem (Pearson r=0.52 , p<.001 ). Line reflects best linear fit with 95% confidence interval indicated by the shaded region. E) Stacked bar plot showing the distribution of difficulty scores across all problems, ordered by number of first attempt successes. F) Comparison of mean difficulty scores per problem across core knowledge categories. (Obj = Objectness, Geo = Geometry and Pattern, Num = Numbers and Counting, GoD = Goal-directedness) G) Comparison of mean difficulty scores across experimenter-assigned complexity levels. (** p<0.01 )
Figure 5: Participant performance across trials. A) Distribution of number of trials completed per participant. B) Histogram showing the distribution of participant accuracy (percentage of correct trials over all trials; mean=80.1% , SD=16.6% ). Dashed line shows the median. C) Distribution of deliberation times (latency from trial onset to first edit; mean=52.74s , median=25.0s , SD=1336.9s ) shown on a log scale for visualization. The top 1% of values were excluded to improve readability; dashed line indicates the median.
Figure 6: Difficulty across problems and correlation with task features and other measures of performance. A) Distribution of mean difficulty scores for each problem. Dashed line shows the median. B) Stacked bar plot showing the distribution of difficulty scores per problem, ordered by number of first attempt successes. C) Comparison of Pearson correlation coefficients between mean difficulty score per problem and different task features and measures. *** denotes p<0.001 .
Figure 7: Performance over time. A) Mean deliberation time per trial over the course of 75 trials. B) Mean difficulty score per trial. C) Deliberation time and difficulty score regression coefficient values for each participant over time. Black diamonds show the mean. The dashed line is at 0.
Figure 8: Percentage of first attempt solutions per problem. The stacked bars show the percentage of submitted solution types (success, common errors, and miscellaneous) for the first attempt on each problem, ordered by number of first attempt successes. Some problems have a higher proportion of common error solutions than others.
Figure 9: Schematic of solution trajectory plots. The solution trajectory plots show how close the grid state is to the test output at any step of the edit sequence. The y-axis values are the normalized distance to the submitted output grid, which are calculated by the number of tiles in the grid that exactly match the tiles of the submitted output in color and location. Edit sequences almost all begin with copying from input (“input”), and proceed through a series of transition states to reach the submitted output. This output can match the correct solution, or fall short.
Figure 10: Efficient convergence on both correct and incorrect solutions in a counting-and-coloring problem. A) Input–output examples and test input. The correct rule required recoloring all connected groups of fewer than three same-colored tiles to green. B) Most common first-attempt outputs. Each output is shown with a colored frame indicating solution type. In addition to the correct solution, two incorrect outputs were repeatedly produced, indicating that failures were concentrated on a small number of systematic alternatives. C) Edit-distance trajectories from input to submitted output. Each line represents one participant’s edit sequence, plotted as normalized distance from their submitted solution at each step, allowing trajectories corresponding to different solution types to be shown on the same y-axis. A fixed vertical jitter was applied so that overlapping trajectories could be visualized. Trajectories for correct and incorrect solutions are short and largely monotonic, showing rapid convergence with little evidence of extended exploration aside from a few participants per solution group who reset their grids mid-trajectory. D) Distribution of normalized extra steps relative to the minimum possible edit length. Successes and common errors overlap near the minimum, indicating comparable path lengths across solution types. For Top Error 2, a secondary mode in the tail reflects a subset of participants who initially deviated before re-converging on the same incorrect outcome.
Figure 11: Shared incorrect solutions despite heterogeneous edit trajectories in a pattern-completion problem. A) Input–output examples and test input. The correct transformation required completing a rotational pattern by adding the red segment. B) Most common first-attempt outputs. Each output is shown with a colored frame indicating solution type. Although the correct solution was produced by 54 participants, three distinct incorrect outputs were also frequent and collectively more common, reflecting consistent misgeneralizations of the transformation rule. C) Edit-distance trajectories from input to submitted output. Each line represents one participant’s edit sequence, plotted as normalized distance from their submitted solution at each step, allowing trajectories corresponding to different solution types to be shown on the same y-axis. A fixed vertical jitter was applied so that overlapping trajectories could be visualized. Compared to Figure 10, trajectories are more variable across all solution types, with many participants exhibiting irregular paths prior to submission. D) Distribution of normalized extra steps relative to the minimum possible edit length. While all solution types include near-minimal sequences, incorrect solutions show broader and more right-skewed distributions, including a secondary mode for Top Error 2, consistent with partial exploration followed by convergence on a shared incorrect solution.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Supplementary Figure S1 : Shared common errors between experiments 1 and 2. We categorized participant first attempt submissions by the common solutions found in experiment 2, and found that the common errors persisted across the different studies.
Supplementary Figure S2 : Solution-specific edit trajectories relative to the correct outcome in the counting-and-coloring problem. A) Input–output examples and test input for the counting-and-coloring problem shown in Figure 10. The correct transformation required recoloring all connected groups of fewer than three same-colored tiles to green. B) Edit-distance trajectories shown separately for each solution type: correct solution (left), Top Error 1 (center), and Top Error 2 (right). Each line represents one participant’s edit sequence. Unlike Figure 10, the y-axis here shows unnormalized edit distance to the correct solution, allowing direct comparison of how trajectories associated with different outcomes approach or diverge from the correct target.
Supplementary Figure S3 : Relatively uniform edit efficiency with occasional restarting in a path-connecting problem. A) Input–output examples and test input. The correct transformation required connecting the red, yellow, and green cells by drawing a gray path. B) Most common first-attempt outputs. Each output is shown with a colored frame indicating solution type. In addition to the correct solution, two incorrect outputs were also frequently produced, indicating that errors were concentrated on a small number of alternatives. C) Edit-distance trajectories from input to submitted output. Each line represents one participant’s edit sequence, plotted as normalized distance from their submitted solution at each step, allowing trajectories corresponding to different solution types to be shown on the same y-axis. A fixed vertical jitter was applied so that overlapping trajectories could be visualized. Across solution types, most trajectories are short and directed, with only a small subset of participants exhibiting more indirect paths or grid resets. D) Distribution of normalized extra steps relative to the minimum possible edit length. For both correct and incorrect solutions, most edit sequences cluster near the minimum, indicating broadly similar efficiency across solution types, with a minority of longer sequences reflecting occasional restarting behavior.
Supplementary Figure S4 : Efficient success with greater variability among common errors in a path-extension problem. A) Input–output examples and test input. The correct transformation required replacing the red cells with purple paths extending diagonally. B) Most common first-attempt outputs. Each output is shown with a colored frame indicating solution type. Each of the two top incorrect outputs was produced by more participants than the correct solution. C) Edit-distance trajectories from input to submitted output. Each line represents one participant’s edit sequence, plotted as normalized distance from their submitted solution at each step, allowing comparison across solution types on a shared y-axis. A fixed vertical jitter was applied so that overlapping trajectories could be visualized. Participants who reached the correct solution on their first attempt typically followed short, direct trajectories, whereas common error trajectories were typically longer and more variable. D) Distribution of normalized extra steps relative to the minimum possible edit length. Successful solutions are tightly clustered near the minimum, while incorrect solutions show a wider spread in sequence length, indicating greater variability in the paths leading to common errors.
The Abstraction and Reasoning Corpus (ARC) benchmarks cognitive generalization, the ability to infer and apply abstract rules from limited examples. This paper presents a multi-stage rule-chaining framework that performs compositional reasoning across symbolic, structural, and conceptual levels. The framework integrates three complementary solvers: (1) a deterministic rule discovery module that induces atomic transformations through geometric, color, and object-based analysis; (2) a pattern-composition engine that reconstructs outputs via block merging, repetition, and spatial heuristics; and (3) a structural abstraction layer that infers hierarchical and nested relationships across grids. These solvers operate sequentially within a progressive fallback hierarchy, where each stage reuses prior reasoning traces to enhance interpretability and generalization. Training passed for 995 tasks out of 1000, further evaluated on 105 tasks out of 120 and solved 230 test tasks out of 240 ARC-AGI-2 tasks. The system achieved strong coverage across deterministic, compositional, and abstract categories, demonstrating an overall accuracy exceeding 95 percent. The proposed architecture bridges symbolic reasoning and pattern synthesis, providing interpretable insight into cognitive generalization. The results suggest that rule chaining and hierarchical composition can advance machine reasoning toward transparent, human-aligned abstraction without relying on task-specific tuning.
AI reasoning models have exceeded human performance on the ARC-AGI-1 benchmark, but does that mean state-of-the-art models have the underlying competence (humanlike abstract reasoning) that the benchmark was designed to test? Here we investigate the abstraction abilities of AI models using the closely related but simpler ConceptARC benchmark. Our evaluations vary input modality (textual vs. visual), use of external Python tools, and reasoning effort. Beyond output accuracy, we evaluate the natural-language rules that models generate to explain their solutions, enabling us to assess whether models recognize the abstractions that ConceptARC was designed to elicit. We show that the best models' rules are frequently based on less abstract, more domain-specific concepts, capturing intended abstractions considerably less often than humans. In the visual modality, AI models' output accuracy drops sharply; however, our rule-level analysis reveals that a substantial share of their rules capture intended abstractions, even as the models struggle to apply these concepts to generate correct solutions. In short, we show that using performance (accuracy) alone can substantially overestimate AI competence in textual modalities while underestimating it in visual modalities, an illustration of the risk of mistaking performance for competence.
The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format fails to evaluate the diversity of abilities that genuine abstract skill acquisition should enable. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task's underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin employs programmatic methods to generate new task instances and transform given inputs for a given ARC task, enabling dynamic generative sampling beyond fixed input-output pairs. Across five state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 25-52 percentage-point performance gap between standard ARC evaluation and evaluation on PotARCin, and find that multi-dimensional evaluation reorders models that standard accuracy ranks alike. We further investigate effects of generative sampling, difficulty of corruption types, and questions of self-consistency, showing that models frequently contradict their own formalized rule even where they have stated it correctly. We also introduce P-ARC, a held-out hand-crafted test set, on which models achieve 1-8% accuracy across all five dimensions, underscoring the importance of more holistic evaluations of abstract reasoning capabilities.