cs.AIOct 21, 2026
SaveARCANA: A Benchmark for Abstraction and Analogical Reasoning
Organizations: University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China · University of Science and Technology Beijing, Beijing, China
Abstract
Analogical reasoning is widely considered a fundamental component of human intelligence. Few benchmarks are designed to explicitly test far analogical reasoning - reasoning between domains that are superficially dissimilar yet share deep relational or structural similarities. In this work, we introduce an analogical reasoning benchmark called Abstraction and Reasoning Corpus of ANAlogy (ARCANA). The tasks of ARCANA require performing far analogical mappings from real-world objects or physical phenomena to abstract grid-based transformations, different from the well-known ARC-AGI benchmark which primarily emphasized innate human priors. We validate ARCANA through a human study, demonstrating that humans solve these tasks reliably. In contrast, evaluations across a range of large language models show a distinct performance gap compared to humans, revealing systematic failure modes that remain challenging for current models. To address this gap, we propose ConceptSIR, a reasoning framework based on concept slippage, introspection, and reflection. Experiments demonstrate that ConceptSIR enables standard LLMs to solve several challenging ARCANA tasks that were unsolved by direct prompting in our evaluation. Overall, our results suggest that far analogical reasoning remains an open problem for contemporary LLMs. All the materials are available at sites.google.com/view/arcanacogtest. The complete codebase, including benchmark, and ConceptSIR, is publicly available at https://github.com/density-11/arcana-test. All ARCANA data and task instances can be accessed and explored at https://arcanacogtest.org/.
Figures & tables
Figure 1: Visualization of three tasks from ARC-AGI-1, ARC-AGI-2, and ARCANA. (a) Copy the 3×3 pattern to the purple block which is the center of the destination. (b) Fill in the columns with different colors, and the block numbers of the same color in the upper left corner determine the columns to be filled. (c) ARCANA focuses on far analogical reasoning involving threading 3D rings onto a column, where the rings settle at equilibrium positions determined by the geometry of their inner diameters.
Figure 2: One task from Copycat theory. In workspace, the task is given as: if changing to , then what will change to? The concept “opposite” activated from and in the alphabet will promote the concept “successor” to slip to the concept “predecessor”. This will finally result in the answer .
Figure 3: Visualization of three ARCANA task instantiations, illustrating distinct real-world targets and their corresponding abstract concepts. (a) Candles on a cake: candles with different cross-sectional areas and lengths are placed on a 3D cake; after burning by a certain amount, they reveal the cake behind them. (b) Melting: teal cells denote ice, blue cells denote cold water, and red cells denote hot water; ice melts only upon contact with hot water, and the melted ice together with the hot water becomes cold water with volume preserved. (c) Charge/data transfer: green cells (charge/data) propagate along directed wires from the source container connected to the red terminal to the sink container connected to the green terminal; at each time step, each wire transfers one cell if the destination container has remaining capacity.
Figure 4: Visualization of cat1 task in animal category. The task requires recognizing topology of the true cat, and changing the color of eyes, ears and tail into red.
Figure 5: Visualization of Kepler_second_law_or_area1 task in other category. We have provided the answer of the test in the figure. The same contour of each matrix represents the orbit of the planet which should be regarded as an “ellipse”, and the red grid represents the Sun (star). In either input or output matrix, the green “lines” represent the line segments joining the planet and the Sun, indicating the beginning and the end positions of one clockwise rotation. The purple grids represent the area swept out by the line segments, remaining consistent from input to output. All examples are designed to possess a unique solution.
| +Typo | -Typo | |
|---|---|---|
| Human (Average, pass@1) | 68.96% | 58.35% |
| Human ( , pass@1) | 96% | 93.5% |
Table 1: The general results of human study. Human (Average, pass@1) represents the proportion of correct answers in all answers, and Human ( ) represents the proportion of tasks solved by at least two participants. +Typo counts answers with typographical mistakes as correct, whereas -Typo treats them as errors.
Figure 6: Human performance and subjective difficulty on ARCANA. (a) Response distribution over five manual categories (CC, CW, WT, WC, WW), with a table summarizing answer vs. analogy correctness (typo excluded). (b) Per-task human accuracy histogram under two scoring conventions (typo counted as wrong vs. correct). (c) Per-task accuracy versus average subjective difficulty (thinking and drawing; mapped to a – scale), with fitted trend lines for both conventions.
| Model | ARCANA | |||||
| Total | CoT (language) | CoT + Ana (VL) | ||||
| +Typo | -Typo | +Typo | -Typo | +Typo | -Typo | |
| GPT 5.2 | 51.20% | 48.33% | 40.19% | 38.28% | 42.11% | 37.32% |
| Gemini 3 Pro | 43.06% | 29.19% | 34.45% | 23.92% | 28.71% | 18.66% |
| Grok 4.1 Fast | 24.88% | 18.66% | 19.62% | 14.35% | 15.79% | 11.00% |
| Claude Opus 4.5 | 8.61% | 7.18% | 7.66% | 6.22% | 6.70% | 5.74% |
Table 2: Experimental results on ARCANA. CoT (language) uses language-only chain-of-thought prompting; CoT + Ana (VL) adds an analogical prompt with visual inputs. +Typo / -Typo indicate whether minor typographical errors are counted as correct. Total is the union of tasks solved under the two setups (LLMs: 12 attempts for pass@3 in total across prompts, two settings, each with two prompt variants, each run three times; humans: 1 attempt for pass@1 ).
| Model | Strict / CI | Typo-tolerant / CI |
|---|---|---|
| GPT 5.2 | [ , ] | [ , ] |
| Gemini 3 Pro | [ , ] | [ , ] |
| Grok 4.1 Fast | [ , ] | [ , ] |
| Claude Opus 4.5 | [ , ] | [ , ] |
Table 3: Paired task-level bootstrap comparison between LLM baselines and human per-task accuracy on ARCANA. denotes LLM accuracy minus human accuracy; all confidence intervals are below zero.
Figure 7: An overview of ConceptSIR framework. Given an ARCANA task, the framework first proposes initial concepts from multi-dimensional prompts. These concepts evolve via Concept Slippage to generate a diverse set of candidates, which are then implemented as executable code under Concept Execution. The system iteratively refines solutions through Reflection, employing mechanisms such as code mutation and concept mutation, while Introspection ensures the final prediction on the test question aligns with the established rules and human preference.
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Visualization of two ARC-AGI-2 task instances that may admit explicit far-analogy interpretations. Both tasks can be understood through analogies to water: (a) water flow and (b) a water stream.
Figure 9: A screenshot of the formal task interface.
Figure 10: A screenshot of informed consent for participants.
Figure 11: A screenshot of instructions page.
Figure 12: A screenshot of tutorial.
Figure 13: A screenshot of edit mode.
Figure 14: A screenshot of attention check.
Figure 15: A typo of missed erase / incomplete edit after a state change in archimedes1 .
Figure 16: A typo of Accidental clicks or small omissions in complex structures in bottle3 .
Figure 17: A typo of Color slip that flips a single cell to an adjacent/nearby color in water_stream1 .
Figure 18: A typo of Off-by-one / off-by-two after translation or rotation of a pattern in fish1 .
Figure 19: A typo of Failure to apply a copy operation, causing minor irrelevant errors in cup2 .
| GPT 5.2 | Gemini 3.1 pro | Grok 4.1 Fast | Claude Opus 4.5 | human | |
|---|---|---|---|---|---|
| cost/task | 11.84 | 4.33 | 1.16 | 1.40 | 6.93 |
| tokens/task | 571K | 475K | 2.46M | 169K | - |
Table 4: The average cost and tokens LLMs and human spend per task.
| Model | ARCANA | |||||
| Total | CoT (language) | CoT + Ana (VL) | ||||
| +Typo | -Typo | +Typo | -Typo | +Typo | -Typo | |
| GPT 5.2 | 18.86% | 17.03% | 17.54% | 15.87% | 20.18% | 18.18% |
| Gemini 3 Pro | 13.12% | 8.57% | 13.96% | 9.17% | 12.28% | 7.97% |
| Grok 4.1 Fast | 6.46% | 4.39% | 6.70% | 4.47% | 6.22% | 4.31% |
| Claude Opus 4.5 | 2.23% | 1.71% | 2.47% | 1.99% | 1.99% | 1.44% |
Table 5: Response-level accuracy on ARCANA. For each LLM, Total aggregates all 12 responses per task across four prompt variants and three runs. CoT (language) uses language-only chain-of-thought prompting; CoT + Ana (VL) adds an analogical prompt with visual inputs. Missing or failed generations are counted as incorrect. +Typo counts minor typographical errors as correct, whereas -Typo uses strict exact match.
Figure 20: Accuracy distribution of GPT 5.2, Gemini 3 Pro and human across various task categories, including answers accuracy and analogy accuracy.
Figure 21: Category-wise analogy accuracy: GPT 5.2 vs. Gemini 3 Pro. Each point denotes one ARCANA category, with GPT 5.2 analogy accuracy on the x-axis and Gemini 3 Pro analogy accuracy on the y-axis. The dashed diagonal indicates parity; points above it correspond to categories where Gemini produces more correct analogical targets than GPT. Point color encodes the accuracy difference (Gemini minus GPT).
| Model | Correct / Total | Accuracy |
|---|---|---|
| GPT-5.2 | 159 / 200 | 79.50% |
| Gemini 3 Pro | 183 / 200 | 91.50% |
| Claude Opus 4.5 | 75 / 200 | 37.50% |
| Grok 4.1 Fast | 29 / 200 | 14.50% |
Table 6: Analogy-recognition accuracy of the four baseline LLMs on the 200 ARCANA tasks.
Figure 22: A typo by GPT 5.2 in fan1 task. The typo might be considered as one of the Accidental clicks or small omissions in complex structures .
Figure 23: A typo by Gemini 3 Pro in cup1 task. The typo might be considered as one of the Off-by-one / off-by-two after translation or rotation of a pattern .
Figure 24: Category-wise accuracy of humans versus LLMs. Each subplot corresponds to one model (GPT 5.2, Gemini 3 Pro, Grok 4.1 Fast, and Claude Opus 4.5). Each point denotes a task category, with human accuracy on the x-axis and model accuracy on the y-axis. Circles report +Typo (typos counted as correct) and triangles report -Typo (typos counted as wrong). The dashed diagonal indicates parity; points below it correspond to categories where humans outperform the model, and points above it correspond to categories where the model exceeds the human average.
| Model | Setting | Response Accuracy | Task Coverage |
|---|---|---|---|
| GPT-5.2 | Language | 19.61% / 19.61% | 41.18% / 41.18% |
| VL | 21.57% / 24.51% | 35.29% / 47.06% | |
| Overall | 20.59% / 22.06% | 41.18% / 47.06% | |
| Gemini 3 Pro | Language | 6.86% / 11.76% | 17.65% / 29.41% |
| VL | 9.80% / 9.80% | 23.53% / 23.53% | |
| Overall | 8.33% / 10.78% | 29.41% / 35.29% |
Table 7: Performance on the 17-task ConceptSIR subset. Each result is reported as Strict / +Typo . Language uses language-only chain-of-thought prompting; VL adds an analogical prompt with visual inputs. Each direct LLM generates six responses per task in each setting and twelve responses per task overall. ConceptSIR and human results are reported only under Overall.
Figure 25: The reasoning trajectory of refraction2 . Detailed concept contributes to the correct program generation and output.
Figure 26: The concept proposal and slippage trajectory of bead_chain1 . Concept proposal and slippage contribute better targets.
Figure 27: The introspection trajectory of candle1 . Introspection filter invalid output.
| Category | num | Task names |
| 1d_wave | 2 | wave2, wave_interference1 |
| 2d_wave | 5 | Doppler1, wave1, wave_diffraction1, |
| wave_diffraction2, wave_diffraction3 | ||
| 3d_cut | 3 | cut_sushi1, cut_sushi2, cut_sushi3 |
| 3d_move | 6 | brush1, mop1, mop2, pipes1, |
| table_and_stick_and_cuboid1, |
Table 8: The category names and tasks in ARCANA.
Figure 28: buoyancy2 in water_deeper_concept category.
Figure 29: melt2 in water_transform category.
Figure 30: melt3 in heat category.
Figure 31: melt4 in water_transform category.
Figure 32: moon_and_tide1 in others category.
Figure 33: magnetic3 in causal_updown category.
Figure 34: pulley3 in pulley category.
Figure 35: Doppler1 in 2d_wave category.