cs.AIOct 21, 2026

ARCANA: A Benchmark for Abstraction and Analogical Reasoning

Authors: Ruqi Chen, Zilei Wang, Wei Wang

Organizations: University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China · University of Science and Technology Beijing, Beijing, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Analogical reasoning is widely considered a fundamental component of human intelligence. Few benchmarks are designed to explicitly test far analogical reasoning - reasoning between domains that are superficially dissimilar yet share deep relational or structural similarities. In this work, we introduce an analogical reasoning benchmark called Abstraction and Reasoning Corpus of ANAlogy (ARCANA). The tasks of ARCANA require performing far analogical mappings from real-world objects or physical phenomena to abstract grid-based transformations, different from the well-known ARC-AGI benchmark which primarily emphasized innate human priors. We validate ARCANA through a human study, demonstrating that humans solve these tasks reliably. In contrast, evaluations across a range of large language models show a distinct performance gap compared to humans, revealing systematic failure modes that remain challenging for current models. To address this gap, we propose ConceptSIR, a reasoning framework based on concept slippage, introspection, and reflection. Experiments demonstrate that ConceptSIR enables standard LLMs to solve several challenging ARCANA tasks that were unsolved by direct prompting in our evaluation. Overall, our results suggest that far analogical reasoning remains an open problem for contemporary LLMs. All the materials are available at sites.google.com/view/arcanacogtest. The complete codebase, including benchmark, and ConceptSIR, is publicly available at https://github.com/density-11/arcana-test. All ARCANA data and task instances can be accessed and explored at https://arcanacogtest.org/.

Figures & tables

Figure 1: Visualization of three tasks from ARC-AGI-1, ARC-AGI-2, and ARCANA. (a) Copy the 3×3 pattern to the purple block which is the center of the destination. (b) Fill in the columns with different colors, and the block numbers of the same color in the upper left corner determine the columns to be filled. (c) ARCANA focuses on far analogical reasoning involving threading 3D rings onto a column, where the rings settle at equilibrium positions determined by the geometry of their inner diameters.
Figure 2: One task from Copycat theory. In workspace, the task is given as: if a b ca\ b\ c changing to a b da\ b\ d , then what will x y zx\ y\ z change to? The concept “opposite” activated from a b ca\ b\ c and x y zx\ y\ z in the alphabet will promote the concept “successor” to slip to the concept “predecessor”. This will finally result in the answer w y zw\ y\ z .
Figure 3: Visualization of three ARCANA task instantiations, illustrating distinct real-world targets and their corresponding abstract concepts. (a) Candles on a cake: candles with different cross-sectional areas and lengths are placed on a 3D cake; after burning by a certain amount, they reveal the cake behind them. (b) Melting: teal cells denote ice, blue cells denote cold water, and red cells denote hot water; ice melts only upon contact with hot water, and the melted ice together with the hot water becomes cold water with volume preserved. (c) Charge/data transfer: green cells (charge/data) propagate along directed wires from the source container connected to the red terminal to the sink container connected to the green terminal; at each time step, each wire transfers one cell if the destination container has remaining capacity.
Figure 4: Visualization of cat1 task in animal category. The task requires recognizing topology of the true cat, and changing the color of eyes, ears and tail into red.
Figure 5: Visualization of Kepler_second_law_or_area1 task in other category. We have provided the answer of the test in the figure. The same contour of each matrix represents the orbit of the planet which should be regarded as an “ellipse”, and the red grid represents the Sun (star). In either input or output matrix, the green “lines” represent the line segments joining the planet and the Sun, indicating the beginning and the end positions of one clockwise rotation. The purple grids represent the area swept out by the line segments, remaining consistent from input to output. All examples are designed to possess a unique solution.
 +Typo-Typo
Human (Average, pass@1)68.96%58.35%
Human ( ≥2\geq 2 , pass@1)96%93.5%
Table 1: The general results of human study. Human (Average, pass@1) represents the proportion of correct answers in all answers, and Human ( ≥2\geq 2 ) represents the proportion of tasks solved by at least two participants. +Typo counts answers with typographical mistakes as correct, whereas -Typo treats them as errors.
Figure 6: Human performance and subjective difficulty on ARCANA. (a) Response distribution over five manual categories (CC, CW, WT, WC, WW), with a 2×22\times 2 table summarizing answer vs. analogy correctness (typo excluded). (b) Per-task human accuracy histogram under two scoring conventions (typo counted as wrong vs. correct). (c) Per-task accuracy versus average subjective difficulty (thinking and drawing; mapped to a 11 – 55 scale), with fitted trend lines for both conventions.
ModelARCANA
TotalCoT (language)CoT + Ana (VL)
+Typo-Typo+Typo-Typo+Typo-Typo
GPT 5.251.20%48.33%40.19%38.28%42.11%37.32%
Gemini 3 Pro43.06%29.19%34.45%23.92%28.71%18.66%
Grok 4.1 Fast24.88%18.66%19.62%14.35%15.79%11.00%
Claude Opus 4.58.61%7.18%7.66%6.22%6.70%5.74%
Table 2: Experimental results on ARCANA. CoT (language) uses language-only chain-of-thought prompting; CoT + Ana (VL) adds an analogical prompt with visual inputs. +Typo / -Typo indicate whether minor typographical errors are counted as correct. Total is the union of tasks solved under the two setups (LLMs: 12 attempts for pass@3 in total across prompts, two settings, each with two prompt variants, each run three times; humans: 1 attempt for pass@1 ).
ModelStrict Δ\Delta / 95%95\% CITypo-tolerant Δ\Delta / 95%95\% CI
GPT 5.2−10.13%-10.13\% [ −17.06%-17.06\% , −3.31%-3.31\% ]−17.67%-17.67\% [ −24.20%-24.20\% , −11.12%-11.12\% ]
Gemini 3 Pro−29.63%-29.63\% [ −35.66%-35.66\% , −23.58%-23.58\% ]−26.17%-26.17\% [ −32.80%-32.80\% , −19.47%-19.47\% ]
Grok 4.1 Fast−40.13%-40.13\% [ −45.53%-45.53\% , −34.46%-34.46\% ]−44.17%-44.17\% [ −50.24%-50.24\% , −38.08%-38.08\% ]
Claude Opus 4.5−51.63%-51.63\% [ −56.16%-56.16\% , −46.85%-46.85\% ]−60.67%-60.67\% [ −65.59%-65.59\% , −55.73%-55.73\% ]
Table 3: Paired task-level bootstrap comparison between LLM baselines and human per-task accuracy on ARCANA. Δ\Delta denotes LLM accuracy minus human accuracy; all confidence intervals are below zero.
Figure 7: An overview of ConceptSIR framework. Given an ARCANA task, the framework first proposes initial concepts from multi-dimensional prompts. These concepts evolve via Concept Slippage to generate a diverse set of candidates, which are then implemented as executable code under Concept Execution. The system iteratively refines solutions through Reflection, employing mechanisms such as code mutation and concept mutation, while Introspection ensures the final prediction on the test question aligns with the established rules and human preference.
Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Visualization of two ARC-AGI-2 task instances that may admit explicit far-analogy interpretations. Both tasks can be understood through analogies to water: (a) water flow and (b) a water stream.
Figure 9: A screenshot of the formal task interface.
Figure 10: A screenshot of informed consent for participants.
Figure 11: A screenshot of instructions page.
Figure 12: A screenshot of tutorial.
Figure 13: A screenshot of edit mode.
Figure 14: A screenshot of attention check.
Figure 15: A typo of missed erase / incomplete edit after a state change in archimedes1 .
Figure 16: A typo of Accidental clicks or small omissions in complex structures in bottle3 .
Figure 17: A typo of Color slip that flips a single cell to an adjacent/nearby color in water_stream1 .
Figure 18: A typo of Off-by-one / off-by-two after translation or rotation of a pattern in fish1 .
Figure 19: A typo of Failure to apply a copy operation, causing minor irrelevant errors in cup2 .
 GPT 5.2Gemini 3.1 proGrok 4.1 FastClaude Opus 4.5human
cost/task11.844.331.161.406.93
tokens/task571K475K2.46M169K-
Appendix
Table 4: The average cost and tokens LLMs and human spend per task.
ModelARCANA
TotalCoT (language)CoT + Ana (VL)
+Typo-Typo+Typo-Typo+Typo-Typo
GPT 5.218.86%17.03%17.54%15.87%20.18%18.18%
Gemini 3 Pro13.12%8.57%13.96%9.17%12.28%7.97%
Grok 4.1 Fast6.46%4.39%6.70%4.47%6.22%4.31%
Claude Opus 4.52.23%1.71%2.47%1.99%1.99%1.44%
Appendix
Table 5: Response-level accuracy on ARCANA. For each LLM, Total aggregates all 12 responses per task across four prompt variants and three runs. CoT (language) uses language-only chain-of-thought prompting; CoT + Ana (VL) adds an analogical prompt with visual inputs. Missing or failed generations are counted as incorrect. +Typo counts minor typographical errors as correct, whereas -Typo uses strict exact match.
Figure 20: Accuracy distribution of GPT 5.2, Gemini 3 Pro and human across various task categories, including answers accuracy and analogy accuracy.
Figure 21: Category-wise analogy accuracy: GPT 5.2 vs. Gemini 3 Pro. Each point denotes one ARCANA category, with GPT 5.2 analogy accuracy on the x-axis and Gemini 3 Pro analogy accuracy on the y-axis. The dashed diagonal indicates parity; points above it correspond to categories where Gemini produces more correct analogical targets than GPT. Point color encodes the accuracy difference (Gemini minus GPT).
ModelCorrect / TotalAccuracy
GPT-5.2159 / 20079.50%
Gemini 3 Pro183 / 20091.50%
Claude Opus 4.575 / 20037.50%
Grok 4.1 Fast29 / 20014.50%
Appendix
Table 6: Analogy-recognition accuracy of the four baseline LLMs on the 200 ARCANA tasks.
Figure 22: A typo by GPT 5.2 in fan1 task. The typo might be considered as one of the Accidental clicks or small omissions in complex structures .
Figure 23: A typo by Gemini 3 Pro in cup1 task. The typo might be considered as one of the Off-by-one / off-by-two after translation or rotation of a pattern .
Figure 24: Category-wise accuracy of humans versus LLMs. Each subplot corresponds to one model (GPT 5.2, Gemini 3 Pro, Grok 4.1 Fast, and Claude Opus 4.5). Each point denotes a task category, with human accuracy on the x-axis and model accuracy on the y-axis. Circles report +Typo (typos counted as correct) and triangles report -Typo (typos counted as wrong). The dashed diagonal indicates parity; points below it correspond to categories where humans outperform the model, and points above it correspond to categories where the model exceeds the human average.
ModelSettingResponse AccuracyTask Coverage
GPT-5.2Language19.61% / 19.61%41.18% / 41.18%
VL21.57% / 24.51%35.29% / 47.06%
Overall20.59% / 22.06%41.18% / 47.06%
Gemini 3 ProLanguage6.86% / 11.76%17.65% / 29.41%
VL9.80% / 9.80%23.53% / 23.53%
Overall8.33% / 10.78%29.41% / 35.29%
Appendix
Table 7: Performance on the 17-task ConceptSIR subset. Each result is reported as Strict / +Typo . Language uses language-only chain-of-thought prompting; VL adds an analogical prompt with visual inputs. Each direct LLM generates six responses per task in each setting and twelve responses per task overall. ConceptSIR and human results are reported only under Overall.
Figure 25: The reasoning trajectory of refraction2 . Detailed concept contributes to the correct program generation and output.
Figure 26: The concept proposal and slippage trajectory of bead_chain1 . Concept proposal and slippage contribute better targets.
Figure 27: The introspection trajectory of candle1 . Introspection filter invalid output.
CategorynumTask names
1d_wave2wave2, wave_interference1
2d_wave5Doppler1, wave1, wave_diffraction1,
wave_diffraction2, wave_diffraction3
3d_cut3cut_sushi1, cut_sushi2, cut_sushi3
3d_move6brush1, mop1, mop2, pipes1,
table_and_stick_and_cuboid1,
Appendix
Table 8: The category names and tasks in ARCANA.
Figure 28: buoyancy2 in water_deeper_concept category.
Figure 29: melt2 in water_transform category.
Figure 30: melt3 in heat category.
Figure 31: melt4 in water_transform category.
Figure 32: moon_and_tide1 in others category.
Figure 33: magnetic3 in causal_updown category.
Figure 34: pulley3 in pulley category.
Figure 35: Doppler1 in 2d_wave category.
More in this topic

Analogical Reasoning for Language Models

See all 14 papers

Explore similar work

CardsList
Jul 25, 2026

ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation

Multilingual reasoning evaluation overwhelmingly relies on translating English benchmarks, a practice that introduces linguistic artifacts and fails to test culturally-grounded reasoning. We introduce ADAGE (Analogical Difficulty-by-design Assessment for Grounded Evaluation), a language-agnostic pipeline that combines native-speaker curation with LLM-assisted generation to construct challenging, translation-free benchmarks for abstract analogical reasoning. We validate ADAGE by constructing benchmarks for Arabic, Amharic, and Japanese. Evaluating 14 open-weight models, we find a consistent cultural reasoning gap: models that perform well on English proverb reasoning struggle substantially on all three native benchmarks, with accuracy dropping by 12--52 percentage points relative to English. We release the pipeline, all three benchmarks, and the full evaluation suite.

Ahmed Haj Ahmed, Alvin Grissom

Haverford College

Multilingual Language Model EvaluationReasoning EvaluationAnalogical Reasoning for Language ModelsCross-Lingual Reasoning in Language ModelsAnalogical ReasoningCross-Cultural Language Model Evaluation
AskPDF
Mar 30, 2026

CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another. Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence. To close this gap, we introduce CARV (Compositional Analogical Reasoning in Vision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark. We extend the analogy from a single pair to multiple pairs, which requires MLLMs to extract symbolic rules from each pair and compose new transformations. Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%. Diagnostic analysis shows two consistent failure modes: (1) decomposing visual changes into symbolic rules, and (2) maintaining robustness under diverse or complex settings, highlighting the limitations of current MLLMs on this task.

Yongkang Du, Xiaohan Zou, Minhao Cheng +1

Pennsylvania State University

Analogical ReasoningMultimodal ReasoningMultimodal Large Language ModelsMultimodal Model Evaluation
AskPDF
May 17, 2026

A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation

Abstract reasoning ability reflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning. To address this, we introduce an automated pipeline named A2RBench, encompassing generation, expansion, evaluation, and analysis. Specifically, in the generation stage, LLMs create diverse tasks demanding genuine reasoning; in the expansion stage, LLMs reuse validated rules and expand new input spaces to generate task variations, achieving scaling. However, such a process may cause hallucinations. To eliminate it, we further establish a theoretical framework and prove that programmatic verification--testing whether the inverse operation perfectly reverses the forward operation (cycle consistency)--guarantees a unique solution. Through extensive evaluations on mainstream LLMs, we find: (1) Current LLMs exhibit fundamental deficiencies in abstract reasoning, with top models significantly underperforming humans on a representative subset (39.8% vs. 68.5%). (2) Current LLMs fall far short of 2D and 1D in the complexity of generated 3D tasks, revealing their lack of understanding of high-dimensional tasks. (3) Counterintuitively, inputs with higher information complexity can simplify the reasoning process.

Qingchuan Ma, Yuexiao Ma, Yongkang Xie +3

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Institute of Artificial Intelligence, Xiamen University

LLM EvaluationSynthetic Benchmark Generation
AskPDF