cs.SEMar 20, 2026

Procedural Refinement by LLM-driven Algorithmic Debugging for ARC-AGI-2

Authors: Yu-Ning Qiu, Lin-Feng Zou, Jiong-Da Wang, Xue-Rong Yuan, Wang-Zhou Dai

Organizations: Nanjing University, Nanjing, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In high-complexity abstract reasoning, a system must infer a latent rule from a few examples or structured observations and apply it to unseen instances. LLMs can express such rules as programs, but ordinary conversation-based refinement is largely outcome-level: it observes that an answer or output is wrong without formally re-checking which abstraction, relation, or transformation justified that outcome. We propose Abduction-Based Procedural Refinement (ABPR), a neuro-symbolic refinement approach that couples an LLM with a Prolog meta-interpreter. ABPR treats each candidate program as an executable declarative hypothesis of the latent rule and reifies its SLD goal--subgoal resolution into compact proof-tree-style derivations, following Shapiro's algorithmic program debugging (APD). In this view, refinement is not merely code-level debugging, but semantic re-checking of the model's hypothesised rule. We evaluate ABPR primarily on ARC-AGI-2, a challenging few-shot abstract rule induction benchmark over grid transformations. ABPR with Gemini-3-Flash achieves 56.67% Pass@2, while GPT-5.5 xHigh with ABPR reaches 98.33% Pass@2 on the public evaluation set. Supplementary experiments on fill-in-the-blank I-RAVEN-X and A-I-RAVEN adaptations provide evidence that the same trace-guided framework extends beyond ARC-specific grid tasks to RAVEN-style relational and analogical abstraction. Repeated-run and sensitivity analyses show that parallel trace-guided search reduces stochastic variance as search breadth and total search depth increase.

Figures & tables

Figure 1: Overview of ABPR: neural generation, symbolic verification, and semantic-trace-guided procedural refinement.
Figure 2: Representative ARC-AGI-2 grid transformation and supplementary RAVEN-style matrix reasoning tasks.
Experiment settingInitial modelRepair modelIter. repairABPR tracePass@2
Full same-model comparison: first pass →\rightarrow execution-feedback repair →\rightarrow ABPR repair
Gemini-3-Flash: first passGemini-3-Flash–−-−-34.03%
Gemini-3-Flash: execution-feedback repairGemini-3-FlashGemini-3-Flash++−-49.17%
Gemini-3-Flash: ABPR repairGemini-3-FlashGemini-3-Flash++++56.67%
Gemini-3-Pro: first passGemini-3-Pro–−-−-31.11%
Gemini-3-Pro: execution-feedback repairGemini-3-ProGemini-3-Pro++−-43.33%
Table 1: ARC-AGI-2 Pass@2 results under matched within-block first-pass and refinement settings. The table separates the model that produces the initial program from the model that performs repair. Cross-model rows report the initialization pool used for the corresponding cross-model repair experiment.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4
SettingMeanStd.
Gemini-3-Flash ABPR56.250.59
Gemini-3-Pro ABPR52.291.05
GPT-5.2 low-reasoning ABPR29.930.98
Claude-Sonnet-4.5 ABPR4.080.37
Qwen3-Max ABPR2.770.78
GPT-5.5 xHigh ABPR98.050.48
Appendix
Table 2: Repeated-run ARC-AGI-2 Pass@2 variance summary. Scores are percentages; Std. is the sample standard deviation across repeated runs.
Figure 5: ARC-AGI-2 reproducibility heatmap. Each cell is averaged over Gemini-3-Pro and Gemini-3-Flash after 10 repeated runs per model at the corresponding ensemble size NN and total search depth TdepthT_{\mathrm{depth}} . Here Tdepth=1T_{\mathrm{depth}}=1 denotes first-pass generation only. Lower values indicate more stable repeated performance.
MethodI-RAVEN-X interactiveA-I-RAVEN-style
SolvedAcc.SolvedAcc.
Prolog direct170/50034.00%141/50028.20%
Prolog execution-feedback repair198/50039.60%170/50034.00%
Prolog ABPR repair443/50088.60%468/50093.60%
Python direct320/50064.00%370/50074.00%
Python execution-feedback repair408/50081.60%443/50088.60%
Appendix
Table 3: Full RAVEN-style symbolic matrix results.
Figure 6: An iteration example for task b0039139.
Figure 7: An iteration example for task 9aaea919.

Explore similar work

CardsList
May 17, 2026

A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation

Abstract reasoning ability reflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning. To address this, we introduce an automated pipeline named A2RBench, encompassing generation, expansion, evaluation, and analysis. Specifically, in the generation stage, LLMs create diverse tasks demanding genuine reasoning; in the expansion stage, LLMs reuse validated rules and expand new input spaces to generate task variations, achieving scaling. However, such a process may cause hallucinations. To eliminate it, we further establish a theoretical framework and prove that programmatic verification--testing whether the inverse operation perfectly reverses the forward operation (cycle consistency)--guarantees a unique solution. Through extensive evaluations on mainstream LLMs, we find: (1) Current LLMs exhibit fundamental deficiencies in abstract reasoning, with top models significantly underperforming humans on a representative subset (39.8% vs. 68.5%). (2) Current LLMs fall far short of 2D and 1D in the complexity of generated 3D tasks, revealing their lack of understanding of high-dimensional tasks. (3) Counterintuitively, inputs with higher information complexity can simplify the reasoning process.

Qingchuan Ma, Yuexiao Ma, Yongkang Xie +3

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Institute of Artificial Intelligence, Xiamen University

LLM EvaluationSynthetic Benchmark Generation
AskPDF
Aug 14, 2026

CEDAR-GRPO: Process-Aware Reinforcement Learning for General Abductive Reasoning in LLMs

Abductive reasoning, often characterized as inference to the best explanation, is central to explanation under uncertainty, from everyday sense-making and investigation to scientific discovery. Yet LLM research has mostly studied abduction through narrow, task-specific benchmarks, making it unclear whether observed gains transfer beyond the benchmark family used for training or evaluation. We ask whether RL post-training can improve abduction as a transferable reasoning capability. We introduce CEDAR-GRPO, a process-aware framework that combines final-answer correctness with abductive rewards for evidence coverage and evidence-to-explanation directionality. Four open-weight LLMs are post-trained on a controlled, domain-neutral mixture of abductive hypothesis-generation and hypothesis-selection tasks. We evaluate them on 11 unseen tasks spanning hypothesis selection, missing-fact generation, defeasible inference, long-context investigation, clinical reasoning, code debugging, and non-abductive controls. CEDAR- GRPO improves every model on every held-out task over both base models and correctness-only GRPO, with average gains of 7.4 and 2.7 points, respectively, and a maximum gain of 30.8 points. Ablations confirm that RL, abductive reward design, and task diversity each contribute to transfer. Process-level metrics further show stronger abductive behavior, including exploration of alternatives, elimination of rivals, backtracking, and uncertainty marking.

Moein Salimi, Danial Parnian, Shaygan Adim +6

Sharif University of Technology · University of Tehran

Abductive ReasoningRL for Language Model ReasoningProcess Supervision
AskPDF
Apr 2, 2026

Compositional Neuro-Symbolic Reasoning

We study structured abstraction-based reasoning for the Abstraction and Reasoning Corpus (ARC) and compare its generalization to test-time approaches. Purely neural architectures lack reliable combinatorial generalization, while strictly symbolic systems struggle with perceptual grounding. We therefore propose a neuro-symbolic architecture that extracts object-level structure from grids, uses neural priors to propose candidate transformations from a fixed domain-specific language (DSL) of atomic patterns, and filters hypotheses using cross-example consistency. Instantiated as a compositional reasoning framework based on unit patterns inspired by human visual abstraction, the system augments large language models (LLMs) with object representations and transformation proposals. On ARC-AGI-2, it improves base LLM performance from 16% to 24.4% on the public evaluation set, and to 30.8% when combined with ARC Lang Solver via a meta-classifier. These results demonstrate that separating perception, neural-guided transformation proposal, and symbolic consistency filtering improves generalization without task-specific finetuning or reinforcement learning, while reducing reliance on brute-force search and sampling-based test-time scaling. We open-source the ARC-AGI-2 Reasoner code (https://github.com/CoreThink-AI/arc-agi-2-reasoner).

Anugyan Das, Omkar Ghugarkar, Vishvesh Bhat +1

CoreThink AI · Stanford University

Neuro-Symbolic ReasoningVisual ReasoningLLM ReasoningProgram Synthesis
AskPDF