cs.LGOct 6, 2026

A Systematic Study of Small Language Models on Abstract Reasoning Tasks

Authors: Nur A Zarin Nishat, Jens Lehmann, Andrei Aioanei, Sahar Vahdati

Organizations: Leibniz University of Hanover TIB – Leibniz Information Centre for Science and Technology and University Library · Amazon

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-specific regularities. We study this distinction in small language models on the ARC-TGI benchmark, which organizes abstract grid transformations into controllable task families and supports resampling, spatial shifts, and cross-benchmark transfer. Across more than 1,000 runs, we profile decoder-only, encoder--decoder, and mixture-of-experts model families under supervised fine-tuning. We examine the efficiency and stability of skill acquisition, robustness beyond the training distribution, interactions with model family and task formulation, and layer-wise attention signatures that accompany behavioral differences. Substantial in-distribution accuracy is attainable, but acquisition is sensitive to optimization and unevenly distributed across task families. Performance deteriorates sharply outside the training distribution, including when the rule is retained but grid scale changes. Greater training-set depth and breadth yield uneven gains, while the effect of additional in-context examples depends on model family. Executable-rule induction also yields correct solutions not observed under direct grid generation. On selected tasks, attention diagnostics show distinct concentration and context-dependence profiles, but do not establish general causal mechanisms. Overall, abstract-reasoning scores are conditional on the model, adaptation regime, evaluation distribution, and response format.

Figures & tables

Figure 1: Controlled study workflow. The curriculum builder creates parallel complexity, task, and prompt views that converge in a matched scheduler. The scheduler instantiates model-family and adaptation choices, launches distributed training or inference, and routes behavioral and selected-task diagnostic evidence to the four research questions. The dashed path denotes the curriculum constraints applied directly to scheduling.
Figure 2: Representative ARC-TGI transformation family. Grid-level variables change object configuration, positions, and colors. Task-level variables determine their joining direction and alignment within each generated episode.
Figure 3: RQ1: hyperparameter sensitivity and leading configurations. Top: reported importance scores for the decoder-only, mixture-of-experts, and encoder–decoder model families. Bottom: the ten highest-accuracy configurations in each sweep. Axes are scaled independently by family, and importance scores are descriptive rather than an additive variance partition. LR, WU, and WD denote learning rate, warm-up ratio, and weight decay.
Figure 4: RQ1: adaptation and acquisition dynamics. (a) Exact-match accuracy for the selected full fine-tuning and LoRA configurations over model-family-specific normalized training progress. Colors identify model families; solid circles denote full fine-tuning and dashed squares denote LoRA. (b) Median generator-level accuracy with interquartile bands over normalized training progress. Endpoint, rank, and compute details appear in Appendix B ; disaggregated generator trajectories appear in Appendix C .
Figure 5: RQ2: family-level heterogeneity. Generators are sorted by exact-match accuracy for each model family. The dashed line is global mean accuracy and dotted lines mark mean ±0.1\pm 0.1 . Low dispersion can reflect consistent success or uniform failure.
RQComparisonConditionDMoEED
RQ1Training breadth60 generators39.339.17.5
193 generators51.645.013.0
AdaptationFull fine-tuning53.932.621.1
LoRA54.432.810.8
RQ2Cross-benchmark transferSeen-family val. max.54.432.821.1
Observed ARC-AGI peak3.52.752.00
Table 1: Behavioral results summary. Exact-match accuracy (%) under the two conditions stated for each RQ1–RQ3 comparison. Complete trajectories, intermediate scales, formulation variants, and compute accounting appear in the corresponding figures and appendices.
Figure 6: RQ3: generator-level effects of in-context experience. Pairwise generator accuracies for one to four demonstrations. Points above or below the diagonal indicate which model family performs better; gray denotes a tie.
Figure 7: RQ4: exploratory layer-wise attention diagnostics. Top: mean attention entropy on the selected small-grid task (upper row) and large-grid task (lower row) after small-grid or large-grid training. Bottom: the baseline-normalized change in mean per-token target-answer log probability after blocking attention from the system message, demonstrations, or test-grid description. Negative knockout values indicate supportive context. Colors identify D (blue), MoE (green), and ED (orange); knockout panels retain native scales.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: RQ1: hyperparameter response distributions. Exact-match accuracy across the tested epoch, learning-rate, warm-up, and weight-decay values for D (top), MoE (middle), and ED (bottom). Points are individual sweep configurations, violins summarize their marginal distributions, and red lines connect marginal means. Each panel marginalizes over the other hyperparameters and should therefore be read as a diagnostic response profile, not an isolated treatment effect.
FamilyUpdateEpochAcc.TimeGPUVRAMMin/epochAdj. GPU hAcc./h
   (%)(sec) (GB)   
DFull2053.90633034 H1009452.7570.340.766
 LoRA2554.36801914 H1009453.4689.100.611
MoEFull532.571115111 H200141371.7061.950.526
 LoRA532.89955801 H200141318.6053.100.618
EDFull3021.12253654 H1009414.0928.180.749
Appendix
Table 2: RQ1: unified adaptation and compute summary. Selected full fine-tuning and LoRA endpoints for each model family. Accuracy is exact grid match; time and peak VRAM describe the realized run. Adjusted GPU hours use the stated H200 =2×=2\times H100 accounting convention. Acc./h is reported accuracy divided by adjusted GPU hours and is descriptive rather than a hardware-independent efficiency measure.
Figure 9: RQ1: generator-level checkpoint dynamics. Held-out exact-match trajectories for D (blue), MoE (green), and ED (orange) across 20 checkpoints. The six largest improvements are marked with solid circles, and three representative declines or zero-change cases with dashed diamonds; remaining generator trajectories are gray. Counts below each panel summarize improved, decreased, and zero-performing families between the sampled endpoints.
Figure 10: RQ1: experience depth. Generator-level exact-match accuracy across 191 fixed transformation families as training examples per family increase from 10 to 100. Panels correspond to D (blue), MoE (green), and ED (orange). Columns are generators, rows are sample counts, and intensity encodes accuracy on a shared 0–1 scale. Persistent vertical structure shows that generator identity remains influential as within-family experience increases.
Figure 11: RQ1: experience breadth. Generator-level exact-match accuracy on 50 fixed evaluation families as the training curriculum expands from 60 to 193 generators. From left to right, panels show D, MoE, and ED. Columns are curriculum sizes and rows are shared evaluation generators, sorted independently by each model’s 60-generator baseline. Color encodes accuracy on a common 0–1 scale; annotations summarize retained, improved, and markedly decreased endpoint cases.
GeneratorsDMoEED
6039.339.17.5
8044.839.28.7
10043.841.59.7
14049.343.212.7
18051.045.112.2
19351.645.013.0
Appendix
Table 3: RQ1: aggregate experience-breadth results. Exact-match accuracy (%) on the same 50 evaluation families as the training curriculum expands from 60 to 193 generators. These averages summarize the same conditions shown at generator level in Figure 11 .
EpochDMoEED
00.50.250.25
12.02.751.0
33.01.251.75
50.52.751.25
7–1.50–
100.25–1.50
Appendix
Table 4: RQ2: unified transfer results. Exact-match accuracy (%). Left: original ARC-AGI evaluation at each available checkpoint; dashes denote unmeasured checkpoints. Right: matched and crossed grid-scale evaluation with generator identity shared across conditions. The left panel is a cross-benchmark comparison, whereas the right panel isolates a spatial shift within the filtered ARC-TGI families.
InterventionConditionDMoEED
FormulationTransduction: few-shot44.333.70.0
Transduction: reasoning42.732.615.6
Induction: few-shot52.359.016.1
Induction: reasoning42.459.212.6
ContextOne shot36.720.143.6
Two shots45.826.537.9
Appendix
Table 5: RQ3: formulation and in-context experience. Exact-match accuracy (%) under direct grid transduction and executable-rule induction with few-shot or reasoning-description prompts, followed by accuracy on fixed task and generator support as demonstrations increase from one to four.

Explore similar work

CardsList
May 29, 2026cs.AI

GraphARC: A Comprehensive Benchmark for Graph-Based Abstract Reasoning

Relational reasoning lies at the heart of intelligence, but existing benchmarks are typically confined to formats such as grids or text. We introduce GraphARC, a benchmark for abstract reasoning on graph-structured data. GraphARC generalizes the few-shot transformation learning paradigm of the Abstraction and Reasoning Corpus (ARC). Each task requires inferring a transformation rule from a few input-output pairs and applying it to a new test graph, covering local, global, and hierarchical graph transformations. Unlike grid-based ARC, GraphARC instances can be generated at scale across diverse graph families and sizes, enabling systematic evaluation of generalization abilities. We evaluate state-of-the-art language models on GraphARC and observe clear limitations. Models can answer questions about graph properties but often fail to solve the full graph transformation task, revealing a comprehension-execution gap. Performance further degrades on larger instances, exposing scaling barriers. More broadly, by combining aspects of node classification, link prediction, and graph generation within a single framework, GraphARC provides a promising testbed for future graph foundation models.

Saku Peltonen, August Bøgh Rønberg, Andreas Plesner +1

ETH Zürich Zürich, Switzerland

Multiplex Graph TransformersGraph RepresentationsWireless Foundation ModelsText Corpora
AskPDF
Jun 25, 2026cs.CL

DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models

The Abstraction and Reasoning Corpus (ARC) contains tasks that require summarizing patterns from limited grid samples and predicting output grids. Recently, many large language model based approaches have attempted to transform it into a text-based reasoning task. However, methods based on open-source models have generally yielded unsatisfactory results, while those relying on closed-source models are too costly. Current efforts mainly focus on data augmentation, constructing ARC-like data for more comprehensive supervised fine-tuning. In this work, we argue that solving ARC-like problems requires not only positive sample supervision but also the ability to improve model reasoning by distinguishing negative samples. To this end, we draw on the idea of preference alignment and propose DiARC, a method that constructs preference pairs to enable the model to distinguish between them. Specifically, we propose three ways to construct negative samples, including output-level visual transformations, DSL-level rule inversion, and task-specific rule editing. The resulting negative samples provide informative near-miss alternatives while keeping the observed demonstrations unchanged. Experimental results across multiple ARC-like benchmarks show that DiARC consistently improves performance over baseline models. The code is released at https://github.com/szu-tera/DiARC.

Yuxuan Yang, Feiyang Li, Yile Wang

College of Computer Science and Software Engineering, Shenzhen University

Reasoning SkillsNegative ResultsText CorporaPositive
AskPDF
May 17, 2026cs.AI

A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation

Abstract reasoning ability reflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning. To address this, we introduce an automated pipeline named A2RBench, encompassing generation, expansion, evaluation, and analysis. Specifically, in the generation stage, LLMs create diverse tasks demanding genuine reasoning; in the expansion stage, LLMs reuse validated rules and expand new input spaces to generate task variations, achieving scaling. However, such a process may cause hallucinations. To eliminate it, we further establish a theoretical framework and prove that programmatic verification--testing whether the inverse operation perfectly reverses the forward operation (cycle consistency)--guarantees a unique solution. Through extensive evaluations on mainstream LLMs, we find: (1) Current LLMs exhibit fundamental deficiencies in abstract reasoning, with top models significantly underperforming humans on a representative subset (39.8% vs. 68.5%). (2) Current LLMs fall far short of 2D and 1D in the complexity of generated 3D tasks, revealing their lack of understanding of high-dimensional tasks. (3) Counterintuitively, inputs with higher information complexity can simplify the reasoning process.

Qingchuan Ma, Yuexiao Ma, Yongkang Xie +3

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Institute of Artificial Intelligence, Xiamen University

Reasoning BenchmarkLarge Language Model BenchmarksReasoning SkillsAuto ResearchImproved Generalization
AskPDF