cs.LGNov 7, 2026

AbductionBench: Benchmarking Abductive Reasoning in Large Language Models

Authors: Moein Salimi, Amir Qeysarbeigi, Babak Hosseini Mohtasham, Faraz Doagoye Tehrani, Mahdi Naieni, Hanieh Sartipi, Ali Nouri, Amin Aghakasiri, 

Organizations: Sharif University of Technology · University of Tehran · Tarbiat Modares University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Abductive reasoning is the inference of plausible explanations for observed evidence. Evaluating this capability requires more than a single task: a system may need to generate an explanation, discriminate among competing hypotheses, revise its hypotheses as evidence arrives, or decide what evidence to seek next. Yet these capabilities are typically evaluated through isolated benchmarks with different task formulations and scoring procedures, so claims that a model is ``good at abduction'' may reflect only a narrow aspect of this capability. We introduce AbductionBench, a unified framework whose taxonomy characterizes abductive tasks. We instantiate this framework on 27 datasets spanning ten domains and evaluate eight LLMs from four model families evaluated on single-choice selection tasks. Beyond harmonizing heterogeneous tasks, AbductionBench separates final-answer evaluation from analysis of the observable reasoning traces and interaction histories produced alongside those answers. We analyze written traces as observable model outputs rather than faithful representations of internal computation, characterizing evidence use, hypothesis comparison, uncertainty, contradiction handling, and answer commitment. Jev matches or exceeds small LLMs when selecting among explanations over compact evidence but falls substantially behind on longer cases, suggesting that candidate discrimination and extended evidence integration place different demands on a system. In interactive tasks, the proportion of actions judged relevant declines at later steps, and a relevant first question is associated with a correct final answer; in the passive task, models rarely revise their initial hypothesis as new evidence arrives. Together, these findings highlight the value of evaluating abductive reasoning through both final answers and observable evidence use, hypothesis revision, and information gathering.

Figures & tables

Figure 1: Accuracy by CoT trace length (steps) against each model’s IO baseline (dashed); bands are ±\pm SEM, points with n<5n<5 omitted.
Figure 2: Eleven normalized trace measures (0–1) for the CoT traces of Gemini-3.8-Flash, Gemma-4-31B, and GPT-5.6-Luna. (a) Mean per model. (b) Mean for correct ( y=1y=1 ) and incorrect ( y=0y=0 ) answers. Helpful Fraction and Gold Alive are judged against the gold answer.
Figure 3: Judged trace measures by relative position in the chain (0–100%), pooled over the eight LLMs. (a) Mean, each row min–max scaled. (b) Correct-minus-incorrect difference, computed within each dataset and averaged; each row scaled symmetrically around zero, so blue marks positions where correct chains score higher and red where incorrect chains score higher. Row intensities are not comparable across measures.
Figure 4: Relevance and revision over the course of an episode. The left panel (a) tracks the share of actions judged relevant to the gold answer at each step of the interactive episodes, one colored panel per model (95% Wilson bands) against the other seven in gray; a step is plotted only once at least 30 of the model’s episodes reach it, so models that answer early trace shorter curves, and corner labels give relevance at the first and last plotted step. MedQDx counts its questions only. The right panel (b) shows, for AthenaBench’s passive setting, the share of each model’s 50 episodes whose predicted actor never changed, changed without changing the outcome, or flipped between wrong and right from the first turn to the last.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

#Name and CitationDomainData DeliveryHyp. Mode
1MedQDx Werthaim et al. ( 2026 )Healthcare: Diagnostic QuestioningIG
2Med-Inquire He et al. ( 2026 )Healthcare: Clinical DiagnosisIG
3Cloud-OpsBench Wang et al. ( 2026 )Computing Systems: Cloud Root-Cause AnalysisIG
4VivaBench Chiu et al. ( 2026 )Healthcare: Clinical DiagnosisIG
5e-CARE Du et al. ( 2022 )Commonsense: Causal ReasoningSG / Sel
6ART Bhagavatula et al. ( 2020 )Narrative Reasoning: Abductive ExplanationSG / Sel
Appendix
Table 2: Summary of datasets used. Data Delivery Mode: I = Interactive, S = Static. Hypothesis Processing Mode: G = Generation, Sel = Selection; G / Sel denotes datasets with separate generation and selection tasks.
Benchmark / formulationGSQMRPUA
ART: α\alpha NLI ( Bhagavatula et al., 2020 )–+––––––
ART: α\alpha NLG ( Bhagavatula et al., 2020 )+–J–––––
e-CARE: cause selection ( Du et al., 2022 )–+––––––
e-CARE: explanation of supplied pair ( Du et al., 2022 )r–J–––––
UNcommonsense ( Zhao et al., 2024 )+–J–––––
True Detective ( Del and Fishel, 2023 )–+––––––
Appendix
Table 3: Coverage of representative published protocols. G: hypothesis generation; S: candidate selection/ranking/verification; Q: explanatory-adequacy assessment; M: multiple explanations; R: revision/repair; P: held-out use of a generated hypothesis; U: confidence calibration; A: active evidence acquisition. Symbols and qualifications are defined below.
WorkScope and interfaceEvaluation targetRelation to AbductionBench
GEAR ( He et al., 2025 )Four source benchmarks reformulated as executable hypothesis generation; fixed observations.Shared consistency, prediction-domain coverage, and behavioral diversity; separate hidden-case tests.Direct evaluation-framework predecessor with released code; common executable representation rather than preservation of heterogeneous native interfaces.
Wiring the ‘Why’ ( Salimi et al., 2026a )Nine datasets, twelve formulations: commonsense, narratives, diagnosis, missing facts; static empirical suite.Generation and selection; exact, ranking, overlap, and judge metrics; fixed direct prompts.Closest taxonomy and multi-dataset study. Selective experiments ground the survey under a controlled protocol; our emphasis is heterogeneous benchmark execution.
HypoBench ( Liu et al., 2026 )Seven real and five synthetic tasks, 194 datasets; natural-language hypotheses for classification.Predictive utility, IND/OOD testing, planted hypothesis recovery, qualitative ratings.Multi-dataset generation precedent with public implementation; not a general selection or interactive-diagnosis collection.
HypoSpace ( Chen et al., 2026 )Three structured domains with enumerable admissible sets, plus real-data case studies.Validity, uniqueness, recovery under fixed evidence.A genuine multi-domain diagnostic framework for underdetermination, rather than a heterogeneous benchmark runner.
CEDAR-GRPO ( Salimi et al., 2026b )Eleven held-out tasks, including adjacent tasks and non-abductive controls.Post-training transfer, task outcomes, and trace-level metrics.Substantial empirical and software overlap: released evaluation orchestration and task scripts; training transfer is the central objective.
Elenchos ( Steiglechner et al., 2026 )Procedurally generated kernel mutations; interactive probes within a specified ontology.Corrupted-kernel identification, mutation attribution, and query efficiency.Dedicated extensible evaluation framework, but not an aggregation of existing datasets.
Appendix
Table 4: Related evaluation efforts, separated by objective and protocol rather than ranked by dataset count. Resource sizes use different counting units and are not directly comparable.
Figure 7: Final-answer accuracy of interactive episodes whose first action was judged irrelevant (gray) or relevant (color) to the gold answer, per task and pooled ( All ), per model; 95% Wilson intervals. ×\times : relevant-to-irrelevant ratio over all tasks.
More in this topic

Abductive Reasoning

Abductive reasoning studies how to infer plausible explanations or hidden causes from incomplete observations.

See all 51 papers

Explore similar work

CardsList
Apr 9, 2026

Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs

Regardless of its foundational role in human discovery and sense-making, abductive reasoning--the inference of the most plausible explanation for an observation--has been relatively underexplored in Large Language Models (LLMs). Despite the rapid advancement of LLMs, the exploration of abductive reasoning and its diverse facets has thus far been disjointed rather than cohesive. This paper presents the first survey of abductive reasoning in LLMs, tracing its trajectory from philosophical foundations to contemporary AI implementations. To address the widespread conceptual confusion and disjointed task definitions prevalent in the field, we establish a unified two-stage definition that formally categorizes prior work. This definition disentangles abduction into Hypothesis Generation, where models bridge epistemic gaps to produce candidate explanations, and Hypothesis Selection, where the generated candidates are evaluated and the most plausible explanation is chosen. Building upon this foundation, we present a comprehensive taxonomy of the literature, categorizing prior work based on their abductive tasks, datasets, underlying methodologies, and evaluation strategies. In order to ground our framework empirically, we conduct a compact benchmark study of current LLMs on abductive tasks, together with targeted comparative analyses across model sizes, model families, evaluation styles, and the distinct generation-versus-selection task typologies. Moreover, by synthesizing recent empirical results, we examine how LLM performance on abductive reasoning relates to deductive and inductive tasks, providing insights into their broader reasoning capabilities. Our analysis reveals critical gaps in current approaches--from static benchmark design and narrow domain coverage to narrow training frameworks and limited mechanistic understanding of abductive processes...

Moein Salimi, Shaygan Adim, Danial Parnian +3

Department of Computer Engineering, Sharif University of Technology · Department of Mathematical Sciences, Sharif University of Technology

Abductive Reasoning in Language ModelsAbductive Reasoning
AskPDF
Aug 4, 2026

Don't Let Me Ask for It: LLMs Show Deficiencies in Active Multi-Turn Information Acquisition for Abductive Inference

Abductive reasoning requires forming hypotheses that explain observed evidence and revising them as new evidence becomes available. While large language models (LLMs) are often evaluated on whether they solve abductive reasoning tasks correctly, less is known about how they acquire evidence, update their hypotheses, and decide when to stop. We introduce Alien Abduction game, an interactive probe for studying these behaviours under different interaction modes. The modes vary in whether evidence is provided upfront or across turns, and whether queries are selected by the model or examples are provided by the oracle. Across models, providing evidence upfront leads to higher success rates than distributing it across turns. In multi-turn settings, some models commit before using the available evidence, while others exhaust the turn budget without converging. Models also achieve higher success rates when examples are provided by the oracle than when they select their own queries, although their final hypotheses are more consistent with the evidence they selected. These findings suggest that models may form hypotheses that fit self-selected evidence without sufficiently distinguishing them from alternatives, and may struggle to validate and refine their hypotheses or determine when to stop.

Shahrukh Mohiuddin, Chalamalasetti Kranti, Sherzod Hakimov +1

Computational Linguistics, Department of Linguistics University of Potsdam, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany

LLM EvaluationMulti-Turn Language Model ReasoningAbductive Reasoning in Language ModelsLanguage Model Probing
AskPDF
Jul 14, 2026

LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos

Large language models (LLMs) excel at pattern recognition and text generation, but their capacity for abductive inference - inferring latent hypotheses that explain observed behavior - remains poorly understood. Here, we introduce Elenchos (named after the Socratic method of cross-examination), a generative evaluation framework that measures abductive reasoning as a structural inverse problem. Given a reference formal system, such as the lambda-calculus, and a potentially mutated counterpart, agents must determine whether a mutation has occurred and infer the rule modifications responsible for the resulting behavioral differences. Evaluating frontier and mid-tier LLMs reveals a consistent detection-attribution dissociation: models often recognize that a system has been altered but struggle to identify the latent mutations causing the observed discrepancies. Performance degrades substantially under interacting mutations, where models frequently recover only a subset of the underlying mutations. Preliminary evidence also suggests diminishing returns from increased inference-time reasoning, with only modest improvements under larger reasoning budgets, though this finding requires further validation.

Julius Steiglechner, Lucas Mahler, Gabriele Lohmann

Dept. of Biomedical Magnetic Resonance, University Hospital Tübingen, Tübingen, Germany · Max-Planck-Institute for Biological Cybernetics, Magnetic Resonance Center, Tübingen, Germany

LLM EvaluationAbductive Reasoning in Language ModelsLLM Reasoning
AskPDF