AbductionBench: Benchmarking Abductive Reasoning in Large Language Models
Organizations: Sharif University of Technology · University of Tehran · Tarbiat Modares University
Abstract
Abductive reasoning is the inference of plausible explanations for observed evidence. Evaluating this capability requires more than a single task: a system may need to generate an explanation, discriminate among competing hypotheses, revise its hypotheses as evidence arrives, or decide what evidence to seek next. Yet these capabilities are typically evaluated through isolated benchmarks with different task formulations and scoring procedures, so claims that a model is ``good at abduction'' may reflect only a narrow aspect of this capability. We introduce AbductionBench, a unified framework whose taxonomy characterizes abductive tasks. We instantiate this framework on 27 datasets spanning ten domains and evaluate eight LLMs from four model families evaluated on single-choice selection tasks. Beyond harmonizing heterogeneous tasks, AbductionBench separates final-answer evaluation from analysis of the observable reasoning traces and interaction histories produced alongside those answers. We analyze written traces as observable model outputs rather than faithful representations of internal computation, characterizing evidence use, hypothesis comparison, uncertainty, contradiction handling, and answer commitment. Jev matches or exceeds small LLMs when selecting among explanations over compact evidence but falls substantially behind on longer cases, suggesting that candidate discrimination and extended evidence integration place different demands on a system. In interactive tasks, the proportion of actions judged relevant declines at later steps, and a relevant first question is associated with a correct final answer; in the passive task, models rarely revise their initial hypothesis as new evidence arrives. Together, these findings highlight the value of evaluating abductive reasoning through both final answers and observable evidence use, hypothesis revision, and information gathering.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| # | Name and Citation | Domain | Data Delivery | Hyp. Mode |
|---|---|---|---|---|
| 1 | MedQDx Werthaim et al. ( 2026 ) | Healthcare: Diagnostic Questioning | I | G |
| 2 | Med-Inquire He et al. ( 2026 ) | Healthcare: Clinical Diagnosis | I | G |
| 3 | Cloud-OpsBench Wang et al. ( 2026 ) | Computing Systems: Cloud Root-Cause Analysis | I | G |
| 4 | VivaBench Chiu et al. ( 2026 ) | Healthcare: Clinical Diagnosis | I | G |
| 5 | e-CARE Du et al. ( 2022 ) | Commonsense: Causal Reasoning | S | G / Sel |
| 6 | ART Bhagavatula et al. ( 2020 ) | Narrative Reasoning: Abductive Explanation | S | G / Sel |
| Benchmark / formulation | G | S | Q | M | R | P | U | A |
|---|---|---|---|---|---|---|---|---|
| ART: NLI ( Bhagavatula et al., 2020 ) | – | + | – | – | – | – | – | – |
| ART: NLG ( Bhagavatula et al., 2020 ) | + | – | J | – | – | – | – | – |
| e-CARE: cause selection ( Du et al., 2022 ) | – | + | – | – | – | – | – | – |
| e-CARE: explanation of supplied pair ( Du et al., 2022 ) | r | – | J | – | – | – | – | – |
| UNcommonsense ( Zhao et al., 2024 ) | + | – | J | – | – | – | – | – |
| True Detective ( Del and Fishel, 2023 ) | – | + | – | – | – | – | – | – |
| Work | Scope and interface | Evaluation target | Relation to AbductionBench |
|---|---|---|---|
| GEAR ( He et al., 2025 ) | Four source benchmarks reformulated as executable hypothesis generation; fixed observations. | Shared consistency, prediction-domain coverage, and behavioral diversity; separate hidden-case tests. | Direct evaluation-framework predecessor with released code; common executable representation rather than preservation of heterogeneous native interfaces. |
| Wiring the ‘Why’ ( Salimi et al., 2026a ) | Nine datasets, twelve formulations: commonsense, narratives, diagnosis, missing facts; static empirical suite. | Generation and selection; exact, ranking, overlap, and judge metrics; fixed direct prompts. | Closest taxonomy and multi-dataset study. Selective experiments ground the survey under a controlled protocol; our emphasis is heterogeneous benchmark execution. |
| HypoBench ( Liu et al., 2026 ) | Seven real and five synthetic tasks, 194 datasets; natural-language hypotheses for classification. | Predictive utility, IND/OOD testing, planted hypothesis recovery, qualitative ratings. | Multi-dataset generation precedent with public implementation; not a general selection or interactive-diagnosis collection. |
| HypoSpace ( Chen et al., 2026 ) | Three structured domains with enumerable admissible sets, plus real-data case studies. | Validity, uniqueness, recovery under fixed evidence. | A genuine multi-domain diagnostic framework for underdetermination, rather than a heterogeneous benchmark runner. |
| CEDAR-GRPO ( Salimi et al., 2026b ) | Eleven held-out tasks, including adjacent tasks and non-abductive controls. | Post-training transfer, task outcomes, and trace-level metrics. | Substantial empirical and software overlap: released evaluation orchestration and task scripts; training transfer is the central objective. |
| Elenchos ( Steiglechner et al., 2026 ) | Procedurally generated kernel mutations; interactive probes within a specified ontology. | Corrupted-kernel identification, mutation attribution, and query efficiency. | Dedicated extensible evaluation framework, but not an aggregation of existing datasets. |
Abductive Reasoning
Abductive reasoning studies how to infer plausible explanations or hidden causes from incomplete observations.
See all 51 papers