ABD: Default Exception Abduction in Finite First Order Worlds

Serafim Batzoglou

cs.AIFeb 21, 2026

ABD: Default Exception Abduction in Finite First Order Worlds

Authors: Serafim Batzoglou

Abstract

We introduce ABD, a benchmark for default-exception abduction over finite first-order worlds. Given a background theory with an abnormality predicate and a set of relational structures, a model must output a first-order formula that defines exceptions, restoring satisfiability while keeping exceptions sparse. We formalize three observation regimes (closed-world, existential completion, universal completion) with exact SMT verification. Evaluating ten frontier LLMs on 600 instances, the best models achieve high validity but parsimony gaps remain, and holdout evaluation reveals distinct generalization failure modes across regimes.

Figures & tables

(a) Dataset Statistics
Scenario	N	Prompt	Holdout	$\|D\|$	Theories
FULL	195	9.0	5.0	9–11	T1, T2, T3, T4, T5
PARTIAL	243	8.3	5.0	9–11	T1, T2, T3, T4, T5
SKEPTICAL	162	6.0	5.0	10–12	T1, T2, T3, T4, T5, T6, T7
Total	600

Table 1: Abduction Benchmark Summary. (a) Dataset statistics by scenario. (b) Default theories used in the benchmark; each theory defines when

\mathsf{Ab}(x)

blocks a default conclusion.

	FULL					PARTIAL					SKEPTICAL					Overall
Model	PV%	PSV%	AST	Gap	GRef	PV%	PSV%	AST	Gap	GRef	PV%	PSV%	AST	Gap	GRef	PV%	PSV%	AST	Gap	GRef
High Validity ( $>90\%$ )
Opus-4.6	100	100	12.4	1.26	0.74	98	98	15.9	1.02	0.55	99	99	15.8	0.86	0.17	99	99	14.7	1.05	0.51
GPT-5.2	85	84	36.6	0.93	0.41	92	92	16.3	1.30	0.84	100	100	15.7	1.49	0.80	92	92	22.0	1.25	0.70
Gemini-3.1	99	99	12.5	1.27	0.76	97	97	11.4	1.40	0.93	98	98	13.5	1.18	0.50	98	98	12.3	1.30	0.76
Grok4.1f	96	94	12.8	1.32	0.80	95	95	9.0	1.69	1.23	96	96	11.8	1.46	0.82	95	95	11.0	1.51	0.98

Table 2: Abduction Prompt-Set Performance. Models are grouped by overall repaired prompt validity into high (

>90\%

), intermediate, and low (

<50\%

) bands; within each band, rows are ordered by overall Gap. Percentages are benchmark instances, not worlds. PV% = prompt-valid after conservative suffix repair; PSV% = strict prompt-validity without repair; AST = mean formula size among prompt-valid formulas; Gap = normalized extra exceptions above the solver lower bound; GRef = normalized extra exceptions vs. the planted generator reference. Gap and GRef average over prompt-valid formulas only. Lower is better for AST, Gap, and GRef. Best values bold.

	Opus-4.6		GPT-5.2		Gemini-3.1		Grok4.1f		DSR		GPT-5.4		Grok4		Gemini-3		Kimi-K2t		Hermes4		GPT-4o
Theory	PV%	GRef	PV%	GRef	PV%	GRef	PV%	GRef	PV%	GRef	PV%	GRef	PV%	GRef	PV%	GRef	PV%	GRef	PV%	GRef	PV%	GRef
T1	100	0.81	97	1.10	97	1.03	97	1.46	95	1.46	95	0.07	92	0.95	97	0.99	76	1.97	3	4.00	16	5.29
T2	99	0.74	94	0.84	99	0.93	98	1.05	96	1.10	82	0.05	91	0.76	81	0.71	75	1.33	2	1.01	16	2.00
T3	99	0.71	88	0.67	98	0.71	97	1.02	97	0.97	82	-0.14	90	0.74	70	0.63	74	1.36	1	2.00	28	1.25
T4	99	0.46	88	0.78	97	0.89	91	1.12	95	1.13	86	-0.18	86	0.71	60	0.86	59	1.44	0	—	14	4.66
T5	98	0.16	96	0.61	98	0.56	92	0.80	99	0.86	85	-0.21	90	0.62	92	0.53	82	0.99	8	2.82	5	5.17

Table 3: Abduction Prompt-Set Performance by Theory. PV% = prompt-validity after conservative suffix repair; GRef = normalized extra exceptions vs. the planted generator reference, averaged over prompt-valid formulas only. T1–T5 aggregate across all scenarios; T6–T7 are ABD-Skeptical only. Lower is better for GRef. Best values bold.

Model	PV%	HV%	PGap	HGap	$\Delta$ Gap	AST
Opus-4.6	98.8%	62.2%	1.05	2.24	+0.97	14.7
GPT-5.2	92.2%	61.5%	1.25	2.45	+0.85	22.0
Gemini-3.1	98.0%	69.7%	1.30	2.37	+0.98	12.3
Grok4.1f	95.2%	82.2%	1.51	2.54	+0.94	11.0
DSR	95.9%	80.6%	1.55	2.64	+0.96	10.8
GPT-5.4	85.2%	24.8%	0.42	1.37	+0.56	65.9

Table 4: Generalization Drop Profile. Aggregated across all scenarios. PV%/HV% are instance-level prompt/holdout validity; PGap/HGap are normalized per-world gaps on their respective valid subsets;

\Delta

Gap is computed on survivors valid on both prompt and holdout; AST = mean formula size among prompt-valid formulas. Best values bold.

Model	FULL					PARTIAL					SKEPTICAL
	PV%	HV%	PGap	HGap	$\Delta$ Gap	PV%	HV%	PGap	HGap	$\Delta$ Gap	PV%	HV%	PGap	HGap	$\Delta$ Gap
Opus-4.6	100%	87%	1.26	2.66	+1.34	98%	51%	1.02	2.09	+0.92	99%	49%	0.86	1.59	+0.25
GPT-5.2	85%	54%	0.93	2.74	+1.49	92%	68%	1.30	2.52	+1.02	100%	61%	1.49	2.06	-0.02
Gemini-3.1	99%	87%	1.27	2.60	+1.35	97%	68%	1.40	2.44	+1.00	98%	52%	1.18	1.80	+0.21
Grok4.1f	96%	88%	1.32	2.59	+1.27	95%	85%	1.69	2.85	+1.14	96%	69%	1.46	1.83	-0.02
DSR	96%	85%	1.42	2.86	+1.42	96%	85%	1.54	2.72	+1.11	96%	70%	1.69	2.23	+0.09

Table 5: Holdout Generalization by Scenario. PV%/HV% are prompt/holdout validity; PGap/HGap are normalized per-world gaps on their respective valid subsets;

\Delta

Gap is computed on survivors valid on both prompt and holdout. Best HV% and

\Delta

Gap per scenario bold.

	FULL						PARTIAL						SKEPTICAL
	[0,15)		[15,30)		[30, $\infty$ )		[0,15)		[15,30)		[30, $\infty$ )		[0,15)		[15,30)		[30, $\infty$ )
Model	H $\|$ P	$\Delta$	H $\|$ P	$\Delta$	H $\|$ P	$\Delta$	H $\|$ P	$\Delta$	H $\|$ P	$\Delta$	H $\|$ P	$\Delta$	H $\|$ P	$\Delta$	H $\|$ P	$\Delta$	H $\|$ P	$\Delta$
Opus-4.6	98	+1.5	62	+0.6	50	0.0	80	+1.1	30	+0.5	0	—	73	+0.3	26	+0.1	10	+2.2
GPT-5.2	97	+1.7	65	+0.9	17	+0.8	89	+1.1	42	+0.7	12	+0.9	88	-0.1	24	+0.3	12	+1.7
Gemini-3.1	93	+1.6	71	+0.8	100	+0.1	78	+1.0	38	+0.7	—	—	73	+0.2	18	0.0	33	+0.5
Grok4.1f	99	+1.4	64	+0.7	89	+0.1	93	+1.2	33	-0.1	0	—	87	0.0	31	+0.2	—	—

Table 6: Holdout Generalization by Formula Complexity. H

|

P columns report holdout validity conditional on prompt validity (percent);

\Delta

columns report survivor-conditioned

\Delta

Gap. Hermes4 and GPT-4o are omitted due to too few valid responses.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Antecedent $\phi(x)$	Consequent $\psi(x)$
T1	$\exists y\bigl(R(x,y)\land P(y)\bigr)$	$Q(x)$
T2	$\exists y\bigl(R(x,y)\land P(y)\bigr)$	$\exists z\bigl(S(x,z)\land Q(z)\bigr)$
T3	$\exists y\bigl(S(x,y)\land P(y)\bigr)$	$\exists z\bigl(R(x,z)\land Q(z)\bigr)$
T4	$\exists y\bigl(R(x,y)\land P(y)\bigr)$	$\exists z\bigl(S(x,z)\land\forall w(R(z,w)\to P(w))\bigr)$
T5	$\exists y\bigl(R(x,y)\land P(y)\bigr)$	$\forall z\bigl(S(x,z)\to Q(z)\bigr)$
T6	$P(x)$	$\exists y\bigl(R(x,y)\bigr)$

Appendix

Table C.1: Theory library. Each theory is a single default rule

\phi(x)\land\lnot\mathsf{Ab}(x)\to\psi(x)

ID	Allowed in $\alpha$	Forbidden in $\alpha$	Repaired
T1	$P,R,S$	$\mathsf{Ab},Q$	$Q$
T2	$P,R$	$\mathsf{Ab},S,Q$	$S,Q$
T3	$P,S$	$\mathsf{Ab},R,Q$	$R,Q$
T4	$P,Q,R$	$\mathsf{Ab},S$	$S$
T5	$P,R,S$	$\mathsf{Ab},Q$	$Q$
T6	$P,Q,S$	$\mathsf{Ab},R$	$R$

Appendix

Table C.2: Predicate scoping per theory. Repaired = consequent predicates excluded from

\alpha

to prevent shortcuts.

Predicate	Arity	Full / Partial	Skeptical	True atoms
$P$	1	$[0.20,\;0.60]$	$[0.40,\;0.60]$	$\lfloor n\cdot\rho_{P}\rfloor$
$Q$	1	$[0.20,\;0.60]$	$[0.20,\;0.50]$	$\lfloor n\cdot\rho_{Q}\rfloor$
$R$	2	$[0.12,\;0.25]$	$[0.15,\;0.30]$	$\lfloor n^{2}\cdot\rho_{R}\rfloor$
$S$	2	$[0.08,\;0.18]$	$[0.10,\;0.25]$	$\lfloor n^{2}\cdot\rho_{S}\rfloor$

Appendix

Table D.1: Predicate density ranges by scenario. For each world the sampler draws

\rho

uniformly from the listed interval;

n=|D|

. ABD-Full and ABD-Partial share the same ranges; ABD-Skeptical uses denser binary predicates and a narrower

P

range.

	ABD-Partial		ABD-Skeptical
Theory	$R$	$S$	$R$	$S$
T1 (TH2)	0.20	0.10	0.05	0.08
T2 (TH7)	0.20	0.10	0.05	0.05
T3 (TH10)	0.20	0.10	0.05	0.05
T4 (TH11)	0.20	0.10	0.05	0.05
T5 (TH12)	0.20	0.10	0.05	0.05

Appendix

Table D.2: Unknown-atom rates (fraction of binary atoms masked). ABD-Partial uses the default column; ABD-Skeptical uses theory-specific rates to keep universal validity tractable.

	ABD-Full		ABD-Partial		ABD-Skeptical
Metric	Prompt	Hold	Prompt	Hold	Prompt	Hold
Worlds/inst	9.0	5.0	8.3	5.0	6.0	5.0
PW Ref Cost	1.78	1.88	1.68	1.75	2.05	2.10
PW OptCost	1.27	1.33	1.20	1.32	1.36	1.54
PW Ref Gap	0.51	0.55	0.48	0.43	0.69	0.56

Appendix

Table F.1: Per-world summary statistics for prompt and holdout sets. Values are means across all instances in each scenario. Reference Gap = Reference Cost

-

OptCost, per world.

	Opus-4.6	GPT-5.2	Gemini-3.1	Grok4.1f	DSR	GPT-5.4	Grok4	Gemini-3	Kimi-K2t	Hermes4	GPT-4o
T1	1.3/3.2 (4)	1.3/3.7 (3)	1.5/3.2 (4)	1.4/3.2 (5)	1.5/3.3 (5)	0.1/0.4 (1)	1.1/3.2 (3)	1.1/1.9 (3)	1.5/3.1 (4)	—	—
T2	1.4/3.2 (56)	1.1/3.2 (34)	1.4/3.2 (54)	1.4/2.9 (44)	1.6/3.6 (44)	0.3/1.3 (16)	0.8/1.5 (31)	1.2/2.8 (36)	1.3/3.5 (33)	—	2.0/5.3 (6)
T3	1.4/2.8 (37)	0.9/2.6 (17)	1.2/2.5 (39)	1.5/2.8 (39)	1.5/2.8 (34)	0.2/1.2 (7)	1.1/2.6 (27)	1.4/2.5 (19)	1.5/2.9 (26)	—	1.4/3.6 (4)
T4	1.3/2.3 (46)	0.9/2.4 (22)	1.4/2.1 (44)	1.3/2.3 (45)	1.5/2.6 (41)	0.0/0.4 (6)	0.9/1.4 (25)	1.4/2.2 (21)	1.7/2.6 (24)	—	5.3/6.4 (0)
T5	0.8/1.8 (27)	0.6/2.2 (17)	0.9/2.0 (27)	0.9/2.0 (23)	0.9/1.9 (27)	0.1/1.1 (4)	0.7/1.9 (21)	0.9/2.0 (24)	1.3/2.3 (24)	2.8/5.5 (5)	5.8/7.1 (1)

Appendix

Table G.1: Per-theory results: ABD-Full . Each cell shows PGap / HGap (survivors).

	Opus-4.6	GPT-5.2	Gemini-3.1	Grok4.1f	DSR	GPT-5.4	Grok4	Gemini-3	Kimi-K2t	Hermes4	GPT-4o
T1	1.0/2.5 (5)	0.9/3.3 (2)	1.0/3.0 (3)	1.4/3.0 (5)	1.4/3.0 (5)	0.5/2.2 (2)	1.3/3.1 (4)	2.3/2.4 (3)	1.3/2.9 (4)	—	—
T2	1.2/2.6 (39)	1.4/2.8 (49)	1.5/2.8 (56)	1.6/3.0 (63)	1.6/2.9 (59)	0.7/2.6 (21)	1.6/2.9 (52)	1.3/2.3 (26)	1.7/3.0 (54)	1.8/3.0 (2)	2.9/3.2 (9)
T3	1.2/2.1 (36)	1.3/2.5 (32)	1.3/2.3 (38)	1.6/2.7 (46)	1.5/2.6 (40)	0.8/2.6 (11)	1.5/2.7 (39)	1.2/2.1 (20)	1.7/2.6 (34)	—	1.7/2.6 (24)
T4	0.9/1.7 (23)	1.4/2.6 (44)	1.5/2.3 (37)	2.0/3.1 (52)	1.7/2.9 (57)	0.6/1.8 (12)	1.7/2.9 (39)	1.3/2.2 (18)	1.9/3.2 (41)	—	5.0/5.8 (4)
T5	0.6/1.4 (21)	0.9/1.8 (25)	1.0/1.9 (31)	1.4/2.4 (26)	1.2/2.2 (30)	0.4/1.3 (12)	1.4/2.6 (26)	0.9/2.1 (19)	1.3/2.3 (25)	—	5.8/6.6 (0)

Appendix

Table G.2: Per-theory results: ABD-Partial . Format as Table G.1 .

	Opus-4.6	GPT-5.2	Gemini-3.1	Grok4.1f	DSR	GPT-5.4	Grok4	Gemini-3	Kimi-K2t	Hermes4	GPT-4o
T1	1.0/1.8 (10)	1.4/2.3 (14)	1.2/2.3 (10)	1.8/2.0 (13)	1.8/2.5 (14)	0.2/1.6 (4)	1.2/2.4 (4)	1.0/2.0 (10)	2.5/2.3 (18)	4.0/4.0 (1)	5.5/9.2 (1)
T2	1.1/2.6 (14)	1.8/3.2 (14)	1.5/2.9 (16)	1.8/2.4 (16)	1.8/3.4 (16)	0.6/0.7 (8)	1.4/2.8 (11)	1.2/2.1 (14)	3.3/4.4 (18)	1.9/5.8 (1)	2.6/4.9 (7)
T3	1.4/2.4 (13)	1.7/3.1 (12)	1.4/2.2 (10)	1.8/2.7 (14)	1.7/2.7 (14)	0.2/0.5 (6)	1.3/2.4 (10)	1.2/1.8 (11)	3.2/4.3 (16)	2.0/4.4 (1)	2.8/4.3 (5)
T4	0.9/1.7 (13)	2.1/2.7 (17)	1.5/2.0 (15)	1.5/1.9 (8)	2.3/2.9 (24)	0.6/1.0 (12)	1.2/2.6 (11)	1.7/2.6 (14)	2.8/3.3 (20)	—	—
T5	1.1/1.0 (13)	2.4/1.3 (22)	1.7/1.2 (15)	2.0/1.6 (15)	2.5/1.5 (21)	0.8/0.9 (11)	1.6/1.3 (12)	1.7/1.3 (18)	2.5/1.5 (19)	4.9/4.2 (2)	6.9/— (0)
T6	0.2/0.1 (3)	0.2/0.7 (4)	0.2/0.1 (2)	0.4/1.3 (7)	0.3/0.8 (5)	0.0/0.3 (3)	0.2/0.1 (2)	0.2/0.2 (4)	0.2/0.5 (2)	—	2.9/3.2 (12)

Appendix

Table G.3: Per-theory results: ABD-Skeptical . Format as Table G.1 . Note negative

\Delta

Gap for T5: holdout gap is often lower than prompt gap because worst-case costs can be smaller in holdout worlds for this theory.

Model	N	Mean	Med	P90	Max	$>$ 3	$>$ 5
ABD-Full
Opus-4.6	195	1.26	1.25	2.00	2.7	0.0%	0.0%
GPT-5.2	152	0.93	0.90	1.99	2.4	0.0%	0.0%
Gemini-3.1	194	1.27	1.20	1.97	7.9	1.0%	1.0%
Grok4.1f	171	1.32	1.25	2.12	3.0	0.0%	0.0%
DSR	172	1.42	1.39	2.16	3.2	0.6%	0.0%

Appendix

Table G.4: Normalized prompt-gap distribution by scenario. N = predictions valid on all prompt worlds.

>

3 and

>

5 are tail percentages.

Theory	Opus-4.6	GPT-5.2	Gemini-3.1	Grok4.1f	DSR	GPT-5.4	Grok4	Gemini-3	Kimi-K2t	Hermes4	GPT-4o
T1	+0.5 (19)	+0.4 (19)	+0.6 (17)	+0.5 (23)	+0.4 (24)	+0.6 (7)	+1.5 (11)	+0.5 (16)	+0.2 (26)	—	—
T2	+1.5 (109)	+1.3 (97)	+1.5 (126)	+1.3 (123)	+1.5 (119)	+0.8 (45)	+1.1 (94)	+1.2 (76)	+1.5 (105)	—	+1.7 (22)
T3	+1.0 (86)	+1.0 (61)	+1.0 (87)	+1.0 (99)	+1.1 (88)	+0.8 (24)	+1.1 (76)	+0.7 (50)	+1.0 (76)	—	+1.0 (33)
T4	+0.8 (82)	+0.8 (83)	+0.9 (96)	+0.9 (105)	+0.9 (122)	+0.4 (30)	+1.0 (75)	+0.9 (53)	+1.0 (85)	—	—
T5	+0.6 (61)	+0.3 (64)	+0.5 (73)	+0.5 (64)	+0.4 (78)	+0.2 (27)	+0.6 (59)	+0.5 (61)	+0.5 (68)	+0.8 (7)	—
T6	—	—	—	+0.4 (7)	+0.1 (5)	—	—	—	—	—	+0.2 (12)

Appendix

Table G.5: Holdout

\Delta

Gap by Theory. T1–T5 aggregate across all three scenarios; T6–T7 are ABD-Skeptical only. Each cell shows

\Delta

Gap = HGap

-

PGap over survivors valid on both prompt and holdout, with N = survivor count. Cells with fewer than 5 survivors are suppressed. Lower is better. Best per theory bold.

Model	PV	P+H	H $\|$ P	H $\|$ P%
Opus-4.6	593	591	373	63.1%
GPT-5.2	522	521	344	66.0%
Gemini-3.1	588	586	417	71.2%
Grok4.1f	512	511	440	86.1%
DSR	543	542	455	83.9%
GPT-5.4	511	509	145	28.5%

Appendix

Table G.6: Instance-Level Holdout Survival Counts. All counts are benchmark instances. H

|

P% = holdout validity conditional on prompt validity. Best values bold.

Theory	ABD-Full	ABD-Partial	ABD-Skeptical
T1	4/42 (9.5%)	5/42 (11.9%)	14/239 (5.9%)
T2	63/437 (14.4%)	32/604 (5.3%)	36/186 (19.4%)
T3	62/323 (19.2%)	16/431 (3.7%)	53/185 (28.6%)
T4	77/417 (18.5%)	48/573 (8.4%)	75/219 (34.2%)
T5	63/247 (25.5%)	30/258 (11.6%)	58/208 (27.9%)
T6	—	—	86/129 (66.7%)

Appendix

Table G.7: Reference-beating rate by theory (all models pooled).

k/N

= predictions beating the planted generator reference out of prompt-valid predictions.

Ref AST	ABD-Full	ABD-Partial	ABD-Skeptical
$[0,5)$	—	—	0/9 (0.0%)
$[5,10)$	—	—	51/254 (20.1%)
$[10,15)$	45/198 (22.7%)	33/510 (6.5%)	111/629 (17.6%)
$[15,20)$	112/546 (20.5%)	60/636 (9.4%)	67/187 (35.8%)
$[20,30)$	59/565 (10.4%)	30/595 (5.0%)	91/279 (32.6%)
$[30,+)$	53/157 (33.8%)	5/155 (3.2%)	11/38 (28.9%)

Appendix

Table G.8: Reference-beating rate by planted generator reference AST size (all models pooled).

Model	Valid%	Invalid%	Parse%	Repair%	Missing%
Opus-4.6	98.8%	0.5%	0.0%	0.0%	0.7%
GPT-5.2	92.2%	6.0%	1.8%	0.2%	0.0%
Gemini-3.1	98.0%	2.0%	0.0%	0.0%	0.0%
Grok4.1f	95.2%	3.9%	0.9%	0.7%	0.0%
DSR	95.9%	3.9%	0.2%	1.6%	0.0%
GPT-5.4	85.2%	14.0%	0.8%	6.7%	0.0%

Appendix

Table H.1: Failure Mode Breakdown. Aggregated across all abduction tasks. Parse = unrepaired parse failure; Repair = trailing right-parenthesis repair applied before evaluation; Invalid = parsed but not valid on all prompt worlds; Valid = prompt-valid predictions. Best values bold.

Failure Mode	ABD-Full	ABD-Partial	ABD-Skeptical
Auto-repaired parse	47	26	18
Parse error	87	43	86
All invalid (prompt)	208	343	218
Partial invalid (prompt)	288	270	54
Brittle (holdout)	357	583	571
of which catastrophic	121	174	332

Appendix

Table H.2: Failure-mode counts across all 6,367 model predictions.

Pattern	ABD-Full	ABD-Partial	ABD-Skeptical
Exists-only	126	215	225
Mixed quantifiers	79	32	47
Propositional (and)	0	9	19
Forall-only	0	3	9
Other	1	10	2

Appendix

Table H.3: Brittle formula syntactic patterns. “Exists-only” = existential quantifiers without universals.

Explore similar work

DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models
Jun 17, 2026Patrick Cooper, Alvaro VelasquezLLM Evaluation Abductive Reasoning
University of Colorado Boulder
Don't Let Me Ask for It: LLMs Show Deficiencies in Active Multi-Turn Information Acquisition for Abductive Inference
Aug 4, 2026Shahrukh Mohiuddin, Chalamalasetti Kranti, Sherzod Hakimov +1LLM Evaluation Language Model Probing
Computational Linguistics, Department of Linguistics University of Potsdam, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
Graph of States: Solving Abductive Tasks with Large Language Models
Mar 22, 2026Yu Luo, Rongchen Gao, Lu Teng +9Abductive Reasoning LLM Reasoning with Graphs
Nankai University · Wenzhou Medical University · Alibaba Cloud +2

Jun 17, 2026J/K move · Enter open · S save

DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models

Patrick Cooper, Alvaro Velasquez

University of Colorado Boulder

LLM Evaluation Abductive Reasoning Symbolic Reasoning Nonmonotonic Reasoning Synthetic Benchmark Generation Structured Reasoning

Open paper AskPDF arXiv

A rule-based logic solver resolves every instance in our benchmark in under 50 microseconds with 100% accuracy; the best frontier language model reaches 65% at best and drops to 23.5% under rendering-robust evaluation (worst case over four surface renderings). We introduce DeFAb (Defeasible Abduction Benchmark), a dataset and generation pipeline that converts four decades of publicly funded knowledge bases into formally grounded instances for defeasible abduction: constructing hypotheses that explain anomalies by overriding defaults while preserving unrelated expectations. Because every hypothesis must pass polynomial-time checks for valid derivation, conservativity, and minimality, DeFAb makes logical rigor the instrument for measuring creativity and theoretical reasoning, scoring the disciplined construction of theory revisions rather than fluent but theory-destroying prose. The pipeline pairs taxonomic hierarchies (OpenCyc, YAGO, Wikidata) with behavioral property graphs (ConceptNet, UMLS) to produce 372,648+ instances across 33.75M materialized rules from 18 sources, in three levels with polynomial-time verifiable gold standards. Four frontier models do not reliably internalize defeasible reasoning: rendering-robust Level 2 accuracy is 7.8-23.5%; chain-of-thought variance (~36 pp) exceeds any inter-model gap; and a matched contamination control isolates a +19.4 pp Level 3 gap. We further release DeFAb-Hard (a 235-instance Level 3 difficulty variant; best model 53.3% vs 100% symbolic) and CONJURE (a kernel-verified transformative-creativity variant of 560 Lean 4/Mathlib instances whose gold answers are definitions the proof kernel did not previously contain, judge-free verifier; a pilot finds zero novel concepts). The same verifier doubles as an exact reward for preference optimization (DPO, RLVR/GRPO). Released under MIT at https://huggingface.co/datasets/PatrickAllenCooper/DeFAb.

Abstract

Explore similar work

DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models

Don't Let Me Ask for It: LLMs Show Deficiencies in Active Multi-Turn Information Acquisition for Abductive Inference

Graph of States: Solving Abductive Tasks with Large Language Models