Deogen: Automatic Benchmark Generation for Reasoning over Policies

Sam Kouteili; William Liang; Keeran Murray; Adriana Zhou; Ramesh Viswanathan; Scott Shapiro; Ruzica Piskac

cs.AIOct 28, 2026

Deogen: Automatic Benchmark Generation for Reasoning over Policies

Authors: Sam Kouteili, William Liang, Keeran Murray, Adriana Zhou, Ramesh Viswanathan, Scott Shapiro, Ruzica Piskac

Organizations: Yale University · Leibniz AI, Inc. · Yale Law School

Abstract

Language models are increasingly being deployed in regulated settings, where a model must apply given rules to the facts of a situation to determine a verdict. Benchmarking this capability, known as deontic reasoning, remains a costly and manual process, with existing datasets largely hand-crafted. In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning. By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories. Each path down a tree becomes a benchmark entry, with each tree producing combinatorially many entries relative to tree depth. Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information. Evaluations show frontier models return the correct verdict 89-92% of the time. Notably, performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases. In all, DeoGen presents an automatic, scalable, and parametric method for generating deontic reasoning benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: DeoGen pipeline, illustrating transformation from text to a decision structure and its tabular representation.

Panel of Figure 2 — Figure 2: Fragment of the AFL medical-expense tree. The plan document admits long-term-care insurance premiums only when they are eligible under the IRS Code, so the qualified-contract branch expands directly into a subtree compiled from §213(d)(10). Numeric values are handled by constructing trees that segment the number bins.

		Total possible entries		Evaluation set
Verdict Questions	Tree Depth	Def.	NEI	Def.	NEI
IRS Code (§§105(b), 125, 129, 152, 213; 9,156 words, 109 assertions)
Is this individual a “dependent” of the taxpayer under §152?	25	2,125	2,048	100	100
(isDependent)
Is the person a qualifying child, as defined in §152(c)?	16	144	156	61	54
(isQualifyingChild)

Table 1: Benchmark statistics. For each determination: maximum decision-tree depth; the population of possible benchmark entries: definite entries (root-to-leaf paths) and NEI entries (root-to-internal-node paths). For the evaluation set, we draw up to 100 verdict-reaching + 100 NEI determinations, unless generated trees do not contain so many paths.

	IRS Code		AFL		Fellows		Average
Model	Def.	NEI	Def.	NEI	Def.	NEI	Def.	NEI
Claude Sonnet 5	91.0	32.1	93.7	9.4	100.0	60.0	92.1	22.8
GPT-5.5	88.3	55.8	91.3	19.1	62.5	80.0	89.2	40.0
Qwen3.6-35B-A3B	83.9	5.9	92.2	1.5	87.5	0.0	87.2	4.0
Gemma 4 31B	90.6	34.5	95.5	15.2	87.5	80.0	92.5	26.7
Gemma 4 E4B	71.3	22.9	80.5	11.6	87.5	20.0	75.1	18.2

Table 2: Accuracy (%) on definite scenarios (Def.) and on underdetermined scenarios (NEI), whose keyed answer is Not enough information , per policy and averaged over all entries (AFL aggregates the plan document and certificate). Zero-shot; deterministic grading. Binomial 95% CIs are at most

\pm 3.5

points; every definite–NEI gap is significant (

p{<}0.001

, two-proportion

z

-tests), the Gemma 4 31B–Claude definite gap is not.

Figure 3: Accuracy as a function of the number of facts shown. Path depth acts as a difficulty dial.

Figure 4: Error decomposition across zero-shot evaluations on the binary-verdict policies (IRS and AFL; Fellows excluded). Bars show the distribution of outcomes within definite ( $n=842$ per model) and underdetermined ( $n=784$ per model) scenarios. Definite-case errors are divided into wrong verdicts and spurious abstentions; underdetermined-case errors are unwarranted commitments to a verdict. One Qwen3.6 NEI response was unparsed.

Model	Recall	Precision
Claude Sonnet 5	0.293	0.824
GPT-5.5	0.257	0.852
Qwen3.6-35B-A3B	0.194	0.935
Gemma 4 31B	0.194	0.884
Gemma 4 E4B	0.162	0.912

Table 3: Citation grounding against the provision-level gold for AFL and IRS: recall (did the model cite every provision its scenario depends on) and precision (were the cited provisions correct).

	Zero-shot		Few-shot
Model	Def.	NEI	Def.	NEI
Claude Sonnet 5	92.1	22.8	91.4	35.7
GPT-5.5	89.2	40.0	87.1	51.1
Qwen3.6-35B-A3B	87.2	4.0	90.0	1.5
Gemma 4 31B	92.5	26.7	92.5	36.4
Gemma 4 E4B	75.1	18.2	74.0	22.3

Table 4: Few-shot ablation, averaged over all entries: worked examples including an NEI demonstration are drawn from a held-out determination. Demonstrating abstentions generally improves NEI performance; the 10–13-point gains are significant (

p{<}0.001

Measure	Result
Human–gold label agreement ( $n$ = 50, 2 annotators)	32 %
NL rendering judged faithful to facts	92 %
Inter-annotator agreement	52 %

Table 5: Human validity study on a stratified sample. Annotators answer from the natural-language scenario and policy alone; they struggle to recover the gold label, yet judge the renderings faithful.

Figure 5: Accuracy across zero-shot scenarios in AFL and IRS which included age determinations (three IRS, two AFL). Every model performs worse on atypical-age scenarios ( $n=138$ ) than typical ones ( $n=777$ ).

Synthetic Benchmark Generation

Synthetic benchmark generation creates controlled, verifiable evaluation tasks and datasets to test models across diverse settings.

See all 288 papers

DeonticBench: A Benchmark for Reasoning over Rules
Apr 6, 2026Guangyao Dou, Luis Brena, Akhil Deo +4Logical Reasoning Legal Reasoning
Johns Hopkins University · Télécom Paris, Institut Polytechnique de Paris
DAR: Deontic Reasoning with Agentic Harnesses
Jun 3, 2026Guangyao Dou, William Jurayj, Nils Holzenberger +1Logical Reasoning Agentic Reasoning
Johns Hopkins University · Télécom Paris, Institut Polytechnique de Paris
RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications
Sep 28, 2026Jianpeng Zhao, Haihua Xu, Haoyang Zhang +7LLM Safety Benchmarks LLM Decision-Making
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China · ByteDance, Beijing, China

Apr 6, 2026J/K move · Enter open · S save

DeonticBench: A Benchmark for Reasoning over Rules

Guangyao Dou, Luis Brena, Akhil Deo, William Jurayj+3

Johns Hopkins University · Télécom Paris, Institut Polytechnique de Paris

Logical Reasoning Legal Reasoning Deontic Logic Program Synthesis

Open paper AskPDF arXiv

Reasoning with complex, context-specific rules remains challenging for large language models (LLMs). In legal and policy settings, this manifests as deontic reasoning: reasoning about obligations, permissions, and prohibitions under explicit rules. While many recent benchmarks emphasize short-context mathematical reasoning, fewer focus on long-context, high-stakes deontic reasoning. To address this gap, we introduce DEONTICBENCH, a benchmark of 6,232 tasks across U.S. federal taxes, airline baggage policies, U.S. immigration administration, and U.S. state housing law. These tasks can be approached in multiple ways, including direct reasoning in language or with the aid of symbolic computation. Besides free-form chain-of-thought reasoning, DEONTICBENCH enables an optional solver-based workflow in which models translate statutes and case facts into executable Prolog, leading to formal problem interpretations and an explicit program trace. We release reference Prolog programs for all instances. Across frontier LLMs and coding models, best hard-subset performance reaches only 44.4% on SARA Numeric and 46.6 macro-F1 on Housing. We further study training with supervised fine-tuning and reinforcement learning for symbolic program generation. Although training improves Prolog generation quality, current RL methods still fail to solve these tasks reliably. Overall, DEONTICBENCH provides a benchmark for studying context-grounded rule reasoning in real-world domains under both symbolic and non-symbolic settings.

Abstract

Synthetic Benchmark Generation

Explore similar work

DeonticBench: A Benchmark for Reasoning over Rules

DAR: Deontic Reasoning with Agentic Harnesses

RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications