Reasoning with complex, context-specific rules remains challenging for large language models (LLMs). In legal and policy settings, this manifests as deontic reasoning: reasoning about obligations, permissions, and prohibitions under explicit rules. While many recent benchmarks emphasize short-context mathematical reasoning, fewer focus on long-context, high-stakes deontic reasoning. To address this gap, we introduce DEONTICBENCH, a benchmark of 6,232 tasks across U.S. federal taxes, airline baggage policies, U.S. immigration administration, and U.S. state housing law. These tasks can be approached in multiple ways, including direct reasoning in language or with the aid of symbolic computation. Besides free-form chain-of-thought reasoning, DEONTICBENCH enables an optional solver-based workflow in which models translate statutes and case facts into executable Prolog, leading to formal problem interpretations and an explicit program trace. We release reference Prolog programs for all instances. Across frontier LLMs and coding models, best hard-subset performance reaches only 44.4% on SARA Numeric and 46.6 macro-F1 on Housing. We further study training with supervised fine-tuning and reinforcement learning for symbolic program generation. Although training improves Prolog generation quality, current RL methods still fail to solve these tasks reliably. Overall, DEONTICBENCH provides a benchmark for studying context-grounded rule reasoning in real-world domains under both symbolic and non-symbolic settings.
Figures & tables
Figure 1: Walkthrough of a DeonticBench instance in the symbolic setting. (1) Given the full problem context, the model performs deontic reasoning to identify and apply the relevant rules. (2) The LLM translates the problem into Prolog code. (3) The generated Prolog is executed by SWI-Prolog solver. The illustrated example is a 2017 tax-liability case.
Domain
Split
#Tasks
Tokens (Mean)
Statute
Case
Prolog
SARA Numeric
Whole
100
6118
83
945
Hard
35
6118
89
1236
SARA Binary
Whole
276
6118
47
361
Hard
30
6118
52
453
Airline
Whole
300
3626
187
880
Table 1: Statistics of datasets, including counts of tasks and average token lengths (via cl100k_base tokenizer) by split.
Accuracy
Macro F1
Model
Setting
SARA Num.
Airline
SARA Bin.
USCIS-AAO
Housing
GPT-4.1
Few-Shot
23.7−12.3+13.5
41.5−11.5+11.0
39.1−18.1+17.2
53.0−18.6+18.3
46.6−10.8+11.6
Zero-Shot
6.7−6.7+10.5
1.7−1.7+3.3
40.5−18.0+16.4
55.5−19.1+18.7
44.7−10.9+10.5
Direct
18.8−10.2+12.6
6.7−5.4+5.8
30.3−15.2+15.5
50.9−19.6+16.5
20.2−7.9+10.1
O3
Few-Shot
15.2−12.4+13.3
90.8−7.0+5.5
29.5−15.6+16.9
49.4−18.1+18.4
43.0−10.1+10.0
Zero-Shot
44.4−15.8+15.6
18.5−8.5+9.0
32.3−15.4+17.5
48.5−18.5+18.7
39.6−10.7+11.6
Table 2: Main results across five domains. SARA Numeric and Airline report accuracy; SARA Binary , USCIS , and Housing report macro F1. Values are reported as mean −lower+upper , where upper and lower denote distances from the mean to the 97.5th and 2.5th bootstrap percentiles, respectively.
Figure 2: Performance decomposition analysis for SARA Numeric and Airline. Each model shows three bars (left to right: Direct, Zero-Shot, Few-Shot). Plots for other domains are in Figure 4 (Appendix C.3 ).
Accuracy
Macro F1
Model
Setting
SARA Num.
Airline
SARA Bin.
USCIS-AAO
Housing
Qwen2.5-32B (base)
Few-Shot
1.3−1.3+4.4
15.9−7.1+7.9
29.1−15.2+17.3
10.3−10.3+10.7
35.1−8.8+10.1
Zero-Shot
0.7−0.7+5.0
0.0−0.0+0.0
18.2−11.5+13.9
42.8−17.9+17.5
30.9−9.7+10.5
Direct
4.4−4.4+7.1
0.9−0.9+2.9
62.8−17.7+15.8
36.1−18.4+17.4
24.3−8.4+8.9
SFT
Few-Shot
1.3−1.3+4.4
53.5−11.0+11.5
41.6−17.5+18.7
35.0−17.3+18.3
42.0−11.3+10.5
Zero-Shot
2.2−2.2+6.4
0.0−0.0+0.0
26.4−15.8+18.2
48.5−18.6+18.3
40.5−10.1+10.7
Table 3: Results for locally trained models across five domains. SARA Numeric and Airline report accuracy; SARA Binary , USCIS-AAO , and Housing report macro F1. Values are reported as mean −lower+upper , where upper and lower denote distances from the mean to the 97.5th and 2.5th bootstrap percentiles, respectively.
Domain
Prompt
Wrong Rule
Entity / Fact
Numerical
Impl. Bugs
SARA Numeric
Few-Shot
31%
42%
22%
6%
Zero-Shot
35%
52%
13%
—
SARA Binary
Few-Shot
55%
45%
—
—
Zero-Shot
47%
42%
12%
—
Airline
Few-Shot
—
—
100%
—
Zero-Shot
12%
6%
75%
7%
Table 4: Distribution of error types across domains and prompting strategies, based on hard cases where GPT-5.1, GPT-5.2, and O3 failed on every K generation. Percentages are normalized per row; the dominant category per row is bolded . Wrong Rule : the model applied the wrong statute, subsection, or fee schedule. Entity / Fact : correct rule but wrong facts encoded (e.g., filing status, dependency relationships, bag type). Numerical : correct rule and facts but arithmetic or threshold errors (e.g., phaseout computation, bag count optimization). Impl. Bugs : low-level Prolog programming errors unrelated to reasoning.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
SFT
DPO
Base model
Qwen2.5-32B-Instruct
SFT checkpoint
Method
LoRA
LoRA
LoRA rank
8
8
LoRA target modules
All
All
Sequence length
8,000
8,000
Per-device batch size
1
1
Appendix
Table 5: Training hyperparameters for SFT and DPO fine-tuning of Qwen2.5-32B-Instruct.
Parameter
DR. GRPO
Base model
DPO checkpoint
Method
LoRA
LoRA rank ( r )
128
LoRA α
128
Learning rate
2×10−6
Epochs
5
Appendix
Table 6: Training hyperparameters for DR. GRPO training.
Accuracy
Macro F1
Model
Setting
SARA Num.
Airline
SARA Bin.
USCIS-AAO
Housing
GPT-5.2-Codex
Few-Shot
26.6−12.3+13.4
95.5−5.5+3.2
28.8−16.3+17.0
34.9−15.2+17.1
41.3−9.1+9.9
Zero-Shot
45.8−14.3+17.1
25.6−9.4+10.6
16.6−13.2+13.4
47.4−17.4+18.3
31.3−9.8+10.1
Direct
29.5−15.2+13.3
47.0−10.7+10.5
36.5−16.8+18.5
52.7−18.2+19.4
17.5−7.8+8.5
Qwen3-Coder-Next
Few-Shot
0.0−0.0+0.0
0.0−0.0+0.0
28.9−15.1+16.9
16.4−13.0+13.5
36.9−10.0+11.3
Zero-Shot
0.0−0.0+0.0
0.0−0.0+0.0
21.7−15.1+15.6
24.5−12.0+14.8
32.8−10.5+10.8
Appendix
Table 7: Results for coding LLMs across five domains. SARA Numeric and Airline report accuracy; SARA Binary , USCIS-AAO , and Housing report macro F1. Values are reported as mean −lower+upper , where upper and lower denote distances from the mean to the 97.5th and 2.5th bootstrap percentiles, respectively.
Figure 3: Reasoning effort ablation on SARA Numeric hard cases. Each panel corresponds to a different prompting strategy (Direct, Zero-Shot, and Few-Shot). Bars indicate mean accuracy, with error bars showing 95% bootstrap confidence intervals.
Figure 4: Performance decomposition analysis for SARA Binary, Housing, and USCIS. Each model shows three bars (left to right: Direct, Zero-Shot, Few-Shot). The colored portion represents correct predictions; the solid pink portion represents wrong answers; the hatched pink portion represents abstentions (Prolog timeout or crash).
Subset
Cases
Accepted
Dismissed
Total
242
121 (50.0%)
121 (50.0%)
2022
11
2 (18.2%)
9 (81.8%)
2023
19
17 (89.5%)
2 (10.5%)
2024
188
78 (41.5%)
110 (58.5%)
2025
24
24 (100.0%)
0 (0.0%)
Hard
28
14 (50.0%)
14 (50.0%)
Appendix
Table 8: Descriptive statistics for the USCIS-AAO dataset are reported in aggregate and for the hard subset. The upper panel presents overall totals, while indented subrows provide year-specific case counts and within-year percentages for Accepted and Dismissed cases. Mean token counts are computed using the cl100k_base encoding.
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China · ByteDance, Beijing, China
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences