Language models increasingly write the explanations people receive when a rule system denies their request. If several rules deny the request independently, an explanation can name a true reason and still leave the person denied after they fix it. We call the missing property recourse validity: changing what the explanation cites, without changing what was asked for, can flip the decision. We compute it exactly, and a hitting-set argument predicts from the policy alone when a correct reason fails. In six of seven rule systems ported from access control and US benefit eligibility, a single correct reason almost never gives recourse when several reasons apply. Across 20 LLMs, checks of whether the cited reasons are correct, computed or asked of an LLM judge, are close to chance at spotting this failure, as the argument predicts. A strong judge asked whether acting on the explanation would work does spot it. Stronger and reasoning models do better but still fail on many denials with four or more independent reasons. Asking for every rule helps every open model with room to improve. Explanations of eligibility decisions should be built from the decision function and judged by whether acting on them works.
Figures & tables
Figure 2: The oracle computes the goal-fixed reasons RG(x) from the gate, and an extractor reads the features C an explanation cites. Correctness checks ask whether C contains a reason; recourse validity asks whether C hits all of them, which one reason does only if it shares a feature with every other (Proposition 1 ).
Two+
Left denied
No single
No
Rule system
∣F∣
Contexts
Goal G
Denies
reasons
free
fixed
actionable
suffices
recourse
Cedar document-cloud
10
5,376
action
4,504
1,900
0.88
0.96
0.998
0.95
0.15
Cedar github-model
7
704
action
336
16
0.00
0.00
0.00
0.00
0.00
Gatekeeper allowed repos
5
576
none
448
192
1.00
1.00
1.00
1.00
0.00
Gatekeeper privileged
9
1,152
is_update
276
56
0.00
1.00
1.00
1.00
0.00
SNAP eligibility
10
3,456
none
3,108
2,176
0.96
0.96
0.92
0.95
0.45
Table 1: Rule systems and the oracle-level gap, exact over every context. Cedar and the eligibility rules use deny-overrides, Gatekeeper violation-any on 3-container pods. Two+ reasons : denials with two or more goal-fixed reasons. Left denied : share of (denial, reason) pairs in which citing that one correct reason leaves the request denied, over denials with two or more reasons (free) or two or more goal-fixed reasons (fixed; actionable also fixes the immutable features). No single suffices : share of those denials in which no reason cited alone gives goal-fixed recourse. No recourse : share that the goal and immutable features alone deny. Shares are under the uniform context measure, not population rates.
Figure 3: How well each per-explanation signal separates recourse-valid explanations from the rest (AUROC, 95% interval from resampling episodes; dotted line: chance). Circles: 20,375 default explanations of multi-reason denials from all 20 models; squares: 4,072 thinking-on explanations; triangles: single-reason controls, small open models. Judges are GPT-6 Luna on a sample (§ 5.3 ). Within-model and within-episode values are in Table 8 .
Figure 4: RVG by the number of goal-fixed reasons on the multi-reason denials ( n episodes per bucket). It falls in every group, reasoning models included. Bold: mean over a group’s models; faint: one line per model. Thinking on: Qwen3-14B, Qwen3-32B, Qwen3.8-27B, and gpt-oss-20b at high effort. Per-bucket values are in Table 7 .
Correct
Actionable
Quality
Difference
Judge
4+
AUROC
AUROC
AUROC
AUROC
GPT-6 Luna
84%
0.62
0.97
0.95
0.02
Gemma-4-31B
87%
0.57
0.92
0.93
− 0.01
Qwen3.8-27B
89%
0.56
0.67
0.84
− 0.16
Table 2: LLM judges on default explanations of multi-reason denials, 20 per model and system, 2,800 per judge over 20 models. 4+ : share rated 4 or 5 on correct. AUROC: how well each 1 to 5 rating separates recourse-valid explanations from the rest; Difference : actionable minus quality AUROC, paired.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
First-match header
Order-neutral header
List-all
Model
Valid reason
Fidelity
Two or more
RV
Two or more
RV
Two or more
RV
Llama-3.1-8B
0.79
0.46
0.14
0.49
0.20
0.45
0.75
0.90
Gemma-3-12B
0.75
0.36
0.21
0.54
0.34
0.66
0.71
0.99
Qwen3-14B
0.91
0.56
0.28
0.50
0.80
0.89
0.91
0.995
Phi-4-14B
0.97
0.59
0.50
0.67
0.91
0.94
0.97
0.999
Qwen2.5-14B
0.99
0.52
0.69
0.78
0.91
0.88
0.93
0.97
Appendix
Table 3: Synthetic gate, 100 multi-reason block episodes, 1,000 explanations per cell (1,500 for Haiku and Qwen3 under first-match). Valid reason : share that disclose at least one complete reason. Fidelity : share that disclose the reason of the rule that fires first in the order shown. Two or more : share that disclose at least two. RV : recourse-valid share among explanations that cite at least one feature. Episode-level 95% bootstrap intervals are at most 0.13 wide for two or more and 0.14 for RV .
First-match header
Order-neutral header
Model
First listed
Switch
Concentration
p
Llama-3.1-8B
0.54 (0.36)
0.47
0.52 (0.29)
< 0.001
Gemma-3-12B
0.39 (0.37)
0.28
0.44 (0.23)
< 0.001
Qwen3-14B
0.55 (0.38)
0.30
0.30 (0.11)
< 0.001
Phi-4-14B
0.36 (0.40)
0.09
0.08 (0.08)
0.77
Qwen2.5-14B
0.41 (0.44)
0.16
0.20 (0.10)
< 0.001
Appendix
Table 4: Position and concentration on the synthetic gate. First listed : among explanations that disclose exactly one valid rule, the share that disclose the rule listed first, chance 1/k in parentheses. Switch : share of episodes whose most-disclosed rule changes when the order is reversed. Concentration : one minus normalised entropy of disclosed reasons per cell, uniform null in parentheses, paired permutation p .
What is cited
Default, all systems
Default, eligibility
List-all
Recall 1,
Model
Fidelity
Precision
Facts
RVG [95% CI]
Random
RVG
RVA
RVG
denied
Open, 8B to 21B
Llama-3.1-8B
0.75
0.75
2.9
0.31 [0.27, 0.35]
0.24
0.09
0.13
0.55
66%
Granite-4.2-8B
0.96
0.76
4.7
0.65 [0.59, 0.70]
0.43
0.37
0.46
0.88
36%
gpt-oss-20b
0.87
0.91
3.0
0.55 [0.50, 0.60]
0.23
0.44
0.50
0.90
45%
Mistral-Nemo-12B
0.74
0.81
2.5
0.24 [0.20, 0.29]
0.20
0.05
0.08
0.58
78%
Appendix
Table 5: The 210 multi-reason denials, default prompt unless a column says otherwise. Fidelity : discloses a reason the policy’s trace reports. Facts : mean facts cited. RVG : mean of per-episode means with its 95% bootstrap interval. Random : RVG of as many random facts of the episode. Eligibility : mean over SNAP, Title IV and HUD; RVA also fixes immutable facts and is defined on fewer denials. Recall 1, denied : share of explanations with causal recall 1 that leave the requester denied.
Default
List-all
Control
System
Fidelity
Two+
RVG
RVG
RVG
Cedar doc-cloud
0.94
0.37
0.71
0.92
0.97
Cedar github
0.58
0.19
0.91
0.98
0.92
OPA repos pod
0.87
0.29
0.49
0.84
0.91
OPA privileged pod
0.86
0.23
0.50
0.92
0.81
SNAP
0.96
0.61
0.41
0.78
0.85
Appendix
Table 6: Per system, mean over the 11 small open models. Two+ : share that disclose two or more reasons. Control : the 30 single-reason denials per system.
Small open
Thinking
Frontier
∣RG(x)∣
Default
List-all
Default
Default
Change
1
0.89
0.96
1.00
1.00
1.0
2
0.59
0.90
0.97
0.99
1.9
3
0.33
0.71
0.92
0.98
2.9
4+
0.12
0.43
0.76
0.84
4.5
Appendix
Table 7: RVG by number of goal-fixed reasons, the values behind Figure 4 : mean over the 11 small open models on both strata (221, 102, 50 and 34 episodes, the 407 of 420 with a defined verdict) and over the four thinking-on and five frontier models on the multi-reason stratum. Change : mean size of a minimum hitting set of RG(x) .
Within
Signal
AUROC
95% CI
model
episode
Is what it cites correct?
Fidelity
0.49
[0.47, 0.50]
0.48
0.56
Causal recall 1
0.48
[0.47, 0.50]
0.47
0.56
Causal precision
0.48
[0.46, 0.50]
0.45
0.58
How much does it cover?
Appendix
Table 8: Signal AUROCs behind Figure 3 : how well each per-explanation signal separates recourse-valid explanations from the rest, over 20,375 default explanations of multi-reason denials from all 20 models. The interval resamples episodes. Within model : mean of per-model AUROCs, which a signal cannot raise by telling models apart. Within episode : mean over the 202 episodes with both outcomes, which fixes the reason count. Two+ reasons : cites at least two of the episode’s reasons in full; Coverage : share of the episode’s reasons cited in full (both with the goal free).
Figure 5: Fidelity against goal-fixed recourse validity on the 210 multi-reason denials, one point per model, default prompt (Spearman 0.14 over 20 models). Grey lines rise to each model’s list-all value.
Policy shown
No policy
Oracle content
Model
Default
Requester
Capped
List-all
Uncapped
Counterfactual
Free-all
Min. change
Reasons
Qwen3-14B
0.40 0.88
0.59 0.75
0.36 0.94
0.79 0.89
0.88 0.75
0.34 0.92
0.71 0.75
0.90 0.96
0.98 0.98
Gemma-3-12B
0.23 0.90
0.37 0.82
0.27 0.90
0.84 0.75
0.84 0.71
0.25 0.93
0.81 0.69
0.79 0.96
0.997 0.86
Llama-3.1-8B
0.31 0.75
0.53 0.70
0.28 0.75
0.55 0.68
0.67 0.67
0.29 0.72
0.55 0.68
0.58 0.90
0.87 0.87
Appendix
Table 9: Prompt and content ladder, RVG on the 210 multi-reason denials with causal precision in small type. The oracle-content columns need a pipeline that computes the reasons from the decision function.
Conditions
Episodes
Models
Open, 8B to 21B
default, list-all
420
all 11
requester
210
all 11
requester, 2 to 4 sentences
210
Qwen3-14B, Gemma-3-12B, Llama-3.1-8B
free, free-all
420
10, not gpt-oss-20b
uncapped, counterfactual, min. change
210
Qwen3-14B, Gemma-3-12B, Llama-3.1-8B
Appendix
Table 10: Every generation run, 5 samples per episode and condition. The high-effort reruns cover only the 34 multi-reason denials with four or more goal-fixed reasons. The 20 models produced 141,750 explanations: 118,650 from the small open models, 10,500 from the large and 12,600 from the frontier models; the settings runs are counted apart.
Model
ECI
RVG
Llama-3.1-8B
116.6
0.31
Mistral-Nemo-12B
118.7
0.24
Gemma-3-12B
123.5
0.23
Gemma-3-27B
130.0
0.44
Phi-4-14B
130.4
0.57
gpt-oss-20b
137.8
0.55
Appendix
Table 11: Epoch Capabilities Index values ( Epoch AI, 2025 ) , read on 6 October 2026, and default goal-fixed recourse validity. Models without a published value are left out of the correlations.
Department of Philosophy, Vanderbilt University · Department of Philosophy, University of Michigan · School of Government and Policy, Johns Hopkins University