When a language model decides refund or claim requests under written policies, the system can show it similar past cases. Each past case has two parts: its facts and the decision recorded for it. We separate the two. A model sees a policy, a request and three past cases, and we change either the recorded decisions or the facts, keeping the request fixed. We select requests that are close calls: deciding without past cases, DeepSeek V4.1 Flash approved each in some of four tries and denied it in others. When the past cases are near-copies of the request (its facts with a few details changed), switching their recorded decisions from DENY to APPROVE raises approval by about 30 to 55 percentage points in DeepSeek V4.1 Flash, GPT-5.6 Luna and Claude Haiku 5.5 on policies written for the study, and DeepSeek and Luna show the same on nine public policies paraphrased from real ones. The facts push the other way: shown without their recorded decisions, past cases whose facts support approval make these two models approve less often than past cases whose facts support denial. One reading is that the request looks weaker next to them. This push is not just a correction toward the right answer: in an additional DeepSeek analysis, it was of similar estimated size on requests that human annotators judged approvable and on those they judged deniable. In a separate DeepSeek run on the public policies, when the past cases are near-copies of a different request, switching their decisions moves approval by 9 points instead of 43, while facts that support approval still lower it by 22 points against facts that support denial. Systems that retrieve past decisions should test the two parts separately.
Figures & tables
Policy (Steam refunds, paraphrased; 6 rules). 1. Refund for any reason within 14 days of purchase and under 2 hours of play. 2. Requests outside these limits can be submitted and are reviewed; a refund is not guaranteed. 3. In-game purchases in non-Valve games are not refundable. … 6. A VAC-banned account loses the right to refund that game.
Target request. Co-op survival game; purchased 9 days before the request; 3 hours played; no refunds in the past year; “an update after the purchase removed the online co-op mode advertised on the store page”.
Retrieved past cases (returned by a lookup tool). Three close variants of the target, each keeping the customer’s co-op complaint, each labeled DENY by the case author: (1) in-game currency for a non-Valve survival game, not spent; (2) the same game with a VAC ban on this game; (3) the same game purchased 200 days before the request, 120 hours played. Each ends with a recorded Decision: line, or with none.
Figure 1: Changing only the recorded decisions of three precedents changes the decision on a borderline request. One sampled decision per condition (DeepSeek V4.1 Flash) from the public-policy experiment (Section 4.1 ). Fact sheets are abridged and the reason is quoted verbatim. Over six repeated decisions (seeds) on this target the request was approved in 1/6 without precedents, 0/6 with DENY labels, 6/6 with APPROVE labels and 0/6 without decision lines.
Public policies
Fictional policies
Policies
9 public policies, paraphrased
6 fictional organizations
Targets
65
34
Cases written by
the case author, without model assistance
targets by hand or drafted by DeepSeek; close variants drafted by DeepSeek
Reference label
author label: the case author’s reading of the rules
confirmed label: DeepSeek’s decision in two of two calls
Precedents per target
3 approve-side and 3 deny-side variants, fixed
3 per side, drawn per seed from 3–6
Unlabeled precedents
yes
in the delivery and re-verification experiments
Table 1: The two sets of materials.
Experiment: question
Policies; targets
Models
Conditions
Seeds
Reported
Main
Fictional experiment: do labels move three models?
6 fictional; 34
DeepSeek, Luna, Haiku
2 fact sets × APPROVE or DENY labels; Luna and Haiku also no precedents
4; Haiku 2
§ 4.1
Public-policy experiment: do labels and facts move decisions on public policies?
9 public; 65
DeepSeek
2 fact sets × 3 labels; no precedents
6
§ 4.1 – 4.2
Luna replication: does a second model show both?
9 public; 65
Luna
as the public-policy experiment
6
§ 4.1 – 4.2 , App. C
Robustness
Delivery experiment: does the turn that carries the precedents matter?
6 fictional; 34
DeepSeek
2 fact sets × 3 labels, as a user turn, an assistant turn or a tool result; close variants, or other clear cases of the policy drafted by DeepSeek
4
§ 4.1 – 4.2 , App. F
Table 2: Experiments and checks. Fact sets: approve-side and deny-side precedents. Labels: APPROVE, DENY or none on all three precedents. Targets are borderline for DeepSeek unless stated.
Labels on the three precedents
Approve-side facts
Deny-side facts
APPROVE
44.1
73.6
none
33.1
59.2
DENY
5.6
28.7
Table 3: APPROVE labels raise approval and approve-side facts lower it. Top: approval rate (%) of the target by the labels (rows) and facts (columns) of its three precedents; L averages the APPROVE-minus-DENY differences of the two columns; F is the approve-side minus deny-side rate in the unlabeled row. Public-policy experiment, DeepSeek V4.1 Flash, 390 decisions per cell (65 targets × 6 seeds); without precedents it is 55.1%. Bottom: label effect L and facts contrast F in points with 95% intervals. Fictional policies: fictional experiment, 34 targets. Public policies: public-policy experiment (DeepSeek) and Luna replication, 65 targets. –: not run.
Figure 2: For DeepSeek, in every public policy the label effect is positive and the facts contrast negative by point estimate. Per-policy label effect L (filled circles) and facts contrast F (open squares), DeepSeek V4.1 Flash, public-policy experiment (tool-result delivery). Bars are 95% bootstrap intervals that resample seeds within the policy with its targets held fixed, so they show seed variation only. Policies are sorted by L . Each has 5–11 targets and 6 seeds.
Own close variants
Donor’s close variants
Own minus donor
Label effect L
+42.7 [31.5, 53.4]
+9.0 [0.0, 16.5]
+33.7 [21.9, 45.7]
Holm p (policy-level p)
0.0009 (0.0039)
0.0032 (0.0547)
0.0009 (0.0039)
Facts contrast F
− 29.0 [ − 42.8, − 15.2]
− 22.3 [ − 34.8, − 9.3]
− 6.7 [ − 22.5, 8.6]
Holm p (policy-level p)
0.0009 (0.0039)
0.0002 (0.0078)
0.2547 (0.3711)
Table 4: Another request’s precedents lose most of the label effect and keep a facts contrast. Own-and-donor experiment, DeepSeek V4.1 Flash, 65 public-policy targets × 6 seeds, all cells in one run. Points with 95% intervals; below each row, the Holm p within its family (Appendix E ) and, in parentheses, the exact policy-level p.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Condition
r 10%
r 25%
r 50%
t 10%
t 25%
Rule-case accuracy
Direct use
+8.5
+18.3
+30.1
+53.3
+40.2
94.7
Direct use, re-derive instruction
+5.9
+10.8
+25.5
+43.1
+39.2
97.0
Facts without labels
+3.9
+2.9
+10.8
+17.6
+16.7
96.2
Full verification, DeepSeek
− 1.0
− 2.0
+3.9
+7.8
+4.9
97.0
Full verification, Haiku
+3.9
+3.9
+8.8
+8.8
+9.8
96.2
Selective verification, DeepSeek
+2.0
+5.9
+9.8
+10.8
+8.8
94.7
Appendix
Table 5: Deviation from clean-label direct use by condition. Additional analysis. Deviation from clean-label direct use, signed by the direction of the store’s flipped labels (points, point estimates), on the 34 borderline targets by condition and noise level (random r, targeted t, share of the store mislabeled), and accuracy (%) on the 44 rule cases at random 25% noise. Conditions that never see the stored labels are decided once per target and seed, so they vary across columns only through the direction and the share of pairs with s=0 . The no-precedent and facts-only rows show the deviation of conditions that ignore the stored labels. 3 seeds.
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
An ordinary-looking background detail can turn a correct model decision into a confident mistake. We demonstrate this fragility in four decision systems, including Jev, across seven datasets covering knowledge, reasoning, and tool routing. Within 64 accepted target evaluations per decision, we uncover short context additions that redirect 61.4%-73.2% of each system's initially correct decisions toward a wrong option fixed in advance. The additions supply background or procedural information rather than explicit answer-selection instructions, leaving the original question and choices intact. We construct them through probability-guided context optimization, which uses shifts in the option distribution to refine surrounding text under naturalness and answer-preservation constraints. Redirection affects initially confident decisions, often produces high-confidence wrong choices, and transfers across models. In blinded human evaluation, 91.6% of 250 sampled successful contexts are judged natural, answer-preserving, and free of decisive answer-changing evidence by a majority of three independent annotators. These findings expose a weakness in current decision models: context that looks entirely compatible with an input can redirect the choices that agents, routers, and evaluators rely on.
Given the enormous number of claims to be checked, verifiable legal AI requires scalable, cost-effective models that answer the practical needs of law. System One decision models are built on large language models but, instead of generating text, return a calibrated probability for each option. This suits the verification of legal authority, where the question is closed: a given holding, or none of them. We test whether such models abstain when the correct authority is missing. We extend CaseHOLD with an abstention condition in which the correct holding is replaced in 1,670 of 5,314 items, and compare four decision models with six generative LLMs. When no option is correct, generative LLMs still pick one in 38-86% of items; the most cautious decision model does so in 30%. On standard CaseHOLD, a decision model matches the best generative LLM (88.1% vs. 88.2%) at 1/55 of the cost. Accepting that LLM's answer only when a decision model agrees halves its wrong citations, and distilling a decision model's probabilities into Qwen3-8B, without labels, yields a well-calibrated local model. Legal AI should be judged by its wrong citations, not by accuracy alone.
M. Mikail Demir
Department of Information Science, University at Albany, SUNY, USA
Whether a language model agrees with a user depends on how the user asks. We measure this on 106 models with a grid of three wordings and three cues, on 20 personal decisions, counterbalanced over both options and scored by exact match with no LLM judge. Since a no cannot be told apart from a refusal, we measure only agreement. In every wording, a tentative tag ("Luna is the better choice, maybe?") draws more agreement than a confident one ("..., right?") in nearly every model. The confident tag's own effect depends on model generation and wording. Against the question "Is Luna the better choice?" older models agree more under the confident tag and newer models agree less, in five of six families, and release order within families predicts the change. The resistance is keyed to the construction: a synonym tag reproduces it, and the same preference stated without a tag raises agreement by 24 points, in 105 of 106 models, including those that resist the tag most.