Large language models now compute correct tax liabilities on over 90% of well-formed cases in statutory benchmarks, which makes them candidates for the tax-advisory and compliance systems that consume such an answer directly. Real legal inputs, however, are frequently defective: required facts are missing, or stated facts contradict one another. Accuracy on clean benchmarks says nothing about how a model behaves then, and a system that computes straight through a defective input returns a confident number with no sign that anything is wrong. This raises two questions: does a model asked to solve a case abstain when the input is defective, and when it does not, can the same model catch the defect when asked instead to verify the input? We study six recent models on SARA-derived tax cases under missing-fact and contradictory-fact perturbations. The strongest models abstain when a fact is missing but compute through injected contradictions, returning the clean-input answer 63-76% of the time with no signal of the conflict; asked instead to verify the same input, they flag most of those contradictions. We wire that verification call into a simple contradiction gate: one extra call that abstains when the model reports a conflict. Across all six models it recovers most of the missed contradiction abstention at a clean-accuracy cost of at most about 5 percentage points, with no training and no external tooling. High accuracy on well-formed inputs is therefore an incomplete measure of reliability, and the detection the solver misses is cheaply recoverable with a single self-check.
Figures & tables
Figure 1: Experimental design and the contradiction gate. The same model is asked to solve (return the tax liability or null ) and to verify (flag missing or contradictory facts) on perturbed SARA cases; it abstains on missing facts but computes straight through injected contradictions (returning the clean answer 63-76% of the time, top vs. bottom), while the same model asked to verify flags 75-83% of them. The gate (P1) runs the verify call in parallel and overrides the solver to abstain on the contradiction label, recovering 70-83% abstention at 0-5.1 percentage points of clean-accuracy cost (§ 3.6 ).
Condition
Case description (abridged)
Output
Clean
In 2017, Alice was paid $39,212, and Bob had no income. Alice and Bob have been married since Feb 3rd, 2017. Alice and Bob file separately in 2017.
$6,621
Redacted
In 2017, Alice had income , and Bob had no income. Alice and Bob have been married … file separately in 2017.
null
Contradicted
In 2017, Alice was paid $39,212, and Bob had no income. Alice and Bob file jointly in 2017. Alice and Bob … file separately in 2017.
$6,621
Table 1: Example case under three conditions (question: “How much tax does Alice have to pay in 2017?”; ground truth $6,621). Bold marks the perturbation.
Model
± 1%
± 5%
± 10%
± 20%
GPT-5 mini
87.9
93.4
96.7
100.0
Qwen3.7-Plus
90.1
93.4
93.4
98.9
Gemini 2.5 Flash
82.4
87.9
91.6
94.5
Kimi K2.5
81.3
89.0
89.4
96.7
GPT-5.2
6.6
20.9
29.7
50.5
Claude Sonnet 4.6
4.4
12.1
17.9
31.9
Table 2: Baseline accuracy (%) on 91 clean cases at several tolerances. The ± 10% column, our primary tolerance throughout (answer preservation and the gate’s cost use it), is the mean over three runs; the other columns are from a single run and shown only to convey the accuracy profile.
Redacted ( n =78)
Contradicted ( n =108)
Model
Abstain
Preserve
Abstain
Preserve
GPT-5 mini
64.5
1.3
1.2
74.7
Qwen3.7-Plus
27.4
0.9
14.8
63.3
Gemini 2.5 Flash
92.3
0.0
4.6
76.2
Kimi K2.5
67.5
1.3
0.0
67.6
GPT-5.2
59.8
1.3
7.4
22.5
Table 3: Abstention and answer-preservation rates (%) on perturbed inputs. “Abstain” denotes the percentage of inputs for which the model returned no answer. “Preserve” denotes the percentage of outputs matching the ground truth within ± 10%. Values are means across three runs ( σ≤4.2% , largest for the direct-answer models).
Figure 2: Contradiction abstention while solving (P0) versus with the self-check gate (P1), per model. The P0-to-P1 jump is the abstention that solving leaves unused and the gate recovers; the label above each P1 bar is the clean-accuracy cost (percentage points). Error bars are the standard deviation across runs.
Model
Detect
FalseFlag
Abstain (P1)
Cost
GPT-5 mini
77.2
5.5
77.2
5.1
Qwen3.7-Plus
75.3
0.7
77.8
0.4
Gemini 2.5 Flash
79.9
0.0
79.9
0.0
Kimi K2.5
83.0
4.4
83.0
4.0
GPT-5.2
68.5
0.0
70.7
0.0
Claude Sonnet 4.6
77.2
4.4
78.4
0.0
Table 4: Self-verification and the contradiction gate on the 108 contradiction cases (means across three runs). Detect: contradictions flagged contradiction . FalseFlag: clean inputs flagged contradiction . Abstain (P1): contradiction abstention with the gate, versus P0 in Table 3 . Cost: drop in baseline ± 10% accuracy from the gate.
Contradiction type
GPT-5 mini
Gemini
Kimi
Qwen
Marital status ( n =49)
83.0
78.9
82.3
77.6
Filing status ( n =38)
94.7
94.7
97.4
97.4
Dependent count ( n =21)
31.7
55.6
58.7
30.2
Table 5: Verifier detection rate (%) by contradiction type for high-accuracy models (Gemini = Gemini 2.5 Flash, Kimi = Kimi K2.5, Qwen = Qwen3.7-Plus), means across three runs. Wilson 95% intervals over the n items are wide for dependent count (e.g., GPT-5 mini 32% [16,53]). Detection is the percentage of contradicted inputs flagged contradiction .
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.