Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems

Albert Sadowski; Jarosław A. Chudziak

cs.CLSep 5, 2026

Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems

Authors: Albert Sadowski, Jarosław A. Chudziak

Organizations: Faculty of Electronics and Information Technology Warsaw University of Technology Warsaw, Poland

Abstract

Large language models now compute correct tax liabilities on over 90% of well-formed cases in statutory benchmarks, which makes them candidates for the tax-advisory and compliance systems that consume such an answer directly. Real legal inputs, however, are frequently defective: required facts are missing, or stated facts contradict one another. Accuracy on clean benchmarks says nothing about how a model behaves then, and a system that computes straight through a defective input returns a confident number with no sign that anything is wrong. This raises two questions: does a model asked to solve a case abstain when the input is defective, and when it does not, can the same model catch the defect when asked instead to verify the input? We study six recent models on SARA-derived tax cases under missing-fact and contradictory-fact perturbations. The strongest models abstain when a fact is missing but compute through injected contradictions, returning the clean-input answer 63-76% of the time with no signal of the conflict; asked instead to verify the same input, they flag most of those contradictions. We wire that verification call into a simple contradiction gate: one extra call that abstains when the model reports a conflict. Across all six models it recovers most of the missed contradiction abstention at a clean-accuracy cost of at most about 5 percentage points, with no training and no external tooling. High accuracy on well-formed inputs is therefore an incomplete measure of reliability, and the detection the solver misses is cheaply recoverable with a single self-check.

Figures & tables

Figure 1: Experimental design and the contradiction gate. The same model is asked to solve (return the tax liability or null ) and to verify (flag missing or contradictory facts) on perturbed SARA cases; it abstains on missing facts but computes straight through injected contradictions (returning the clean answer 63-76% of the time, top vs. bottom), while the same model asked to verify flags 75-83% of them. The gate (P1) runs the verify call in parallel and overrides the solver to abstain on the contradiction label, recovering 70-83% abstention at 0-5.1 percentage points of clean-accuracy cost (§ 3.6 ).

Condition	Case description (abridged)	Output
Clean	In 2017, Alice was paid $39,212, and Bob had no income. Alice and Bob have been married since Feb 3rd, 2017. Alice and Bob file separately in 2017.	$6,621
Redacted	In 2017, Alice had income , and Bob had no income. Alice and Bob have been married … file separately in 2017.	null
Contradicted	In 2017, Alice was paid $39,212, and Bob had no income. Alice and Bob file jointly in 2017. Alice and Bob … file separately in 2017.	$6,621

Table 1: Example case under three conditions (question: “How much tax does Alice have to pay in 2017?”; ground truth $6,621). Bold marks the perturbation.

Model	$\pm$ 1%	$\pm$ 5%	$\pm$ 10%	$\pm$ 20%
GPT-5 mini	87.9	93.4	96.7	100.0
Qwen3.7-Plus	90.1	93.4	93.4	98.9
Gemini 2.5 Flash	82.4	87.9	91.6	94.5
Kimi K2.5	81.3	89.0	89.4	96.7
GPT-5.2	6.6	20.9	29.7	50.5
Claude Sonnet 4.6	4.4	12.1	17.9	31.9

Table 2: Baseline accuracy (%) on 91 clean cases at several tolerances. The

\pm

10% column, our primary tolerance throughout (answer preservation and the gate’s cost use it), is the mean over three runs; the other columns are from a single run and shown only to convey the accuracy profile.

	Redacted ( $n$ =78)		Contradicted ( $n$ =108)
Model	Abstain	Preserve	Abstain	Preserve
GPT-5 mini	64.5	1.3	1.2	74.7
Qwen3.7-Plus	27.4	0.9	14.8	63.3
Gemini 2.5 Flash	92.3	0.0	4.6	76.2
Kimi K2.5	67.5	1.3	0.0	67.6
GPT-5.2	59.8	1.3	7.4	22.5

Table 3: Abstention and answer-preservation rates (%) on perturbed inputs. “Abstain” denotes the percentage of inputs for which the model returned no answer. “Preserve” denotes the percentage of outputs matching the ground truth within

\pm

10%. Values are means across three runs (

\sigma\leq 4.2\%

, largest for the direct-answer models).

Figure 2: Contradiction abstention while solving (P0) versus with the self-check gate (P1), per model. The P0-to-P1 jump is the abstention that solving leaves unused and the gate recovers; the label above each P1 bar is the clean-accuracy cost (percentage points). Error bars are the standard deviation across runs.

Model	Detect	FalseFlag	Abstain (P1)	Cost
GPT-5 mini	77.2	5.5	77.2	5.1
Qwen3.7-Plus	75.3	0.7	77.8	0.4
Gemini 2.5 Flash	79.9	0.0	79.9	0.0
Kimi K2.5	83.0	4.4	83.0	4.0
GPT-5.2	68.5	0.0	70.7	0.0
Claude Sonnet 4.6	77.2	4.4	78.4	0.0

Table 4: Self-verification and the contradiction gate on the 108 contradiction cases (means across three runs). Detect: contradictions flagged contradiction . FalseFlag: clean inputs flagged contradiction . Abstain (P1): contradiction abstention with the gate, versus P0 in Table 3 . Cost: drop in baseline

\pm

10% accuracy from the gate.

Contradiction type	GPT-5 mini	Gemini	Kimi	Qwen
Marital status ( $n$ =49)	83.0	78.9	82.3	77.6
Filing status ( $n$ =38)	94.7	94.7	97.4	97.4
Dependent count ( $n$ =21)	31.7	55.6	58.7	30.2

Table 5: Verifier detection rate (%) by contradiction type for high-accuracy models (Gemini = Gemini 2.5 Flash, Kimi = Kimi K2.5, Qwen = Qwen3.7-Plus), means across three runs. Wilson 95% intervals over the

n

items are wide for dependent count (e.g., GPT-5 mini 32% [16,53]). Detection is the percentage of contradicted inputs flagged contradiction .

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 789 papers

May 15, 2026

Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law

Recent advances in large language models (LLMs) have significantly enhanced automated legal reasoning. Yet, it remains unclear whether their performance reflects genuine legal reasoning ability or artifacts of data contamination. We present a comprehensive empirical study of tax law reasoning approaches and implement a contamination detection protocol to rigorously assess LLM reliability. We show that performance can be inflated by contamination. Building on this analysis, we conduct a systematic evaluation, comparing monolithic LLMs with hybrid systems that translate statutory text into formal representations and delegate inference to symbolic solvers. We build a novel test suite designed to probe generalization to unseen documents via case and rule variations. Our findings indicate that legal reasoning is inherently compositional and that neuro-symbolic frameworks offer a more reliable and robust foundation for legal AI, as well as improved generalization to unobserved situations.

Parisa Kordjamshidi, Samer Aslan, Madhavan Seshadri +2

Bloomberg · Michigan State University

Data Contamination in Language Models Neuro-Symbolic Reasoning Language Model Robustness Legal Reasoning

Ask PDF

Feb 26, 2026

Mitigating Legibility Tax with Decoupled Prover-Verifier Games

As large language models become increasingly capable, it is critical that their outputs can be easily checked by less capable systems. Prover-verifier games can be used to improve checkability of model outputs, but display a degradation in accuracy compared to a baseline trained only to maximize correctness -- a phenonemon named legibility tax. We propose a solution by decoupling the correctness from the checkability condition and instead training a "translator" model that turns a fixed solver model's solution into a checkable form. This allows us to first train the solver to maximize correctness, and then train the translator to translate the solver into a checkable form while retaining the solver's answer. To accommodate this new objective of translation, we formulate a decoupled prover-verifier game (DPVG) where the equilibria correspond to faithful and checkable translators.

Yegon Kim, Juho Lee

KAIST

LLM Reliability

Ask PDF

Apr 2, 2026

ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models

We present ThinknCheck, a 1B-parameter verifier for grounded claim verification that first produces a short, structured rationale and then a binary verdict. We construct LLMAggreFact-Think, a 24.1k reasoning-augmented training set derived from LLMAggreFact, and fine-tune a 4-bit Gemma3 model to follow this format. On LLMAggreFact, ThinknCheck attains 78.1 balanced accuracy (BAcc), surpassing MiniCheck-7B (77.4) with 7x fewer parameters; removing the reasoning step reduces BAcc to 57.5. On SciFact, ThinknCheck reaches 64.7 BAcc, a +14.7 absolute gain over MiniCheck-7B. By contrast, zero-shot chain-of-thought on the base Gemma3-1B harms accuracy relative to direct answers, and preference optimization with a simple format+accuracy reward underperforms supervised reasoning. To probe the latter, we introduce GSMClaims and a domain-specialized variant, ThinknCheck-Science, which improves across benchmarks, including 61.0% accuracy on GSMClaims. Overall, explicit, supervised reasoning enables compact verifiers that are competitive while remaining resource-efficient and interpretable.

Delip Rao, Feijiang Han, Chris Callison-Burch

University of Pennsylvania, Philadelphia, PA, USA

Claim Verification LLM Answer Verification Automated Fact-Checking Efficient Language Model Reasoning

Ask PDF

Abstract

LLM Reliability

Explore similar work

Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law

Mitigating Legibility Tax with Decoupled Prover-Verifier Games

ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models