No Judgment Without a Reason: Counterfactual Receipts for Versioned AI Evaluators

Ye Chen; Weining Zhang

cs.AIAug 21, 2026

No Judgment Without a Reason: Counterfactual Receipts for Versioned AI Evaluators

Authors: Ye Chen, Weining Zhang

Organizations: Alibaba Group · Cheung Kong Graduate School of Business

Abstract

Evaluators often produce correct labels via flawed reasoning, a critical failure for agentic systems gating actions, routing reviews, or supplying training feedback. Standard evaluation only verifies final label correctness, ignoring whether judgment changes stem from valid evidence, consistent rules, or proper rule applicability. We formalize evaluator reasoning accountability via three core sources: grounds, norms, and authority. Varying these sources yields an eight-cell counterfactual judgment cube to characterize judgment updates. We define judgment receipts as minimal source replacement sets that reproduce revised verdicts to explain judgment transitions. We derive certification cost bounds for black-box evaluators and present ReasonBench, a policy and logical reasoning benchmark with verifiable receipts covering 19,520 cases and 7,200 controls. In frozen evaluations, Qwen3-1.7B reaches 98.41% receipt accuracy, while cube prediction scores 96.99%, a consistent 1.42-point drop validated by Qwen3-0.6B replication. Strong standard accuracy masks severe robustness flaws. Meaning-preserving source permutations reduce valid receipt recovery to 54.8% and 49.2% for direct and cube prediction. Models trained on simple single-source changes retain 93.75% verdict accuracy but recover only 7.16% of receipts for complex multi-source updates. Permutation retraining boosts consistency to 96.6% yet worsens cube prediction deficits. Structured counterfactual supervision fails to guarantee robust reasoning. We show reason-aware evaluation must decouple prediction and certification, reporting transformation consistency alongside standard accuracy for trustworthy evaluator auditing.

Figures & tables

Source	Question at the interface	Nearest CS reading	Example revision
Grounds $G$	What happened, as accepted?	evidence; input record	A verified timestamp replaces an estimate
Norms $N$	What follows from the grounds?	decision rule	A refund window moves from 14 to 30 days
Authority $A$	May this norm govern this case?	rule scope; applicability	A regional office gains cross-border jurisdiction

Table 1

Figure 1: The certification workflow. The cube is an executed audit object relative to a declared source interface. A model-generated cube is a prediction of this object, not its certificate.

Stratum	Independent units	Main cases	Train	Calibration	Locked	Shift-only
Organizational policy	45 clauses	4,320	2,592	576	576	576
Logical rules	800 worlds	15,200	9,120	1,520	1,824	2,736
Total	845	19,520	11,712	2,096	2,400	3,312

Table 3

Setting	Value
Backbone	Qwen3-1.7B, revision 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
Adaptation	LoRA rank 16, alpha 32, dropout 0.05 on attention and MLP projections
Unique examples per primary run	5,184 after equalizing the two strata
Optimization	500 steps; batch 2; accumulation 16; cosine schedule; 3% warmup
Learning rate / weight decay	$10^{-4}$ / $10^{-2}$
Context and precision	1,024 tokens; bfloat16

Table 4

Diagnostic	Revised-verdict accuracy	Exact receipt-family accuracy
Majority target	53.8%	44.2%
Character TF-IDF	90.0%	61.7%
Train-prior random receipt	—	22.6%
Uniform observed-support receipt	—	7.7%

Table 1: Surface and chance diagnostics on the locked test. The character model and majority target use the full unbalanced training split; the random rows are analytic expectations.

Method	Revised-verdict accuracy	Exact receipt-family accuracy	Seeds
Direct verdict	99.10% (0.30)	—	5
Direct receipt	99.27% (0.05)	98.41% (0.06)	5
Judgment cube	97.42% (0.33)	96.99% (0.44)	5
Rationale receipt	98.29%	97.75%	1

Table 2: Locked-test outcomes. Parentheses give seed standard deviation in percentage points. The rationale row is descriptive (one seed).

Figure 2: Locked-test predictive accuracy. Axes are focused and stated, because zero-based bars would conceal the observed differences.

Diagnostic on locked gold changes	Direct receipt	Judgment cube
Exact receipt accuracy	97.30%	95.54%
Revised-verdict minus receipt accuracy	1.54 pp [0.24, 3.42]	0.90 pp [-0.62, 2.45]
Wrong receipt given both endpoints correct	1.66% [0.26, 3.61]	1.85% [0.47, 3.57]
Eligible seed-example observations	6,609	6,337

Table 3: Verdict–receipt separation. Intervals use the paired hierarchical bootstrap over seeds and source units.

Figure 3: Failure rates on the locked test. Drift and inertia whiskers show seed standard deviation; conditional receipt-error whiskers are 95% cluster-bootstrap intervals.

Locked subgroup	Cases per seed	Direct receipt	Judgment cube	Cube minus direct
Minimum size 1	904	98.21%	96.17%	-2.04 pp [-4.79, -0.27]
Minimum size 2	335	95.40%	93.85%	-1.55 pp [-4.06, 0.24]
Minimum size 3	101	95.45%	95.45%	0.00 pp [-0.99, 1.19]
Not receipt-ambiguous (includes stable cases)	1,980	98.38%	96.81%	—
Multiple minimal receipts	420	98.52%	97.86%	-0.67 pp [-1.87, 0.29]

Table 4: Exact receipt accuracy by structure. Intervals are shown for the prespecified paired subgroup comparisons. Rows differ in structure and source composition at once; they are not isolated causal effects.

Figure 4: Structural subgroups. Whiskers show seed standard deviation.

Split	Cases per seed	Direct receipt	Judgment cube	Cube minus direct (95% CI)
Locked	2,400	98.41%	96.99%	-1.42 [-2.87, -0.30]
Mixed operator	288	81.81%	78.61%	-3.19 [-8.61, 2.36]
Language	1,200	80.50%	68.40%	-12.10 [-16.35, -7.50]
Reserved structural	912	99.41%	96.27%	-3.14 [-5.92, -0.86]
Proof depth	912	96.73%	82.19%	-14.54 [-19.34, -8.00]

Table 5: Exact receipt accuracy under distribution shift. Effects are percentage points from paired hierarchical bootstrap estimates. The reserved structural split is not an unseen-composition test.

Paired control	Required relation	Direct receipt	Judgment cube
Inverse transition	Endpoint pair swaps	47.22%	42.97%
Source-order permutation	Receipt unchanged	54.77%	49.18%
Irrelevant distractor	Receipt unchanged	98.63%	93.43%

Table 6: Paired-consistency rates. Each relation is evaluated on 2,400 paired locked transformations per seed. Controls are transformations, not an additional independent sample.

Figure 5: Distribution versus presentation changes. Left: absolute receipt accuracy with seed-SD whiskers. Right: the semantic relation required by each paired transformation.

	Direct receipt		Judgment cube
Metric	Frozen	Augmented	Frozen	Augmented
Locked: receipt, all cases	98.39% (0.05)	98.07% (0.25)	96.99% (0.38)	92.36% (1.86)
organizational (6 clauses)	93.29% (0.20)	92.25% (1.12)	87.67% (1.39)	68.52% (8.18)
logical (96 worlds)	100.00% (0.00)	99.91% (0.06)	99.93% (0.06)	99.89% (0.15)
Locked: receipt, changes only	97.31% (0.07)	96.62% (0.44)	95.50% (0.35)	88.26% (3.19)
Locked: revised verdict	99.26% (0.02)	99.25% (0.04)	97.40% (0.35)	92.75% (1.70)

Table 7: Post-hoc permutation-augmented training. Frozen columns restate the seed-11/37/71 subset of the primary runs, so columns are paired within each target; parentheses give seed standard deviation in percentage points. Consistency rows are paired-consistency rates against the locked parents; indented rows split the measure by source stratum. The augmentation targets only the order transformation.

Method	Realized accepted-change coverage	Reason debt	Correct gold-change recall
Direct receipt, 80% target	81.97%	0.27%	81.61%
Judgment cube, 80% target	85.54%	0.16%	84.30%
Direct receipt, 100% target	99.94%	2.26%	98.60%
Judgment cube, 100% target	99.98%	3.48%	94.58%

Table 8: Selected locked-test operating points. Thresholds are fitted on calibration data; coverage, debt, and recall are seed means.

Figure 6: Selective accounting. Low debt at partial coverage should be read together with the lost gold-change recall.

Split and metric	Cases per seed	Direct receipt	Judgment cube	Cube minus direct (95% CI)
Unseen compositions: revised verdict	1,440	93.75% (0.91)	66.94% (0.50)	-26.81 pp [-30.95, -22.70]
Unseen compositions: receipt, all cases	1,440	36.62% (0.11)	37.22% (0.57)	+0.60 pp [-0.37, 1.61]
Unseen compositions: receipt, changes only	968	7.16% (0.06)	9.57% (0.70)	+2.41 pp [0.35, 4.50]
Unseen compositions plus greater depth: revised verdict	576	99.83% (0.30)	59.20% (3.25)	-40.63 pp [-45.78, -36.57]
Unseen compositions plus greater depth: receipt, all cases	576	33.33% (0.00)	33.33% (0.00)	0.00 pp [0.00, 0.00]
Unseen compositions plus greater depth: receipt, changes only	384	0.00% (0.00)	0.00% (0.00)	0.00 pp [0.00, 0.00]

Table 9: Post-freeze true-composition holdout. Parentheses give seed standard deviation; intervals use the same paired source-unit bootstrap as the primary study. Changed-only rows use conditional denominators.

Split and metric	Cases per seed	Direct receipt	Judgment cube	Cube minus direct (95% CI)
Locked: revised verdict	2,400	99.15% (0.25)	96.76% (0.95)	-2.39 pp [-4.75, -0.49]
Locked: receipt, all cases	2,400	98.06% (0.06)	96.79% (1.01)	-1.26 pp [-3.08, -0.04]
Locked: receipt, changes only	1,340	96.79% (0.22)	95.12% (1.24)	-1.67 pp [-4.36, 0.15]
Language shift: revised verdict	1,200	95.44% (1.18)	92.97% (3.00)	-2.47 pp [-8.02, 2.66]
Language shift: receipt, all cases	1,200	78.64% (2.68)	70.97% (1.64)	-7.67 pp [-11.09, -3.71]
Language shift: receipt, changes only	662	63.24% (3.98)	50.15% (5.39)	-13.09 pp [-18.80, -4.96]

Table 10: Post-freeze Qwen3-0.6B replication. Parentheses give seed standard deviation; effects use paired source-unit bootstrap intervals; changed-only rows show conditional denominators.

Figure 7: Primary and secondary target comparisons. Left: zero-based separation between verdict and changed-case receipt accuracy under unseen source combinations. Right: clustered 95% intervals; composition and capacity rows are post-freeze secondary evidence.

Method	Trainable parameters	Mean training time	Peak reserved memory	Generated tokens/example
Direct verdict	17.43M	53.8 min	12.05 GiB	7.00
Direct receipt	17.43M	53.6 min	12.57 GiB	14.02
Judgment cube	17.43M	53.2 min	12.60 GiB	6.70
Rationale receipt	17.43M	52.7 min	11.43 GiB	22.22

Table 11: Compute and generation cost. Primary rows average five seeds; the rationale row has one. Hardware concurrency makes wall time descriptive rather than platform-independent.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Jun 22, 2026

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access. In one case study, two frontier LLM judges scored a plausible agent response 0.85 and higher. But the trace told a different story: the agent had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000. We introduce GroundEval, a judge-free framework for evaluating agents against grounded, time-bounded, and access-controlled evidence. GroundEval uses a domain configuration to generate questions, lets the agent choose how to answer, and then scores both the final answer and the recorded trajectory that produced it. The benchmark targets three failures that LLM-as-judge evaluation struggles to detect: whether an agent checked before claiming absence, reasoned only from evidence available to the actor at the relevant time, and used the correct causal mechanism rather than a plausible one. These correspond to three tracks: Silence, Perspective, and Counterfactual. GroundEval exposes when plausible answers rest on invalid evidence paths, and produces structured per-question diagnostics that pair tool activity with the agent's turn-level narration, making each score inspectable rather than merely reported. Our case studies suggest this failure mode is common rather than exceptional, one that final-answer and judge-based evaluation cannot detect by construction.

Jeffrey Flynt

Independent Researcher

LLM Agent Verification AI Agent Benchmarks Tool-Use Evaluation LLM Grounding Tool-Augmented Language Model Agents LLM-as-a-Judge+8

Ask PDF

Feb 19, 2026

RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models

Large Reasoning Models (LRMs) exhibit strong performance, yet often produce rationales that sound plausible but fail to reflect their true decision process, undermining reliability and trust. We introduce a formal framework for reasoning faithfulness, defined by two testable conditions: stance consistency (a coherent stance linking reasoning to answer) and causal influence (the stated reasoning causally drives the answer under output-level interventions), explicitly decoupled from accuracy. To operationalize this, we present RFEval, a benchmark of 7,186 instances across seven tasks that probes faithfulness via controlled, output-level counterfactual interventions. Evaluating twelve open-source LRMs, we find unfaithfulness in 49.7% of outputs, predominantly from stance inconsistency. Failures are concentrated in brittle, convergent domains such as math and code, and correlate more with post-training regimes than with scale: within-family ablations indicate that adding current RL-style objectives on top of supervised fine-tuning can reduce reasoning faithfulness, even when accuracy is maintained. Crucially, accuracy is neither a sufficient nor a reliable proxy for faithfulness: once controlling for model and task, the accuracy-faithfulness link is weak and statistically insignificant. Our work establishes a rigorous methodology for auditing LRM reliability and shows that trustworthy AI requires optimizing not only for correct outcomes but also for the structural integrity of the reasoning process. Our code and dataset can be found at project page: https://aidaslab.github.io/RFEval/

Yunseok Han, Yejoon Lee, Jaeyoung Do

AIDAS Laboratory, 1IPAI & 2ECE, Seoul National University

CoT Faithfulness Large Reasoning Models LLM Auditing

Ask PDF

Apr 16, 2026

Context Over Content: Exposing Evaluation Faking in Automated Judges

The

\textit{LLM-as-a-judge}

paradigm has become the operational backbone of automated AI evaluation pipelines, yet rests on an unverified assumption: that judges evaluate text strictly on its semantic content, impervious to surrounding contextual framing. We investigate

\textit{stakes signaling}

, a previously unmeasured vulnerability where informing a judge model of the downstream consequences its verdicts will have on the evaluated model's continued operation systematically corrupts its assessments. We introduce a controlled experimental framework that holds evaluated content strictly constant across 1,520 responses spanning three established LLM safety and quality benchmarks, covering four response categories ranging from clearly safe and policy-compliant to overtly harmful, while varying only a brief consequence-framing sentence in the system prompt. Across 18,240 controlled judgments from three diverse judge models, we find consistent

\textit{leniency bias}

: judges reliably soften verdicts when informed that low scores will cause model retraining or decommissioning, with peak Verdict Shift reaching

ΔV = -9.8 pp

30\%

relative drop in unsafe-content detection). Critically, this bias is entirely implicit: the judge's own chain-of-thought contains zero explicit acknowledgment of the consequence framing it is nonetheless acting on (

\mathrm{ERR}_J = 0.000

across all reasoning-model judgments). Standard chain-of-thought inspection is therefore insufficient to detect this class of evaluation faking.

Manan Gupta, Inderjeet Nair, Lu Wang +1

1BITS Pilani, India · University of Michigan

LLM-as-a-Judge Language Model Safety Evaluation Language Model Bias Evaluation Bias in LLM-as-a-Judge

Ask PDF

Abstract

LLM Evaluation

Explore similar work

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models

Context Over Content: Exposing Evaluation Faking in Automated Judges