ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound
Organizations: ThakiCloud · Korea University
Abstract
Evidence-sufficiency benchmarks ask whether a model can tell a complete passage set from one whose reasoning chain is broken. When broken sets are made by editing text, near-perfect sufficiency scores--and even positive crossed-control scores--need not demonstrate multi-hop reasoning. We present ChainCheck, an evaluation-validity audit that pairs validator-certified counterfactual cells with artifact-only baselines. In an exploratory crossed control on 2,214 HotpotQA pairs, a language-model scorer that never reads the chain reaches a nominal AUC of 0.997, and a one-string match reaches the maximum of the crossed chain-sensitivity statistic CS4 (0.500), while the 11 evaluated systems score from 0.308 to 0.484; the two statistics are different metrics. We therefore read positive CS4 as operational bridge-consistency sensitivity, not as semantic multi-hop reasoning. Every system is also sensitive to one extra edit, and the edits are detectable according to model-based naturalness ratings. On an independent HotpotQA set of 2,276 pairs, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate: 5 of 6 trained readouts were edit-dominated, and the reference judge's selectivity interval [-0.007, +0.042] includes zero. Judge scores also depended on batch layout. The MuSiQue and 2Wiki records are public; the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial.
Figures & tables
| Construction of the negative (MuSiQue test) | Surface-only paired AUC | Role |
|---|---|---|
| ABSENT, all support removed | 0.959 | diagnostic |
| PARTIAL, hop removal | 0.922 | diagnostic |
| PARTIAL, bridge substitution | 0.568 | primary |
| System | Order jitter ratio | Hop-removal transfer | ABSENT transfer |
|---|---|---|---|
| M0 single layer | 0.456 | 0.794 | 0.909 |
| M0-last | 0.534 | 0.888 | 0.912 |
| M1 layer mix | 0.421 | 0.964 | 0.985 |
| M2 set pool | 0.148 | 0.782 | 0.916 |
| M3 set attention | 0.150 | 0.768 | 0.911 |
| M4-R readout+LoRA | 0.076 | 0.862 | 0.951 |
| Control | Dataset | Slice | Records | Clusters | (largest share, %) |
|---|---|---|---|---|---|
| Edit-matched | MuSiQue | real | 122 | 52 | 20.9 (16.4) |
| synthetic | 98 | 40 | 14.64 (20.4) | ||
| joint | 220 | 52 | 18.22 (18.2) | ||
| 2Wiki | real | 287 | 28 | 1.7 (76.3) | |
| synthetic | 141 | 19 | 1.63 (78.0) | ||
| joint | 428 | 28 | 1.68 (76.9) |
| Analysis | Records | Finding | Evidence grade |
|---|---|---|---|
| HotpotQA independent confirmation | 2,276 pairs | H1-type and H2-type patterns not replicated | confirmatory |
| Crossed control, arm E | 2,214 pairs | all 11 systems bridge-consistency sensitive (CS4); all edit-count sensitive | exploratory |
| Crossed control, holdouts C1/C2 | 50 / 231 pairs | CS4 direction agrees for every system | indicative |
| Batch-layout diagnostic | 8,856 records | layout dependence confirmed; gate passes after recovery | methodological |
| System | Nominal | CE | EE | [95% CI] | [95% CI] |
|---|---|---|---|---|---|
| M0 single layer | 0.995 | +0.192 | +0.432 | 0.240 [ 0.262, 0.216] | 0.019 [ 0.035, 0.003] |
| M0-last | 0.975 | +0.312 | +0.324 | 0.012 [ 0.036, +0.013] | +0.132 [+0.114, +0.150] |
| M1 layer mix | 0.996 | +0.355 | +0.358 | 0.004 [ 0.021, +0.022] | +0.125 [+0.116, +0.145] |
| M2 set pool | 0.996 | +0.347 | +0.456 | 0.108 [ 0.130, 0.097] | 0.002 [ 0.012, +0.009] |
| M3 set attention | 0.994 | +0.365 | +0.436 | 0.071 [ 0.090, 0.059] | +0.022 [+0.009, +0.031] |
| M4-R readout+LoRA | 0.997 | +0.418 | +0.470 | 0.052 [ 0.066, 0.038] | +0.011 [+0.002, +0.020] |
| Arm E (exploratory) | C1 | C2 | |||
| System | CS4 [adj. CI] | EC [adj. CI] | PT [adj. CI] | CS4 | CS4 |
| M0 single layer | 0.407 [0.396, 0.417] | +0.374 [+0.354, +0.394] | +0.171 [+0.142, +0.198] | 0.380 | 0.451 |
| M1 layer mix | 0.481 [0.476, 0.486] | +0.352 [+0.330, +0.371] | 0.057 [ 0.087, 0.031] | 0.485 | 0.486 |
| M2 set pool | 0.454 [0.445, 0.461] | +0.300 [+0.286, +0.329] | +0.189 [+0.164, +0.215] | 0.440 | 0.464 |
| M3 set attention | 0.450 [0.443, 0.459] | +0.240 [+0.214, +0.260] | +0.133 [+0.114, +0.166] | 0.450 | 0.462 |
| M4-R readout+LoRA | 0.484 [0.478, 0.489] | +0.327 [+0.303, +0.349] | 0.006 [ 0.036, +0.024] | 0.485 | 0.474 |
| Asset | Repository (Hugging Face) | Revision |
|---|---|---|
| Dataset and evaluator | datasets/ThakiCloud/ChainCheck | 825d836d1b |
| Released judges | ThakiCloud/ChainCheck-Judge-* | not used in this paper |
| Backbone, reference judge | Qwen/Qwen3.8-27B | dbdc473dea |
| Additional judge | meta-llama/Llama-3.1-8B-Instruct | 8c22764a7e |
| Additional judge | microsoft/phi-4 | 2db69c1c3e |
| Additional judge | Qwen/Qwen3-32B | a09d5ccb54 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| System | Nominal | Chain effect | Edit effect | seed median [seed-avg. boot.] |
|---|---|---|---|---|
| MuSiQue test (exploratory, post hoc) , | ||||
| M0 single layer | 0.968 | 0.025 | +0.438 | 0.463 [ 0.528, 0.395] |
| M0-last | 0.912 | +0.120 | +0.265 | 0.145 [ 0.225, 0.065] |
| M1 layer mix | 0.971 | +0.157 | +0.312 | 0.154 [ 0.226, 0.074] |
| M2 set pool | 0.989 | +0.176 | +0.441 | 0.265 [ 0.332, 0.221] |
| M3 set attention | 0.992 | +0.228 | +0.420 | 0.191 [ 0.258, 0.145] |
| System | Nominal | Chain effect | Edit effect | seed median [seed-avg. boot.] |
|---|---|---|---|---|
| MuSiQue test (exploratory, post hoc) , | ||||
| M0 single layer | 0.991 | 0.062 | +0.464 | 0.527 [ 0.603, 0.451] |
| M0-last | 0.942 | +0.183 | +0.317 | 0.134 [ 0.228, 0.045] |
| M1 layer mix | 0.978 | +0.268 | +0.299 | 0.031 [ 0.125, 0.039] |
| M2 set pool | 0.991 | +0.290 | +0.451 | 0.161 [ 0.232, 0.119] |
| M3 set attention | 0.987 | +0.317 | +0.442 | 0.125 [ 0.176, 0.065] |
| System | Nominal | Chain effect | Edit effect | seed median [seed-avg. boot.] |
|---|---|---|---|---|
| MuSiQue confirmation (pre-specified) , | ||||
| Qwen3.8-27B (reference) | 0.943 | +0.328 | +0.189 | +0.139 [0.025, 0.262] |
| Llama-3.1-8B-Instruct | 0.779 | +0.147 | +0.082 | +0.066 [ 0.082, 0.197] |
| phi-4 | 0.877 | +0.049 | +0.254 | 0.205 [ 0.336, 0.066] |
| Qwen3-32B | 0.820 | +0.205 | +0.115 | +0.090 [ 0.049, 0.238] |
| Gemma-4-31B-it | 0.873 | +0.414 | 0.033 | +0.381 [0.299, 0.434] |
| System | Nominal | Chain effect | Edit effect | seed median [seed-avg. boot.] |
|---|---|---|---|---|
| MuSiQue confirmation (pre-specified) , | ||||
| Qwen3.8-27B (reference) | 0.929 | +0.459 | 0.031 | +0.429 [0.337, 0.469] |
| Llama-3.1-8B-Instruct | 0.725 | +0.214 | 0.031 | +0.184 [0.061, 0.255] |
| phi-4 | 0.775 | +0.122 | +0.122 | +0.000 [ 0.163, 0.153] |
| Qwen3-32B | 0.775 | +0.347 | 0.133 | +0.214 [0.122, 0.306] |
| Gemma-4-31B-it | 0.862 | +0.403 | 0.143 | +0.260 [0.184, 0.342] |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,527 papers