ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound
Authors: Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun Jeon
Evidence-sufficiency benchmarks ask whether a model can tell a complete passage set from one whose reasoning chain is broken. When broken sets are made by editing text, near-perfect sufficiency scores--and even positive crossed-control scores--need not demonstrate multi-hop reasoning. We present ChainCheck, an evaluation-validity audit that pairs validator-certified counterfactual cells with artifact-only baselines. In an exploratory crossed control on 2,214 HotpotQA pairs, a language-model scorer that never reads the chain reaches a nominal AUC of 0.997, and a one-string match reaches the maximum of the crossed chain-sensitivity statistic CS4 (0.500), while the 11 evaluated systems score from 0.308 to 0.484; the two statistics are different metrics. We therefore read positive CS4 as operational bridge-consistency sensitivity, not as semantic multi-hop reasoning. Every system is also sensitive to one extra edit, and the edits are detectable according to model-based naturalness ratings. On an independent HotpotQA set of 2,276 pairs, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate: 5 of 6 trained readouts were edit-dominated, and the reference judge's selectivity interval [-0.007, +0.042] includes zero. Judge scores also depended on batch layout. The MuSiQue and 2Wiki records are public; the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial.
Figures & tables
Figure 1: Artifact-only baselines against the evaluated systems on the crossed HotpotQA arm E (2,214 pairs, exploratory). (a) Nominal pair AUC of the original contrast (A over D) for the 11 systems; the dashed line is AB2, a per-passage language-model log-probability scorer that never sees the chain (0.997). (b) CS4, the crossed chain-sensitivity statistic of Section 11.2 , with Bonferroni-adjusted document-group bootstrap intervals; the dashed line is AB3, which checks whether one bridge string equals one title and reaches the maximum 0.500. The panels use different metrics on separate axes and are not compared with each other. Values are in Table 6 .
Construction of the negative (MuSiQue test)
Surface-only paired AUC
Role
ABSENT, all support removed
0.959
diagnostic
PARTIAL, hop removal
0.922
diagnostic
PARTIAL, bridge substitution
0.568
primary
Table 1: A surface-only classifier (BM25, Jaccard, length, near-duplicate features) under group-aware cross-validation. Deletion-based negatives are separable from the surface alone; bridge substitution is below the pre-specified gate of 0.65.
Figure 2: Chain effect (open bars, AUC(B > D) −0.5 ) and edit effect (hatched bars, AUC(A > B) −0.5 ) per system, real bridge entities. Trained systems (M0 to M4) show large edit effects in every setting; their chain effects are smaller on MuSiQue and large on 2Wiki. The two contrasts vary separately in these data.
Figure 3: Chain selectivity with 95% pair-bootstrap intervals in all three settings, real bridge entities. Open diamonds mark the reference zero-shot judge. A system is chain-selective when its interval lies right of the dotted zero line. The synthetic-entity version is Figure 5 .
System
Order jitter ratio
Hop-removal transfer
ABSENT transfer
M0 single layer
0.456
0.794
0.909
M0-last
0.534
0.888
0.912
M1 layer mix
0.421
0.964
0.985
M2 set pool
0.148
0.782
0.916
M3 set attention
0.150
0.768
0.911
M4-R readout+LoRA
0.076
0.862
0.951
Table 2: Diagnostics on the MuSiQue test split. Order jitter ratio: mean per-record score standard deviation over three re-extracted passage orders divided by the mean FULL-to-PARTIAL gap; the pre-specified pass criterion is below 0.25. Transfer: nominal pair AUC of systems trained on substitution, scored on sets built by another method.
Control
Dataset
Slice
Records
Clusters
Keff (largest share, %)
Edit-matched
MuSiQue
real
122
52
20.9 (16.4)
synthetic
98
40
14.64 (20.4)
joint
220
52
18.22 (18.2)
2Wiki
real
287
28
1.7 (76.3)
synthetic
141
19
1.63 (78.0)
joint
428
28
1.68 (76.9)
Table 3: Dependence structure of the follow-up controls. Clusters are shared-document groups; Keff is the effective number of clusters and the parenthesis gives the share of records in the largest cluster. A clustered interval counts as informative only with at least 20 effective clusters, at least 20 clusters and a largest share of at most one half. Only the MuSiQue real slice of the edit-matched control meets the rule; the title-masking MuSiQue real slice falls just below it after masking excludes records.
Analysis
Records
Finding
Evidence grade
HotpotQA independent confirmation
2,276 pairs
H1-type and H2-type patterns not replicated
confirmatory
Crossed control, arm E
2,214 pairs
all 11 systems bridge-consistency sensitive (CS4); all edit-count sensitive
exploratory
Crossed control, holdouts C1/C2
50 / 231 pairs
CS4 direction agrees for every system
indicative
Batch-layout diagnostic
8,856 records
layout dependence confirmed; gate passes after recovery
methodological
Table 4: Evidence tiers of the analyses in Section 11 . Confirmatory: decision rules, sample and systems fixed before any score of the set was computed. Exploratory: items whose earlier scores had been read. Indicative: unseen holdouts too small for a verdict under the pre-specified size rule, reported as point estimates only. Methodological: a reproducibility check of the scoring pipeline, not a result about systems. The grade describes how the analysis was designed, not how large an effect is; effect sizes and intervals are in Tables 5 and 6 .
System
Nominal
CE
EE
Σ [95% CI]
Σm [95% CI]
M0 single layer
0.995
+0.192
+0.432
− 0.240 [ − 0.262, − 0.216]
− 0.019 [ − 0.035, − 0.003]
M0-last
0.975
+0.312
+0.324
− 0.012 [ − 0.036, +0.013]
+0.132 [+0.114, +0.150]
M1 layer mix
0.996
+0.355
+0.358
− 0.004 [ − 0.021, +0.022]
+0.125 [+0.116, +0.145]
M2 set pool
0.996
+0.347
+0.456
− 0.108 [ − 0.130, − 0.097]
− 0.002 [ − 0.012, +0.009]
M3 set attention
0.994
+0.365
+0.436
− 0.071 [ − 0.090, − 0.059]
+0.022 [+0.009, +0.031]
M4-R readout+LoRA
0.997
+0.418
+0.470
− 0.052 [ − 0.066, − 0.038]
+0.011 [+0.002, +0.020]
Table 5: HotpotQA independent confirmation (train split, real entities, closed-book-hard, n=2,276 pairs, each its own document group). Intervals are document-group bootstraps. Σm=CEm−∣EE∣ replaces D with the edit-matched cell D′ . M0-last is a reference column outside the H1-type set.
Arm E (exploratory)
C1
C2
System
CS4 [adj. CI]
EC [adj. CI]
PT [adj. CI]
CS4
CS4
M0 single layer
0.407 [0.396, 0.417]
+0.374 [+0.354, +0.394]
+0.171 [+0.142, +0.198]
0.380
0.451
M1 layer mix
0.481 [0.476, 0.486]
+0.352 [+0.330, +0.371]
− 0.057 [ − 0.087, − 0.031]
0.485
0.486
M2 set pool
0.454 [0.445, 0.461]
+0.300 [+0.286, +0.329]
+0.189 [+0.164, +0.215]
0.440
0.464
M3 set attention
0.450 [0.443, 0.459]
+0.240 [+0.214, +0.260]
+0.133 [+0.114, +0.166]
0.450
0.462
M4-R readout+LoRA
0.484 [0.478, 0.489]
+0.327 [+0.303, +0.349]
− 0.006 [ − 0.036, +0.024]
0.485
0.474
Table 6: Crossed edit-trace control. CS4 averages four contrasts in which a chain-intact and a chain-broken cell differ in one passage and every passage version appears equally often in both chain states, so any additive per-passage scorer has CS4 =0 ; a scorer that only checks whether the bridge string in the earlier passage matches the later title (AB3) has CS4 =0.5 . EC compares two broken cells that differ only in one extra edit; PT compares editing the earlier passage with editing (and retitling) the later one. Adjusted intervals are Bonferroni-corrected for the 11 systems in the family (document-group bootstrap, 10,000 resamples); each statistic (CS4, EC, PT) is corrected separately, and the family is the 11 systems listed above the baselines (M0–M3, M4-R, M4-J and the five judges). Baselines are not in the family and carry no verdict. Arm E ( n=2,214 ) reuses HotpotQA confirmation items and is exploratory; C1 ( n=50 , HotpotQA validation) and C2 ( n=231 , 2WikiMultiHopQA train) are unseen holdouts below the size threshold, reported as indicative point estimates only.
Asset
Repository (Hugging Face)
Revision
Dataset and evaluator
datasets/ThakiCloud/ChainCheck
825d836d1b
Released judges
ThakiCloud/ChainCheck-Judge-*
not used in this paper
Backbone, reference judge
Qwen/Qwen3.8-27B
dbdc473dea
Additional judge
meta-llama/Llama-3.1-8B-Instruct
8c22764a7e
Additional judge
microsoft/phi-4
2db69c1c3e
Additional judge
Qwen/Qwen3-32B
a09d5ccb54
Table 7: Public assets. The released judges (Qwen3.5-4B, 9B and Qwen3.8-27B variants) are listed for completeness; they are not the checkpoints evaluated in this paper.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Point-estimate rank contrast on 2WikiMultiHopQA (real bridge entities), shown for completeness. The judge and M0-last differ in selectivity by +0.003 [ − 0.083, 0.091] under the pair bootstrap, so the rank change is not evidence of an ordering. Systems are ordered by nominal pair AUC; solid line, rank by nominal AUC; dashed line, rank by chain selectivity. The vertical axis plots 8+1−rank , so the best system is at the top; ranks quoted in the text use 1 for the best. M4-R and M4-J are the LoRA-adapted readout and judge. The zero-shot judge is rank 8 (last) by nominal AUC and rank 1 (first) by the point estimate of selectivity; ranks are point-estimate ranks, not significance tests. The judge’s interval (+0.163 [0.085, 0.234]) overlaps those of M0-last (+0.159 [0.093, 0.232]), M1 (+0.112 [0.063, 0.169]), M3 (+0.078 [0.030, 0.115]) and M2 (+0.054 [0.005, 0.086]); the judge and M0-last are nearly tied, and M2 and M3 tie by nominal AUC. Values are in Table 8 .
System
Nominal
Chain effect
Edit effect
Σ seed median [seed-avg. boot.]
MuSiQue test (exploratory, post hoc) , n=324
M0 single layer
0.968
− 0.025
+0.438
− 0.463 [ − 0.528, − 0.395]
M0-last
0.912
+0.120
+0.265
− 0.145 [ − 0.225, − 0.065]
M1 layer mix
0.971
+0.157
+0.312
− 0.154 [ − 0.226, − 0.074]
M2 set pool
0.989
+0.176
+0.441
− 0.265 [ − 0.332, − 0.221]
M3 set attention
0.992
+0.228
+0.420
− 0.191 [ − 0.258, − 0.145]
Appendix
Table 8: All ChainCheck metrics on closed-book-hard pairs (real bridge entities). Nominal is AUC(A > D); chain effect is AUC(B > D) −0.5 ; edit effect is AUC(A > B) −0.5 ; selectivity is chain effect minus the absolute edit effect, with a 95% percentile pair-bootstrap interval (2,000 resamples of accepted quads; not clustered by source question or document). For trained systems the point estimate is a median over three seeds, while the interval resamples seed-averaged per-quad outcomes, so it is not an interval for the displayed median. M0–M4 are trained on the nominal contrast; the judge is untrained. MuSiQue nominal values are ledger values from the original run; recomputed from the frozen features, the MuSiQue confirmation nominal of M0 is 0.918 (many exact score ties; cause unresolved; Section 7 ), so its nominal rank is unstable.
System
Nominal
Chain effect
Edit effect
Σ seed median [seed-avg. boot.]
MuSiQue test (exploratory, post hoc) , n=224
M0 single layer
0.991
− 0.062
+0.464
− 0.527 [ − 0.603, − 0.451]
M0-last
0.942
+0.183
+0.317
− 0.134 [ − 0.228, − 0.045]
M1 layer mix
0.978
+0.268
+0.299
− 0.031 [ − 0.125, 0.039]
M2 set pool
0.991
+0.290
+0.451
− 0.161 [ − 0.232, − 0.119]
M3 set attention
0.987
+0.317
+0.442
− 0.125 [ − 0.176, − 0.065]
Appendix
Table 9: All ChainCheck metrics on closed-book-hard pairs (synthetic bridge entities). Nominal is AUC(A > D); chain effect is AUC(B > D) −0.5 ; edit effect is AUC(A > B) −0.5 ; selectivity is chain effect minus the absolute edit effect, with a 95% percentile pair-bootstrap interval (2,000 resamples of accepted quads; not clustered by source question or document). For trained systems the point estimate is a median over three seeds, while the interval resamples seed-averaged per-quad outcomes, so it is not an interval for the displayed median. M0–M4 are trained on the nominal contrast; the judge is untrained.
System
Nominal
Chain effect
Edit effect
Σ seed median [seed-avg. boot.]
MuSiQue confirmation (pre-specified) , n=122
Qwen3.8-27B (reference)
0.943
+0.328
+0.189
+0.139 [0.025, 0.262]
Llama-3.1-8B-Instruct
0.779
+0.147
+0.082
+0.066 [ − 0.082, 0.197]
phi-4
0.877
+0.049
+0.254
− 0.205 [ − 0.336, − 0.066]
Qwen3-32B
0.820
+0.205
+0.115
+0.090 [ − 0.049, 0.238]
Gemma-4-31B-it
0.873
+0.414
− 0.033
+0.381 [0.299, 0.434]
Appendix
Table 10: Zero-shot judges from four model families, same prompt and score definition as the reference judge, real bridge entities. Every judge is reported; none is selected. Synthetic-entity results are in Table 11 .
System
Nominal
Chain effect
Edit effect
Σ seed median [seed-avg. boot.]
MuSiQue confirmation (pre-specified) , n=98
Qwen3.8-27B (reference)
0.929
+0.459
− 0.031
+0.429 [0.337, 0.469]
Llama-3.1-8B-Instruct
0.725
+0.214
− 0.031
+0.184 [0.061, 0.255]
phi-4
0.775
+0.122
+0.122
+0.000 [ − 0.163, 0.153]
Qwen3-32B
0.775
+0.347
− 0.133
+0.214 [0.122, 0.306]
Gemma-4-31B-it
0.862
+0.403
− 0.143
+0.260 [0.184, 0.342]
Appendix
Table 11: As Table 10 , synthetic bridge entities.
Figure 5: Chain selectivity with 95% pair-bootstrap intervals, synthetic bridge entities. Open diamonds mark the reference zero-shot judge.
Figure 6: Nominal dev pair AUC of a single-layer probe on each full-attention layer of the backbone.