Evaluation Contracts and Evaluation Leverage: Sliced-Gap Readings in Structured Prediction Can Move Without Model Change
Organizations: Army Engineering University of PLA, Nanjing, China
Abstract
NLP evaluation reports routinely cite sliced performance differences, most prominently the intra-inter gap in document-level relation extraction, as capability evidence. We show that such gap readings are joint properties of models and evaluation contracts: scoring function, gold version, threshold, output format, and prompt language. With predictions frozen, a method ordering reverses under a gold-standard exchange (+0.0363, CI excluding zero; tie-level after), movement localizing where gold re-partitions: renames are gauge transformations (Delta=0; re-annotation +0.862). We formalize this as an evaluation leverage account centered on the rank-flip condition. Constructively, the audited evaluation reading (AER) is a contract-marginalized, tie-declared, seed-prescribed, gate-filtered procedure whose tie declarations cover every contract-induced flip on the declared WMT19 grids (14/14; held-out 73% vs. union-of-ties 89%), matching the best single metric contract at epsilon=0.10 equivalence. Evidence spans DocRE, HacRED, a UD probe, and released gold revisions; an alternative-reference-set replication extends it beyond our testbeds. Failed predictions retained as boundaries; a 5% violation gate flags unverifiable settings and bounds survivor-selection bias. We release EvalLever, a preregistered workflow with an amendment trail; on LLM-as-judge grids, the audited ensemble stabilizes every unstable pair (13-15 to zero; generic ensemble shrinkage: the preregistered correlation criterion failed, the companion stabilization criterion passed).
Figures & tables
| Claim | gD | gR | Verdict |
|---|---|---|---|
| DREEAM ATLOP | reversed | ||
| KD-DocRE DocuNet | holds, arm | ||
| TTM-RE KD-DocRE | opposite |
| AER audit ( , r60) | de-en | zh-en |
|---|---|---|
| point (CI95) | ||
| best single (post hoc) | ||
| vs. best | ||
| TOST / | pass / pass | pass / pass |
| flips covered by ties |
| Pattern | Reading |
|---|---|
| (i) Judge leverage | judge swap moves rankings: – vs. anchor; -gap (rubric , format ); pair-level – |
| (ii) Judge variance | monotone noise ladders (single-sample SD above five-vote residual); candidate-order flips – of verdicts |
| (iii) AER suppression | unstable pairs – (singles ); generic shrinkage: median-of-z = best single; ensemble , CI ( discreteness); H-J4 fails, H-J3 met |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Bench. | Cand. | Grouping rule | Gold |
|---|---|---|---|
| DocRED | 189,556 pairs (498 docs) | earliest_mention : intra iff both mentions fall in one sentence (earliest-mention coreference) | gD, gR |
| HacRED | 15 seeds | Gold segmentation; intra iff both entities share a sentence | one |
| BC5CDR | 500/500/500 | Mention-pair expansion of concept-level CID gold; title one sentence | official CID |
| RAMS | 5-sentence windows | Intra iff the argument is in the trigger’s sentence | official |
| Model | Prompt–data | F | Rate% | CI95 | 3% | 5% | 10% |
|---|---|---|---|---|---|---|---|
| Q14B | D | J | 4.57 | — | |||
| P4 | H-zh | J | 7.36 | — | — | ||
| M7B | D | F | 7.05 | — | — | ||
| Q32B | H-zh | J | 21.94 | — | — | — | |
| M7B | H-zh | F | 10.91 | — | — | — | |
| Q7B | D | J | 11.40 | — | — | — |
| Tier | Definition |
|---|---|
| T1 | Preregistered and frozen contract, seed prescription met, fault-injected audit passed. |
| T2 | Fully reported, multiple seeds, not audited. |
| T3 | Single contract, single seed—reference only. |
| Requirement | Status | Where |
|---|---|---|
| Freeze contract before runs | met | 147 logged amendments (prereg doc) |
| Retain negative results | met | 5 boundary findings; H1/H-X1 failed; 14-of-15 gate flags |
| Halt defective launches | met | 4 launches halted, re-derived from raw inputs |
| Human verification | partial (point est. only; Wilson fails) | 50 stratified violation checks |
| Value integrity | met | 26/26 paper values matched to frozen files |
| Registered, executed post-review | logged | constrained decoding (am. 19b; Table 5 ) |
| Family | Configuration | Seeds | Source |
|---|---|---|---|
| X2 | BERT-base, two sparsity levels | 5 | internal |
| ATLOP-PE8 / -full | protocol pair (PE8 drops calib. shard) | 3+3 | retrained |
| DREEAM, DocuNet, KD | same PE8, F1 .577–.613 | 2–3 | retrained |
| P3M | PU metric learning, shared scoring | 5 | retrained |
| LLMs | 7/14/32B + Mistral + Phi-4, 4-shot | 1/cell | released |
| Symbol | Meaning |
|---|---|
| contract: scoring, gold, threshold, format, language | |
| , | real-valued reading; the sliced gap (intra inter) is its primary instance |
| arm | realized movement under a contract switch |
| error-set agreement between models (Jaccard; § 3.1 ) | |
| arm D | disagreement-referenced leverage over (§ 3.1 ) |
| flip | sign change of across contracts |
| Claim | Evidence tier | Registered |
|---|---|---|
| Rename invariance (§3.1) | derived; UD slices preregistered | initial |
| C1–C3 main comparisons | T1 | initial (Holm) |
| P1 family separation | derived envelope (Lemma A, accuracy-premise: arm , i.e., ) + T2 slack ordering (multi-seed, not audited) | initial |
| P2 flip registry | registered checks (exploratory status, § 4.2 ) + Lemma B guarantee | initial |
| P3 variance ordering | derived under nested-independent assumptions (Theorem C); T1-conditional magnitudes (registered criterion fails strictest dedup) | initial; H3 amended (am. 6) |
| Seed prescription | T1-conditional (single curve) | initial |
| Family | Resamples | Unit | Seed |
|---|---|---|---|
| Main arms/flips (C1–C3) | 10k | doc | 0926 |
| 57-check reg. | 10k | doc | 0926 |
| AER valid. (r60) | 2000 | segment | 7 |
| Value exps (r55) | 200/200 | seg/sys | 7/11 |
| Claims audit (r56) | 10k | doc | 0926/+1 |
| Error decomp (r57) | — | exact | — |
| Line of work | Pred. | Legal | Der. | Tie |
|---|---|---|---|---|
| frozen | gold | inv. | aware | |
| Metrics tasks ( 2023 ) | ✓ | ✗ | ✗ | ✗ |
| Label-error re-scoring ( 2021 ) | ✓ | ✗ | ✗ | ✗ |
| Gold re-scoring ( 2025 ) | ✓ | ✓ | ✗ | ✗ |
| Gold revision: TACREV ( 2020 ) | ✓ | ✓ | ✗ | ✗ |
| Gold revision: Re-TACRED ( 2021 ) | ✓ | ✓ | ✗ | ✗ |
| Setting | Density | Arm | CI95 |
|---|---|---|---|
| X2 | 0.204 | 0.022 | |
| ATLOP-PE8 | 0.048 | 0.063 | |
| ATLOP-full | 0.047 | 0.043 | |
| DREEAM | 0.027 | 0.001 | |
| P3M | 0.080 | 0.026 | |
| DocuNet | 0.042 | 0.016 |
| Pair | Dim. | CI95 | Holm | ||
|---|---|---|---|---|---|
| Doc KD | thr | -0.259 | |||
| DRE Doc | thr | 0.238 | |||
| PE8 Doc | thr | 0.191 | |||
| AT Doc | thr | 0.190 | |||
| X2 Doc | thr | 0.096 | |||
| AT KD | thr | -0.069 |
| Generator | Runs | Arm | CI95 | Arm(gold) |
|---|---|---|---|---|
| ATLOP-PE8 | 6 | 0.030 | 0.063 | |
| ATLOP-full | 6 | 0.028 | 0.043 | |
| DREEAM | 3 | 0.021 | 0.001 | |
| DocuNet | 2 | 0.029 | 0.016 | |
| KD-DocRE | 2 | 0.020 | 0.013 |
| Arm | Gold [CI] | Grouping [CI] | Thr. | Dens. |
|---|---|---|---|---|
| DocuNet-L | .025 | .021 | .024 | .039 |
| TTM-RE s74 | .038 | .018 | – | .070 |
| TTM-RE s2 | .046 | .022 | – | .037 |
| Qwen NF4 | .006 | – | – | .032 |
| BF16 greedy | – | – | .039 | |
| ( ) | .007–.015 | – | – | – |
| B | Setting | Result | Import |
|---|---|---|---|
| B1 | SST-2, NER | pooled (6 pts., descr.); per-task / ( ) | Construct differs |
| B2 | HacRED, 4 settings | exactly, 4 replications | Layers identical |
| B3 | X2, | Sign undetermined at every | Arm bounds seeds |
| B4 | XNLI, strong models | Acc. near ; no flips | Margin clause |
| B5 | All LLM settings | Smoke vs. full both ways | Full-run gate |
| Family / axis | Reading | Arm or flips |
|---|---|---|
| Retrieval: gold rubric | nDCG@10 | – (5 runs) |
| (graded binarized) | MAP@100, bpref | (by construction) |
| MT de-en: scorer | BLEU chrF | 3/120 pairs flip |
| (16 frozen systems) | BLEU BERTScore | 10/120 flip, |
| MT: format | BLEU 13a none | BLEU |
| QA: gold population | F1 |
| Arm | [CI] | Seeds | ||
|---|---|---|---|---|
| English EWT | ||||
| C stanza | .962 | .919 | 1 | |
| A perc. | .928 | .887 | 15 + | |
| B (BERT) | .927 | .890 | 15 + | |
| RL probe | .931 | .893 | 15 + | |
| B ′ (BERT) | .892 | .921 | 15 - | |
Automated Evaluation
Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.
See all 420 papers