Contract-Explicit Evaluation: From Auditing Learned Readers to Scoped Certificates
Organizations: Army Engineering University of PLA, Nanjing, Jiangsu, China.
Abstract
Evaluation readings of document-level relation extraction (DocRE) systems depend jointly on the model and the evaluation contract: gold version, scoring function, threshold. Rankings reverse across contracts, yet no existing method determines which comparisons survive the reversal. We address this gap with contract-explicit evaluation: a preregistered five-step protocol (Declare, Measure, Scope, Certify, Hedge), operationalizing the classical concordance decomposition of Kendall's rank correlation, that scopes certifiable regions and delivers distribution-free census certificates across two ecosystems. In the DocRE audit (44 training runs of one in-house architecture family), the scoring dimension scopes to a census certificate (three flagged instances, Clopper--Pearson upper bound 0.146); the confirmatory record includes registered negatives, and the audit prices its own simplification finding---the calibration dial's entire margin is one withdrawn commit. In an LLM-judge census (17 judges, 49,938 battles), 12.25% of leaderboard comparisons require hedging at the pre-declared split criterion (bound 0.124; panel-robust 0.143), with an exploratory two-tier reading certifying 69% at a bound \le 0.072 and a validity decomposition against released human labels (65% of contested battles are judge error, not ambiguity). The cross-model certificate is withdrawn on real confidence scores; the failure spectrum maps the boundary. The protocol, map, ledger, and per-instance artifacts are released.
Figures & tables
| Component | Role | Lineage | Certification status |
|---|---|---|---|
| Per-document contract weights (16-hidden MLP) | document-conditioned weighting over contract readings | new | corpus-level collapse confirms conditioning is necessary |
| Contract-consensus objective (median-sign target) | supervision signal | new | Prop. 1; fidelity (training sanity) |
| Soft-tie kernel with annealing | differentiable tie handling | adapted from tie-aware ranking objectives [ 8 ] | loss ablation: inert at registry granularity |
| Leave-one-contract-out penalty ( ) | stability regularizer | new | ablation: the only active lever |
| Risk-controlled gate + CP dial (Part 2) | decision layer over reader commits | Risk-Controlling Prediction Sets (RCPS) [ 40 ] applied to reader decisions | gate-layer J5 ′ –J7 ′ (registered, second cycle) |
| Reader | Sign agreement ( ) | vs. ( ) |
|---|---|---|
| native (single contract) | 0.810 | 0.905 |
| AER, corpus form (training contracts) | 0.714 | 0.905 |
| Consensus median (fixed) | 0.571 | 0.143 |
| Uniform contract mixture (fixed) | 0.381 | |
| Learned, corpus-level weights | — | |
| Learned, trivial features | (dev 0.667 0.095) | — |
| Comparison | sign agreement [CI95] |
|---|---|
| Learned native | |
| Learned AER (corpus) | |
| Learned uniform mixture | |
| Learned consensus median |
| Reader | Coverage | False tie | Hard-ranked | Fidelity |
|---|---|---|---|---|
| native | 7/13 | 19/44 | 6 | 4/6 |
| AER, corpus form | 7/13 | 13/44 | 6 | 4/6 |
| Uniform mixture | 13/13 | 44/44 | 0 | — (degenerate) |
| Learned (ours) | 5/13 | 7/44 | 8 | 8/8 |
| Variant | Full 498 | Dev 148 |
|---|---|---|
| Learned, full features | ||
| Learned, trivial features (length, candidates) | ||
| Learned, corpus-level weights (no document conditioning) |
| Variant | Coverage | False tie | Fid. ( ) | Sign ( ) |
|---|---|---|---|---|
| baseline( =2) | 5/13 | 7/44 | 6/8 | 12/21 |
| =0 | 4/13 | 5/44 | 7/9 | 13/21 |
| =0.5 | 5/13 | 7/44 | 6/8 | 13/21 |
| =8 | 5/13 | 9/44 | 6/8 | 11/21 |
| no anneal(D=1.0) | 5/13 | 7/44 | 6/8 | 12/21 |
| no soft tie(t=0) | 5/13 | 7/44 | 6/8 | 12/21 |
| Reader | Micro fidelity | Macro fidelity | Overall (micro) |
|---|---|---|---|
| native | 0.000 (definitional) | — | 0.871 |
| Consensus median | 0.239 | 0.671 | 0.869 |
| Uniform (AER analogue) | 0.261 | 0.693 | 0.871 |
| Learned, corpus weights | 0.275 0.005 | 0.680 0.005 | |
| Learned, conditioned | 0.268 0.009 | 0.695 0.005 |
| Flip cost high | Flip cost low | |
|---|---|---|
| Exchangeable | Learn , hedging contested pairs (DocRE, conditionally) | Report parity (WMT) |
| Non-exchangeable | Single gold (–) | Report as-is (–) |
| Criterion | Frozen form | ||
|---|---|---|---|
| J5 ′ wrong-commit rate | ( ; ) FAIL | ( ; ) PASS | realized rate , both folds |
| J6 ′ flip coverage | – PASS | – PASS | AER ( ) |
| J7 ′ false ties | – FAIL | – PASS | min fixed baseline ( ) |
| Readout branch | 3 (not certified) | 1 (all pass) | — |
| Conformal layer (J5, ) | ||
|---|---|---|
| Commits / test | 9 | 14 |
| Wrong | 2 | 6 |
| Realized rate | 0.222 | 0.429 |
| (saturated) | 1.0 | 1.0 |
| Verdict | FAIL | FAIL |
| Dimension | flip density | ||
|---|---|---|---|
| scoring ( vs. ) | 0.0652 | 0.0652 | |
| threshold (fixed vs. cal) | 0.2609 | 0.2609 | |
| gold ( vs. ) | 0.5543 | 0.5543 |
| T1 ( ) | T2 ( ) | T3 ( ) | T4 ( ) | |
|---|---|---|---|---|
| minimum | 22 | 15 | 11 | 7 |
| Population | What it is | Where used |
|---|---|---|
| Registry (v3) | 207 checks / 52 flips over 39 generator-pair units, 14 families | Part 1–2 decision layer; Part 3 cross-model boundary (§ 9.4 ) |
| Stratum ( ) | 44 training instances (arm ratio seed) of one family | Part 3 census certificate (this subsection) |
| Program | Mechanism | Feas. | Exceed | Verdict |
|---|---|---|---|---|
| block-gated fixed set | per-block gates too strict at small strata | 0.4% | 0.000 (vacuous) | killed |
| design-effect-scaled CP | variance correction is not a guarantee | 0.2% | 0.250 | killed |
| leave-one-family CP | holdout does not decorrelate overlapping blocks | 19% | 0.155 | killed |
| pair-block aggregation | shared families break block exchangeability | 7% | 0.121 | killed |
| stratified fixed set | validity is pool-luck, not method | 29% | 0.181 | killed |
| family exceedance ( ) | zero-bad floor needs families | 0% | — | wall: |
| Margin stratum | Battles | Contested | 70B | CP bound |
|---|---|---|---|---|
| High ( ) | 5,994 | 0.0% | 0.989 | 0.0006 |
| Mid ( – ) | 28,275 | 7.0% | — | 0.0719 |
| Low ( ) | 15,395 | 26.7% | 0.555 | 0.2712 |
| Overall | 49,664 | 12.25% | — | 0.1244 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Generator A | Generator B | Axis | CI excl. | ||
|---|---|---|---|---|---|
| x2 | dreeam | Gold | 0.016 | -0.005 | |
| x2 | docunet | Threshold | 0.090 | -0.006 | ✓ |
| atlop_pe8 | dreeam | Threshold | -0.046 | 0.002 | ✓ |
| atlop_pe8 | p3m | Gold | 0.031 | -0.005 | ✓ |
| atlop_pe8 | docunet | Threshold | 0.038 | -0.153 | ✓ |
| atlop_pe8 | kddocre | Threshold | -0.047 | 0.022 | ✓ |
| Family | Description | Runs/Seeds | Threshold |
|---|---|---|---|
| X2 multi-label | Companion-internal multi-label model; 20 runs span two decoding regimes ( scores at radii and ) five seeds; continuous scores (sigmoid multilabel) | 20/5 | Yes |
| ATLOP PE8 | Companion retrain of ATLOP [ 48 ] under the formal protocol (calibration split excluded); 3 seeds | 6/3 | Yes |
| ATLOP full | Companion’s first-round full-trained ATLOP variant [ 48 ] (ties retained in reporting) | 6/3 | Yes |
| DREEAM [ 46 ] | Published; evidence-guided attention over ATLOP | 3/3 | Yes |
| P3M (rule bridge) | Companion-internal rule-bridged decoder: native decision rules emit predictions directly, so no continuous scores and no threshold face | 5/5 | No |
| DocuNet [ 2 ] | Published; semantic-segmentation formulation | 2/2 | Yes |
| Rule | Commits | Wrong | Rate | Coverage | False ties |
|---|---|---|---|---|---|
| no-op (copy ) | 39 | 1 | 0.034 | 0/52 | 0/155 |
| consensus-median | 39 | 6 | 0.207 | 0/52 | 0/155 |
| gate-only (no dial) | 24 | 1 | 0.042 | 22/52 | 27/155 |
| gate + CP dial (official) | 23 | 1 | 0.043 | 23/52 | 30/155 |
Language Model Auditing
Language model auditing tests whether models behave reliably, fairly, and safely using behavioral probes and internal or deployment evidence.
See all 174 papers