When Does Better Confidence Improve a Knowledge-Graph Audit?

Zehao Lin; Kai Chen; Xixuan Hao; Erenkai Zhu; Chunyu Li; Helan Hu

cs.CLNov 4, 2026

When Does Better Confidence Improve a Knowledge-Graph Audit?

Authors: Zehao Lin, Kai Chen, Xixuan Hao, Erenkai Zhu, Chunyu Li, Helan Hu

Organizations: 1MemTensor · 3The Hong Kong University of Science and Technology (Guangzhou) · 2Shanghai Jiaotong University

Abstract

Confidence estimates guide human review in LLM-built knowledge graphs, but better error prediction can yield little improvement in an audit. We study this gap through repairability, selection headroom and label validity. Modelling non-adaptive auditing as budgeted covering, we characterise the minimal decision-relevant error statistics and bound the regret of scalar-preserving first-order planning by a structural mass $Δ^{\mathrm{cl}}$ . We simulate ideal repair against benchmark-aligned labels for five extraction configurations and two cross-document graphs on Re-DocRED, followed by frozen confirmation on test and replication on 500 hash-selected training documents. On the held-out cross-document graphs, an error-rate $\times$ fan-out queue leaves 3.98-4.74 percentage points more errors than a joint model with the same actions. By comparison, opening shared actions for that joint planner reduces residual errors by 0.89-1.43 points relative to its own fact-only plans, in point estimates. Equivalence between joint and independent-block models remains inconclusive. Generic LLM verification improves three of four model-budget configurations on each held-out set. LLM adjudication identifies possible benchmark omissions.

Figures & tables

Figure 1: The audit evaluation pipeline. Error predictions and a fixed action menu determine a plan under the review budget. Evaluation simulates ideal repair against benchmark-aligned labels: fact $i$ is correct after review exactly when the chosen actions cover its complete error set, $E_{i}\subseteq\tau_{i}(\mathcal{P})$ .

Panel of Figure 2 — Figure 2: Reachable coverage for one fact: $h$ and $t$ are entity decisions and $\ell$ is the local relation decision. Solid arrows show the cover relations; whole-fact review reaches $S_{i}$ directly (dashed). The decision-relevant closure variable $Z_{i}$ takes values in the five reachable covered sets $\{\varnothing,h,t,ht,S\}$ .

Contrast	Claude	GPT-4.1	mini	Qwen	Qwen-R
Initially wrong facts	3,468	5,850	3,974	5,039	3,339
Raw $\Delta$ (facts)	213 (6.14%)	204 (3.49%)	165 (4.15%)	181 (3.59%)	151 (4.52%)
$\Delta^{\mathrm{cl}}$ (facts)	2	2	0	4	2
(a) Oracle: + shared actions	$+3.52$ ∗ (+122) $[+2.37,+4.79]$	$+0.96$ ∗ (+56) $[+0.59,+1.71]$	$+1.01$ ∗ (+40) $[+0.46,+1.84]$	$+1.65$ ∗ (+83) $[+0.96,+2.40]$	$+2.40$ ∗ (+80) $[+1.23,+3.58]$
(b) Learned: + shared actions	$+0.03$ (+1) $[-0.42,+0.44]$	$+0.02$ (+1) $[-0.05,+0.10]$	$+0.00$ (+0) $[+0.00,+0.00]$	$+0.18$ (+9) $[+0.00,+0.41]$	$-0.03$ (-1) $[-0.11,+0.00]$
(c) Node: + shared actions	$-0.89$ ∗ (-31) $[-1.76,-0.26]$	$-0.14$ (-8) $[-0.36,+0.07]$	$-0.40$ ∗ (-16) $[-0.65,-0.10]$	$+0.08$ (+4) $[-0.32,+0.38]$	$-0.21$ (-7) $[-0.75,+0.18]$

Table 1: Within-document development comparisons (lenient labels, GB,

b=0.2

), in percentage points of initially wrong facts; counts appear in parentheses. Positive values mean fewer residual benchmark errors for the named change. Rows a–c add shared actions; row s compares lattice with shared actions against the fact-scalar, fact-only plan; rows d–f hold actions fixed. Row h is the gap to the computed oracle. Brackets: 95% re-planned bootstrap intervals (

R=100

, fitted models fixed); ∗ excludes zero. TOST margins are

\pm 1\%

of initially wrong facts; ? marks an inconclusive cell (Table 7 ).

Contrast	Qwen	GPT-4.1
Initially wrong facts	5,820	6,404
Raw $\Delta$ (facts)	1642 (28.2%) §	1863 (29.1%) §
$\Delta^{\mathrm{cl}}$ (facts)	66 (1.1%) §	54 (0.8%) §
(a) Oracle: + shared actions	$+6.89$ ∗ $[+5.64,+8.16]$	$+4.06$ ∗ $[+3.11,+4.94]$
(b) Learned: + shared actions	$+1.46$ ∗ $[+0.87,+1.98]$	$+0.69$ ∗ $[+0.28,+1.02]$
(c) Node: + shared actions	$-3.13$ ∗ N $[-4.35,-1.92]$	$-3.98$ ∗ N $[-4.87,-3.02]$

Table 2: Cross-document development comparisons (lenient labels, GB,

b=0.2

), in percentage points of initially wrong facts. Positive values mean fewer residual benchmark errors for the named change. Rows a–c add shared actions; row s compares lattice with shared actions against the fact-scalar, fact-only plan; rows d–f hold actions fixed. Row h is the gap to the computed oracle. TOST margins are

\pm 0.5

pp. Brackets: 95% re-planned bootstrap intervals (

R=100

); ∗ excludes zero; N uses budget-normalised intervals (

n=2000

); † regenerated with the original seed. § Structural masses use fact, entity and relink actions; the comparison rows use the full action menu.

Extractor / budget	test	train500
H1, within documents: lattice $-$ independent (errors; TOST, $\pm 0.5\%$ of wrong facts)
Qwen	$+19$ [ $+6$ , $+35$ ]	$+5$ [ $-6$ , $+12$ ] $\checkmark$
GPT-4.1	$-4$ [ $-14$ , $+6$ ] $\checkmark$	$+3$ [ $-6$ , $+13$ ] $\checkmark$
H1, across documents: joint $-$ independent ( pp; TOST, $\pm 0.25$ pp)
Qwen	$+0.31$ [ $-0.05$ , $+0.72$ ]	$+0.32$ [ $+0.08$ , $+0.60$ ]
GPT-4.1	$+0.27$ [ $+0.08$ , $+0.48$ ]	$+0.06$ [ $-0.11$ , $+0.25$ ]

Table 3: Held-out confirmation under the frozen protocol (Appendix E ). Point estimate and re-planned document-cluster bootstrap interval (

R=1000

; 90% for the TOST cells, 95% otherwise). Intervals are unadjusted. Positive values mean that the first plan leaves more benchmark errors; both H1 blocks display joint/lattice minus independent blocks.

\checkmark

indicates rejection after Holm correction over the 10 cells of the split (

\alpha=0.05

). Confirmation (test) or replication (train500) requires every cell to pass. Lenient labels, GB models fitted on dev only,

b=0.2

unless noted.

Set	Extractor	$b$	$D_{\rm fact}$	$D_{\rm full}$	Gain
test	Qwen	0.2	58	155	$+6$
	Qwen	0.3	189	286	$+40\checkmark$
	GPT-4.1	0.2	44	111	$+32\checkmark$
	GPT-4.1	0.3	107	174	$+59\checkmark$
train500	Qwen	0.2	40	141	$+10$
	Qwen	0.3	102	203	$+22\checkmark$

Table 4: Headroom and verification gains, in benchmark errors.

D_{\rm fact}=R_{\rm A0}-R^{*}_{\rm fact}

compares the baseline with the fact-only oracle;

D_{\rm full}=R_{\rm A0}-R^{*}_{\rm full}

uses the same fact+entity actions as the baseline. Gain is

R_{\rm A0}-R_{\rm A1}

. Checkmarks denote the Holm decisions in Table 3 , which also reports the intervals.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Extraction config	ok/fail	verifiable	wrong	P	R	unverifiable (share)	P, unverif. wrong	P, unverif. right
Single-document (Re-DocRED dev, 500 docs, lenient)
Claude Sonnet 5 (direct)	494/6	7,349	3,468	0.528	0.223	1,064 (12.6%)	0.461	0.588
GPT-4.1 (direct)	500/0	8,440	5,850	0.307	0.148	1,550 (15.5%)	0.259	0.414
GPT-4.1-mini (direct)	499/1	5,472	3,974	0.274	0.086	624 (10.2%)	0.246	0.348
Qwen3.5-27B (direct)	499/1	8,696	5,039	0.421	0.208	1,463 (14.4%)	0.360	0.504
Qwen3.5-27B (reason)	459/41	6,818	3,339	0.510	0.200	1,583 (18.8%)	0.414	0.603

Appendix

Table 5: Extraction outputs. verifiable : facts whose two endpoints align to gold entities; wrong : initially wrong verifiable facts under lenient labels, the denominator of every percentage-point number; P: precision over verifiable facts; R: end-to-end recall over all 17,284 dev gold facts, with failed documents counted as misses. Unverifiable facts have an endpoint that matches no gold entity; they are excluded from P, from every audit population and from the budget denominator. The last two columns bound P by counting all of them as wrong or as right.

	Qwen	GPT-4.1
All wrong facts, mean of the two judges
(i) unsupported / contradicted	0.236 [0.197, 0.275]	0.248 [0.204, 0.294]
(ii) convention mismatch	0.227 [0.192, 0.263]	0.202 [0.169, 0.239]
(iii) outside the schema	0.181 [0.145, 0.219]	0.273 [0.229, 0.314]
(iv) supported, missing from gold	0.205 [0.161, 0.248]	0.158 [0.121, 0.197]
not adjudicated	0.146	0.119

Appendix

Table 6: LLM adjudication of benchmark discrepancies. Two cross-family judges (Claude Opus 5.5 and Gemini 3.1 Pro preview) and a reference judge (GPT-6 Astra) assess stratified samples per extractor: 150 spurious, 50 relation and 50 argument errors, plus 170 links. Overall rates are share-weighted means of the first two judges. Direction, entity and invalid-relation errors remain unadjudicated. Category (iv) denotes possible gold omissions; category (ii) includes convention differences and extractor errors. Brackets give 95% document-cluster or stratified bootstrap intervals.

Extr.	vs.	$\hat{d}$	90% CI	$\delta^{*}$	verdict	$\delta^{*}$ /(a)
Single-document, lattice (closure5) model, margin $\pm 1\%$
Claude	indep.	$+0.52$	$[-0.20,+0.92]$	0.92	equiv.	26%
Claude	direct	$+0.61$	$[-0.32,+1.01]$	1.01	inconcl.	29%
Claude	joint8	$+0.52$	$[-0.12,+0.87]$	0.87	equiv.	25%
GPT-4.1	indep.	$+0.10$	$[-0.04,+0.26]$	0.26	equiv.	27%
GPT-4.1	direct	$+0.05$	$[-0.05,+0.17]$	0.17	equiv.	18%

Appendix

Table 7: Equivalence (TOST,

\alpha=0.05

) of the lattice/joint model with independent blocks, direct utility regression and the full 8-class mask model at the same action set (primary cells only: lenient, GB,

b=0.2

, re-planned bootstrap

R=100

\hat{d}

: extra errors the lattice/joint model removes, in % of the initially wrong facts;

\delta^{*}

: smallest margin at which the cell would be declared equivalent;

\delta^{*}

/(a):

\delta^{*}

relative to the oracle value of the shared actions (Tables 1 – 2 , row a). Relative to the learned gain of the shared actions, the cross-document

\delta^{*}

are 32% and 18% (Qwen, gain

+1.46

) and 38% and 35% (GPT-4.1, gain

+0.69

); within documents the learned gain is about 0. Development margins were selected after the initial intervals were reported; the bootstrap conditions on fitted models.

Extractor	$b=0.1$	$b=0.2$	$b=0.3$
(b) learned + shared actions
Claude	$-0.17$ $[-0.42,+0.12]$ (-6)	$+0.03$ $[-0.42,+0.44]$ (+1)	$+0.06$ $[-0.39,+0.43]$ (+2)
GPT-4.1	$+0.02$ $[-0.05,+0.10]$ (+1)	$+0.02$ $[-0.05,+0.10]$ (+1)	$+0.02$ $[-0.07,+0.08]$ (+1)
mini	$+0.00$ $[+0.00,+0.00]$ (+0)	$+0.00$ $[+0.00,+0.00]$ (+0)	$+0.00$ $[+0.00,+0.00]$ (+0)
Qwen	$+0.16$ $[-0.03,+0.39]$ (+8)	$+0.18$ $[+0.00,+0.41]$ (+9)	$+0.12$ $[-0.06,+0.36]$ (+6)
Qwen-R	$-0.03$ $[-0.11,+0.00]$ (-1)	$-0.03$ $[-0.11,+0.00]$ (-1)	$+0.00$ $[-0.08,+0.06]$ (+0)

Appendix

Table 8: Single-document contrasts at all three budgets (lenient; percentage points of initially wrong facts, fact counts in parentheses). Intervals are re-planned (

R=100

) unless marked N (budget-normalised,

n=2000

); ∗ excludes 0. The two interval types agree on significance in 252 of 267 lenient cells (approximate check only). No multiplicity correction.

Contrast ( $b=0.2$ )	Claude	GPT-4.1	mini	Qwen	Qwen-R
Gao et al. costs: fact review 25 s, entity review 45 s
Oracle + entity reviews	$+83$ ∗ $[+51,+121]$	$+33$ ∗ $[+13,+61]$	$+26$ ∗ $[+8,+55]$	$+60$ ∗ $[+30,+98]$	$+55$ ∗ $[+27,+95]$
Learned + entity reviews	$-3$ $[-8,+0]$	$+0$ $[+0,+0]$	$+0$ $[+0,+0]$	$+0$ $[-3,+3]$	$-2$ $[-6,-0]$
Lattice vs. fact scalar	$+14$ $[-10,+37]$	$+2$ $[-7,+12]$	$-6$ $[-15,+0]$	$+7$ $[-3,+18]$	$+5$ $[-14,+24]$
Gap to oracle	$+221$ ∗ $[+180,+267]$	$+64$ ∗ $[+41,+95]$	$+45$ ∗ $[+23,+77]$	$+111$ ∗ $[+77,+152]$	$+208$ ∗ $[+167,+259]$
Time costs: fact review 25 s, entity review 15 s + 3 s per mention, 60 s per document opened

Appendix

Table 9: Single-document results under two non-unit cost models (lenient, GB,

b=0.2

of the cost of reviewing every verifiable fact; counts of benchmark errors, positive = the named change leaves fewer; budget-normalised 95% intervals,

n=2000

). The fact-scalar model buys no entity review under any cost model. The pattern of Table 1 is unchanged: the oracle gains from entity reviews, learned plans do not.

Ranker	AUROC	$b=0.1$	$b=0.2$	$b=0.3$
Qwen extraction (verifier GPT-4.1-mini): benchmark errors removed by the top $\lfloor bN\rfloor$ facts
Verbal confidence (label-free)	0.754	791	1,521	2,212
Triple log-probability (label-free)	0.631	758	1,417	1,992
Verifier $P(\text{no})$ alone (label-free)	0.783	825	1,599	2,325
A0: GB, fact scalar	0.873	859	1,681	2,441
A1: GB, fact scalar	0.892	858	1,697	2,489

Appendix

Table 10: Label-free and label-trained fact-review plans on the two Experiment A extractions (lenient labels, dev, exploratory). Every ranker reviews the

\lfloor bN\rfloor

facts it scores as riskiest; ties are broken by a hash of the fact key. The label-free rankers use the extractor’s verbal fact confidence, the summed token log-probability of the generated head, relation and tail, or the verifier’s

P(\text{no})

from call G alone (25 failed calls on the Qwen extraction set to the median). A0 and A1 are the out-of-fold GB models of Experiment A, used here as fact-only rankers; they were fitted on dev labels. The A0 fact-scalar rows reproduce Experiment A’s fact-scalar plans exactly. Intervals: document-cluster bootstrap in which the top-

b

set is re-selected on every resample; ∗ excludes 0.

		Qwen extraction				GPT-4.1 extraction
		A0 selection gap		A1 $-$ A0		A0 selection gap		A1 $-$ A0
Population	P	$b=0.1$	$b=0.2$	$b=0.1$	$b=0.2$	$b=0.1$	$b=0.2$	$b=0.1$	$b=0.2$
Random subsample	0.5	$11.2$	$75.3$	$-4.1$	$+21.8$	$18.6$	$69.9$	$+14.6$	$+58.5$
	0.6	$23.6$	$126.6$	$+1.2$	$+35.4$	$24.3$	$106.6$	$+20.1$	$+83.6$
	0.7	$43.4$	$207.5$	$+6.0$	$+35.9$	$38.3$	$165.0$	$+30.2$	$+96.7$
	0.8	$94.0$	$342.9$	$+22.2$	$+38.2$	$74.6$	$249.1$	$+56.0$	$+75.4$

Appendix

Table 11: Precision sweep (dev, exploratory; fact-only top-

b

plans with the fixed out-of-fold GB fact-scalar scores). Top: all correct facts are kept and wrong facts are subsampled at random until precision is P (natural precision 0.421 (Qwen) and 0.307 (GPT-4.1)); means over 50 draws. This simulates an extractor that makes fewer errors that are as easy to rank; the models are not refitted. Bottom: natural subsets, the facts whose A0 fact-scalar

P(\text{error})

is at most its

q

-quantile; P is given for Qwen / GPT-4.1. At the natural precision the same quantities are 10 / 58 and

-1

+16

(Qwen) and 13 / 33 and

+9

+26

(GPT-4.1) (Table 10 ).

	Qwen extr.	GPT-4.1 extr.
Errors removed (re-planned 95%), $b=0.1$ / $0.2$ / $0.3$
A2 $-$ A0 ‡	$-4$ / $+8$ / $+25$ ∗	$+6$ / $+23$ ∗ / $+66$ ∗
A2 $-$ A1 ‡	$+4$ / $+2$ / $-8$	$-2$ / $-4$ / $-5$
A2 $-$ A1x2p (calls) ‡	$+7$ / $+14$ / $+1$	$+1$ / $-2$ / $-1$
A2tok $-$ A1x2p (tokens) ‡	$+6$ / $+10$ / $-10$	$+4$ / $+3$ / $-4$
A1: oracle gap closed (%)	$-9$ / $+5$ / $+15$ ∗	$+12$ / $+31$ ∗ / $+51$ ∗

Appendix

Table 12: Verification controls. A2 augments baseline features with relation and evidence menus ( ‡ : exploratory, dev-wide menu statistics). A1x2p adds a reworded generic check with the same call count; A2tok matches its token budget. The oracle gap uses fact+entity actions. On GPT-4.1 extractions, the Qwen verifier’s repeated response is deterministic. A cross-family kimi-k2.5 check reproduces the difference between extractions in generic-verifier AUROC (difference in differences

-0.014

[

-0.043

+0.010

]).

Extr.	model	$b=0.1$	$b=0.2$	$b=0.3$	$b=0.5$
Qwen	A0 fact scalar	$+0.5$ $[-2.3,+3.7]$	$+0.8$ $[-2.6,+4.7]$	$+0.6$ $[-3.4,+5.2]$	$-0.7$ $[-7.1,+7.0]$
Qwen	A0 lattice	$-0.8$ $[-3.4,+2.3]$	$-1.1$ $[-4.3,+2.6]$	$-1.9$ $[-5.8,+2.6]$	$-6.4$ $[-12.7,+0.8]$
Qwen	A1 lattice	$-1.3$ $[-3.8,+1.5]$	$-1.6$ $[-4.8,+1.9]$	$-2.4$ $[-6.2,+2.0]$	$-7.7$ $[-13.8,-0.5]$
GPT-4.1	A0 fact scalar	$+0.7$ $[-1.4,+2.9]$	$+1.1$ $[-1.4,+3.7]$	$+0.9$ $[-1.9,+4.1]$	$+0.3$ $[-3.8,+4.9]$
GPT-4.1	A1 lattice	$+0.1$ $[-1.6,+1.7]$	$+0.2$ $[-1.9,+2.1]$	$+0.2$ $[-2.5,+2.7]$	$-1.3$ $[-5.8,+2.9]$

Appendix

Table 13: Exploratory check of the naive post-audit report (sum of

P(\text{error})

over unreviewed facts) under a fact-only top-

b

rule with out-of-fold GB predictions: relative bias in % of the true remaining errors, 95% document-cluster bootstrap (plans fixed). The top-

b

cutoff couples facts, so Theorem 4 (c) does not strictly apply; the check is descriptive. Entity (MILP) plans were not checked.

Extraction (verifier)	Quantity	$b=0.1$ (gate)	$b=0.2$ (gate)	$b=0.3$
Qwen extraction (verifier GPT-4.1-mini)	A0 repair yield	99.8% (867/869)	97.6% (1,697/1,739)	94.8% (2,472/2,608)
	A0 fact-only gap	$+2$ $[-10,+14]$	$+42$ $[+22,+65]$ ∗	$+136$ $[+99,+171]$ ∗
	A1 $-$ A0: errors removed	$-8$ $[-23,+6]$	$+6$ $[-9,+21]$	$+33$ $[+11,+53]$ ∗
	share of fact-only gap	$-400$ % $[-937,+63]$	$+14$ % $[-29,+40]$	$+24$ % $[+9,+38]$ ∗
GPT-4.1 extraction (verifier Qwen3.5-27B)	A0 repair yield	99.1% (836/844)	98.2% (1,658/1,688)	96.8% (2,450/2,532)
	A0 fact-only gap	$+8$ $[+0,+18]$	$+30$ $[+16,+42]$ ∗	$+82$ $[+63,+111]$ ∗

Appendix

Table 14: Development verification results for the lattice GB model with fact and entity actions (lenient labels, 500 documents). Qwen has 8,696 verifiable facts and 5,039 errors; GPT-4.1 has 8,440 facts and 5,850 errors. A0 uses extraction features; A1 adds a generic support check. Repair yield is errors removed per budget unit, and the fact-only gap compares A0 with the clairvoyant fact-review plan. Brackets give 95% re-planned bootstrap intervals (

R=100

); ∗ marks intervals excluding zero.

Panel of Figure 3 — Figure 3: Experiment A at the three matrix budgets (lattice GB model with fact and entity reviews). Light bars: A0’s fact-only reference gap, the errors that a clairvoyant fact-review plan removes beyond A0. Dark bars: the errors A1 removes relative to A0. Bars show 95% re-planned intervals.

Panel of Figure 4 — Figure 4: Errors that a fact-review ranking misses relative to the clairvoyant fact-review plan, over the full budget range (fact-only top- $b$ plans; exploratory). Dotted line: the share of wrong facts. A0 and A1 are GB models fitted on dev labels; the other two rankers use no labels.

Set	Extractor	Extract	Verify	Link
test	Qwen	500 (4)	8563 (30)	496 (0)
	GPT-4.1	500 (1)	8765 (0)	499 (0)
train500	Qwen	500 (0)	8625 (32)	500 (0)
	GPT-4.1	500 (0)	8513 (0)	500 (0)

Appendix

Table 15: Saved logical request records, with failures in parentheses. Extraction and linking are per document, verification per verifiable fact. Failed verification retains baseline features. Transport retries, cache hits and Wikipedia resolution are excluded from this accounting.

		test		train500
		Qwen	GPT	Qwen	GPT
H2, $b=0.1$	lenient	$-2$	$+3$	$+2$	$+9$
H2, $b=0.5$	lenient	$+50$	$+124$	$+41$	$+109$
H2, $b=0.2$	strict	$-6$	$+25$	$+32$	$+26$
H2, $b=0.3$	strict	$+9$	$+57$	$+51$	$+46$
H1 within	strict	$+11$	$-7$	$+5$	$-11$

Appendix

Table 16: Within-document secondary results (point estimates in errors;

b=0.2

unless noted). H1 is lattice minus independent-block residual error; H2 is baseline minus verified residual error. Entity value is errors removed by adding entity reviews to fact reviews, for the GB lattice planner and for the oracle.

Set	Extractor	$R_{N}-R_{F}$	$R_{F}-R_{J}$	Gain J/O
		(pp)	(pp)	(errors)
test	Qwen	3.38	0.91	60/387
	GPT-4.1	3.45	0.95	59/296
train500	Qwen	2.52	1.46	88/366
	GPT-4.1	3.44	1.31	82/307

Appendix

Table 17: Descriptive cross-document comparisons from saved plans (lenient labels, GB,

b=0.2

R_{N},R_{F},R_{J}

are residual benchmark errors for the node queue with full actions, the fact-scalar model with fact reviews, and the joint model with full actions. Before rounding, the two percentage-point columns sum to the H3 point-estimate gap, using the same initial-error denominator. Gain J/O is errors removed by opening the full shared-action menu relative to each planner’s own fact-only plan, for the joint model / computed oracle. These are descriptive point estimates; the H3 intervals and Holm decisions are in Table 3 .

	test		train500
Contrast	$\delta^{*}$	/ oracle	$\delta^{*}$	/ oracle
Within, Qwen	35	0.36	12	0.12
Within, GPT	14	0.21	13	0.14
Across, Qwen	0.72	0.11	0.60	0.10
Across, GPT	0.48	0.11	0.25	0.05

Appendix

Table 18: Detectable equivalence margins:

\delta^{*}

is the largest absolute endpoint of the 90% interval, before Holm correction. Within-document units are errors; across-document units are percentage points. Ratios use the computed oracle value of adding shared actions at

b=0.2

(including the recorded cross-document solver approximation). These sensitivity measures use unadjusted intervals.

Population	Extractor	Wrong	$\Delta^{\mathrm{cl}}$	Unverif.
SciREX dev	GPT-4.1	42	0	85.0%
	Qwen	38	0	83.2%
SciREX train	GPT-4.1	163	4	81.6%
	Qwen	154	1	74.4%
DWIE within	GPT-4.1	2,726	0	19.0%
	Qwen	2,519	0	10.8%

Appendix

Table 19: Development screens under lenient labels. Wrong and

\Delta^{\mathrm{cl}}

are counts conditional on verifiable predictions; the last column is the fraction of all extracted predictions excluded as unverifiable. Cross-document

\Delta^{\mathrm{cl}}

is a structural screen, subject to the action caveat discussed in the text.

Knowledge Graphs

Knowledge graphs represent entities and relationships as structured, queryable networks for integrating data and supporting reasoning.

See all 245 papers

From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing
Oct 26, 2026Harine Choi, Chenchen Kuai, Zihao Li +5LLM Evaluation LLM Auditing
Texas A&M Transportation Institute, Texas A&M University System · Zachry Department of Civil and Environmental Engineering, Texas A&M University · Department of Civil, Construction and Environmental Engineering, Marquette University +2
Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence
Oct 15, 2026Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe +1Confidence Estimation in Language Models Language Model Calibration
MBZUAI · TCS Research Labs, India · IIT Delhi
One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
Jul 30, 2026Cesare Zavattari, Alessandro Tommasi, Giuseppe PrencipeScalable Oversight AI Agent Reliability
Dipartimento di Informatica, Università di Pisa, Pisa, Italy

Oct 26, 2026J/K move · Enter open · S save

From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing

Harine Choi, Chenchen Kuai, Zihao Li, Jiwan Jiang+4

Texas A&M Transportation Institute, Texas A&M University System · Zachry Department of Civil and Environmental Engineering, Texas A&M University · Department of Civil, Construction and Environmental Engineering, Marquette University+2

LLM Evaluation LLM Auditing Human-in-the-Loop Evaluation Automated Evaluation

Open paper AskPDF arXiv

For large language models (LLMs) and multimodal large language models (MLLMs), model-reported self-confidence is often miscalibrated, yet it may still be useful for efficient evaluation. At the same time, LLM-as-a-judge offers a scalable alternative to direct human auditing, but its practical benefit depends on calibration quality and inference cost. This paper develops a unified statistical framework for both evaluation routes under alternative sampling strategies. For direct auditing, confidence-stratified sampling is used to recover population accuracy from partially verified outputs without requiring calibrated confidence. Finite-sample uncertainty is characterized, and the analysis shows how highly uneven confidence strata affect audit efficiency and label allocation. For judge-assisted evaluation, the judge is modeled through sensitivity and false-positive rate, yielding an error-corrected estimator with calibration uncertainty. The two routes are then compared through human-label and total-cost efficiency boundaries. Experiments on MMLU-Pro and MMMU-Pro show that miscalibrated self-confidence can still support statistically valid stratified auditing. Under highly imbalanced confidence distributions, proportional allocation substantially reduces estimation error relative to equal-per-stratum allocation. For judge-assisted evaluation, only a small subset of target--verifier pairs satisfies the idealized human-label efficiency condition, and automated inference costs can further limit potential savings. These findings clarify the statistical and economic conditions under which confidence-guided auditing and automated verification are useful.

Abstract

Knowledge Graphs

Explore similar work

From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence

One Human, NNN Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence