Risk Is Not Review Value: Wrong-Answer Exposure Under Bounded Review Budgets

SangJin Park; Myungsub Choi; Jineok Kim; Minseung Kang

cs.AISep 7, 2026

Risk Is Not Review Value: Wrong-Answer Exposure Under Bounded Review Budgets

Authors: SangJin Park, Myungsub Choi, Jineok Kim, Minseung Kang

Organizations: Tynapse Seoul, Republic of Korea

Abstract

LLM assistants often produce more answers than humans can review before users see them. Most evaluations ask whether an answer is wrong, unsupported, or low-confidence. Bounded review budgets instead ask which answers should be checked first under a fixed review budget. Risk alone is not enough: a high-risk answer may be hard to repair, while a moderately risky answer may be directly correctable from available evidence. For generated-answer evaluation, we model review prioritization as exposure reduction, where review value combines estimated wrongness, intervention affordance, impact, and cost. We evaluate review queues with Wrong-Answer Exposure Ratio (WAER), the fraction of wrong answers left unreviewed, and post-repair residual exposure (PRRE), the fraction still exposed after deterministic benchmark-supported repairs. PRRE uses repairability rules that do not numerically reuse the affordance scores used for ranking. On a 720-item TAT-QA/SciFact stress benchmark, review-value ranking keeps answer-level WAER nearly unchanged at 20% budget (0.605 vs. 0.600) but lowers PRRE from 0.881 to 0.716. These results show that trustworthy LLM evaluation should measure not only error detection, but also how limited review capacity reduces exposed wrong answers.

Figures & tables

Figure 1. Pairwise inversion under a bounded review budget. A risk-only queue selects the answer with the highest estimated wrongness, while a review-value queue can prefer a moderate-risk, high-affordance error over a high-risk, low-affordance one. With the same risk model and budget, the two objectives can leave different residual exposure below the cutoff. Two ranked review queues compare risk-only and review-value selection. The review-value queue promotes a moderately risky but directly repairable answer above a higher-risk answer with low intervention affordance, leaving less repairable exposure below the review cutoff.

Error type	$g_{i}$	$d_{i}$	$c_{i}$	$h_{i}$
Numeric perturbation	1.0	1.0	1.00	1
Direction flip	.9	.9	.81	1
Unsupported addition	.7	.6	.42	2
Scope distortion	.3	.3	.09	2
Conclusion mismatch	.5	.4	.20	3

Table 1. Benchmark-defined intervention and impact priors.

g_{i}

denotes evidence availability,

d_{i}

denotes action determinacy,

c_{i}=g_{i}d_{i}

, and

h_{i}

is an error-type impact prior. Current pilots set

q_{i}=1

Strategy	WAER	PRRE
Random	.800 [.800,.800]	.911 [.911,.912]
Risk-only proxy	.605 [.604,.607]	.881 [.877,.885]
Review-value proxy	.600 [.600,.600]	.716 [.713,.719]
Gold-factor RV	.600 [.600,.600]	.656 [.654,.659]
Repair-count oracle	.600 [.600,.600]	.600 [.600,.600]

Table 2. Public-data stress result at 20% review capacity, mean over 32 outer seeds. Brackets show 95% bootstrap confidence intervals for the 32-seed mean and reflect outer-seed variation only. Lower is better.

Metric	$\Delta$ [95% interval]
WAER	-.005 [-.007,-.004]
WDE	.119 [.114,.122]
RVE	-.116 [-.120,-.112]
PRRE	-.165 [-.168,-.160]

Table 3. Paired, dataset-stratified source-fact cluster bootstrap of the 32-seed mean at 20% review capacity. Policies are reranked within each of 1,000 resamples; deltas are review-value minus risk-only, and negative values favor review-value.

Queue objective	WAER	PRRE
Verifier risk $r_{i}$	.600	.767
$r_{i}c_{i}$	.600	.600
$r_{i}c_{i}h_{i}$	.600	.667
Gold-factor RV	.600	.656

Table 4. Same-risk verifier ablation at 20% review capacity. The first three rows share verifier risk

r_{i}

; Gold-factor RV is an evaluation-only reference. Lower is better.

Error type	$c_{i}$	WAER	$\rho_{i}/n$
Numeric perturbation	1.00	.067	60/60
Direction flip	.81	.500	60/60
Unsupported addition	.42	.783	32/60
Conclusion mismatch	.20	.625	0/120
Scope distortion	.09	1.000	0/60

Table 5. Base-seed error-type outcomes for the review-value proxy at 20% budget.

c_{i}

is the ranker affordance prior;

\rho_{i}/n

reports actual outcome-side deterministic repairability. Lower WAER is better.

Strategy (PRRE)	5%	10%	20%	40%
Risk-only proxy	.989	.962	.881	.704
Review-value proxy	.903	.828	.716	.556
Gold-factor RV	.900	.811	.656	.556
Repair-count oracle	.900	.800	.600	.556

Table 6. PRRE over review budgets, mean over 32 seeds. Review-value ranking lowers post-repair residual exposure at every budget; reference rows are evaluation-only. Lower is better.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Policy	5%	10%	20%	40%
Random	.950/.978	.900/.956	.800/.911	.600/.823
Self-conf.	.949/.951	.900/.905	.793/.823	.590/.718
Rule/surface	.944/.944	.893/.893	.794/.813	.591/.615
Explicit	.900/.900	.800/.800	.600/.600	.438/.556
Scope-aware	.900/.900	.800/.833	.600/.767	.223/.556
Same-risk $r_{i}c_{i}$	.900/.900	.800/.800	.600/.600	.389/.556

Appendix

Table A. Complete deterministic ladder over budgets (32-seed mean). Each cell reports WAER/PRRE; lower is better.

Metric	Estimate	Point [95% interval]
WAER	Risk-only	.605 [.604,.607]
	Review-value	.600 [.600,.600]
	$\Delta$	-.005 [-.007,-.004]
WDE	Risk-only	.549 [.547,.553]
	Review-value	.668 [.664,.671]
	$\Delta$	.119 [.114,.122]

Appendix

Table B. Paired, dataset-stratified source-fact bootstrap of the 32-seed mean at 20% budget (1,000 resamples). Queues are reranked per resample; negative

\Delta

favors review-value.

Strategy	5%	10%	20%	40%
Random	.949/.978	.898/.953	.803/.909	.603/.816
Self-conf.	.930/.950	.890/.920	.760/.830	.540/.720
Rule/surface	.950/.970	.920/.940	.810/.840	.570/.600
Explicit	.940/.940	.890/.890	.750/.750	.450/.540
Strong verifier	.900/.910	.810/.850	.640/.750	.410/.660
Rule $\rightarrow$ verifier	.900/.900	.800/.820	.670/.720	.420/.630

Appendix

Table C. LLM-assisted verifier pilot over budgets on 200 public-data-derived items. Each cell reports WAER/PRRE; lower is better.

Queue objective	WAER	PRRE	WDE
LLM $r_{i}$	.640	.750	.688
$r_{i}c_{i}$	.630	.640	.774
$r_{i}c_{i}h_{i}$	.600	.720	.643
Gold-factor RV	.600	.640	.719

Appendix

Table D. LLM-assisted same-risk ablation at 20% budget on the 200-item pilot; lower is better.

Error type	WAER	$n$
Conclusion mismatch	.697	33
Direction flip	.824	17
Numeric perturbation	.000	17
Scope distortion	1.000	17
Unsupported addition	.625	16

Appendix

Table E. LLM-assisted strong-verifier error-type breakdown at 20% budget; lower WAER is better.

	WAER		PRRE
Wrong rate	risk	review	risk	review
5%	.155	.156	.632	.551
10%	.184	.168	.651	.541
15%	.243	.195	.681	.555
20%	.280	.212	.707	.547
25%	.327	.229	.737	.562

Appendix

Table F. Wrong-rate sweep at 20% budget, 32-seed mean; lower is better.

$\delta$	risk-only PRRE	review-value PRRE
-0.2	.881	.604
-0.1	.881	.652
0.0	.881	.716
0.1	.881	.802
0.2	.881	.888

Appendix

Table G. Prior sensitivity at 20% budget; lower PRRE is better.

Radius	Metric	Win	$\Delta$ [range]
$\pm.1$	RVE	100.0%	-.125 [-.238,-.059]
	PRRE	100.0%	-.161 [-.262,-.045]
$\pm.2$	RVE	100.0%	-.151 [-.331,-.054]
	PRRE	94.3%	-.152 [-.281,.017]
$\pm.3$	RVE	100.0%	-.192 [-.431,-.049]
	PRRE	84.3%	-.149 [-.281,.048]

Appendix

Table H. Random prior sensitivity at 20% budget. Deltas are review-value minus risk-only; negative favors review-value.

Check	Risk PRRE	Review PRRE	$\Delta$
Original templates	.881	.716	-.165
Light paraphrase	.852	.716	-.136
Cue-stripped paraphrase	.798	.721	-.077
Count budget	.881	.716	-.165
Surface-cost budget	.877	.722	-.155

Appendix

Table I. Template-cue and cost-proxy sanity checks at 20% budget. Paraphrase rows report PRRE after deterministic surface rewrites. Cost rows compare count-budget and surface-cost-budget selection. Lower PRRE is better.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
Aug 18, 2026Sher Badshah, Ali Emami, Hassan SajjadLLM-as-a-Judge Selective Prediction
Dalhousie University · New York University Abu Dhabi · Emory University
Review Rarely Beats the Stronger Model: Separating Verification from Capability Transfer in LLM Pipelines
Oct 11, 2026Yirui Zhang, Yash Raj ShresthaLLM Answer Verification LLM Evaluation
University of Lausanne
Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence
Oct 15, 2026Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe +1Confidence Estimation in Language Models Language Model Calibration
MBZUAI · TCS Research Labs, India · IIT Delhi

Aug 18, 2026J/K move · Enter open · S save

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

Sher Badshah, Ali Emami, Hassan Sajjad

Dalhousie University · New York University Abu Dhabi · Emory University

LLM-as-a-Judge Selective Prediction LLM Uncertainty Estimation FDR Control

Open paper AskPDF arXiv

Using LLMs as judges has become standard practice for evaluating model outputs at scale. This is particularly common for subjective, open-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists. However, objective tasks introduce a distinct reliability challenge for reference-free LLM judging. In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation. Although the former enables efficient evaluation, the judge may hallucinate or lack sufficient evidence for its verdict. Conversely, tool augmentation can provide additional evidence but introduces extra computational cost and requires an appropriate mechanism to determine when and how that evidence should be used reliably. More importantly, neither approach alone provides formal control over the risk of accepted verdicts or guarantees their reliability at a specified level. We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level~

α

with high probability, using finite-sample Clopper--Pearson intervals. When the parametric mode is not sufficiently confident, the instance is routed to a retrieval-augmented mode, where the judge gathers web evidence and re-evaluates the instance under a second calibrated threshold. The finite-sample guarantee carries over to this two-threshold routing without additional assumptions. Across open-domain QA benchmarks and judges of varying scales, the framework maintains the target error rate while achieving substantially higher coverage than single-mode baselines.

Abstract

LLM Evaluation

Explore similar work

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

Review Rarely Beats the Stronger Model: Separating Verification from Capability Transfer in LLM Pipelines

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence