cs.AIOct 13, 2026

One-Sided Verification Under Limited Expert Review

Authors: Fan Nie, Haotian Ye, Shurui Liu, Zihao Wang, Stefano Ermon, James Zou

Organizations: Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI has become increasingly helpful in scientific discovery, yet human expert review remains essential for those problems without reliable automated evaluation, posing a fundamental challenge as candidate generation scales beyond available review capacity. This paper studies verification under limited human effort, where rare valid solutions need to be prioritized while invalid candidates should be filtered out to reduce human workload. We adopt the one-sided verification objective that maximizes the rejection rate of invalid solutions (TNR) under a constraint on the false-rejection rate of valid ones (FNR), making human review cost an explicit term to minimize. Drawing on the intuition that verifying a specific error can be easier than discovering one, we introduce OSVerify, an algorithmic framework that decouples error proposal from error validation. The error proposer searches for errors in solutions across different granularities by hierarchical decomposition, while the validator distinguishes substantial errors from minor or hallucinated ones. Experiments on competition- and research-level benchmarks show that OSVerify reduces the number of human reviews of invalid solutions by 46% on average compared to voting baselines under the same empirical low FNR constraint. Our post-training of the validator further improves the reduction to 57%. As high-impact real-world use cases, we apply OSVerify to identify errors in published research papers that are subsequently confirmed by the original authors and help resolve three Langlands open conjectures that are confirmed by experts, demonstrating its utility to facilitate human-AI research collaboration.

Figures & tables

Figure 1: One-sided verification for open-ended scientific discovery. (a) AI-generated candidates can outpace limited expert review capacity. (b) OSVerify decouples verification into hierarchical error proposal and repeated error validation, converting solution verification to error validation. (c) Under empirical 5% FNR constraint, OSVerify reduces the invalid candidates requiring review by 52% on Putnam100 and 34% on FirstProof-Eval, and the trained variant raises the reduction to 83% and 41%.
Figure 2: (a) Frequencies of LLM Reject on Putnam100 with 64 samples per (x,y)(x,y) ; the dashed line marks empirical FNR below 5%. (b) The TNR increases and saturates as the number of judgments nn increases, regardless of the FNR constraint. (c) LLM judgments on the error-level distribution (x,y,e)(x,y,e) achieve higher TNR at matched FNR than judgments on the solution-level distribution (x,y)(x,y) , particularly in the low-FNR regime. All experiments use gpt-oss-120b .
 Putnam100H2VFirstProof-Eval
Method0%5%10%0%5%10%0%5%10%
gpt-oss-120b
LLM-as-a-Judge\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{54.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.0}}\makebox[20.00003pt][r]{54.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{43.3}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.9}}
Voting\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{38.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.9}}\makebox[20.00003pt][r]{45.5}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.8}}\makebox[20.00003pt][r]{60.9}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 4.2}}\makebox[20.00003pt][r]{69.2}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.6}}\makebox[20.00003pt][r]{75.6}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.7}}\makebox[20.00003pt][r]{29.5}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 10.2}}\makebox[20.00003pt][r]{35.9}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.5}}\makebox[20.00003pt][r]{48.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.2}}
OSVerify\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{70.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 14.3}}\makebox[20.00003pt][r]{77.9}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.3}}\makebox[20.00003pt][r]{83.2}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0.6}}\makebox[20.00003pt][r]{84.7}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0.6}}\makebox[20.00003pt][r]{84.7}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0.5}}\makebox[20.00003pt][r]{50.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 15.1}}\makebox[20.00003pt][r]{58.0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.1}}\makebox[20.00003pt][r]{60.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.3}}
+ trained validator\makebox[20.00003pt][r]{85.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.9}}\makebox[20.00003pt][r]{89.6}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.7}}\makebox[20.00003pt][r]{92.3}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.5}}\makebox[20.00003pt][r]{82.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.1}}\makebox[20.00003pt][r]{83.2}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0.7}}\makebox[20.00003pt][r]{83.2}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0.7}}\makebox[20.00003pt][r]{57.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.0}}\makebox[20.00003pt][r]{62.3}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0.8}}\makebox[20.00003pt][r]{62.7}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0.3}}
Table 1: Empirical TNR (%) under empirical FNR constraints of 0%, 5%, and 10%. Entries report means and standard deviations over 2,000 resampling trials. Zero includes the always-pass fallback when no nontrivial operating point satisfies the constraint.
Figure 3: (a) Deeper decomposition raises TNR for training-free and trained validators, with most of the gain reached by depth two. (b) More validation votes raise TNR on fixed proposal pools; the stricter the FNR constraint, the more votes it takes.
Figure 4: (a) Training rewards for proposed errors to overturn or confirm, with EMA decay 0.9. (b) Empirical TNR under different input-plus-output token budgets on FirstProof-Eval. (c) Applying OSVerify to existing verifiers Multiagent Debate and PFBV on H2V.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

 Putnam100H2VFirstProof-Eval
Method0%5%10%0%5%10%0%5%10%
gpt-oss-120b
LLM-as-a-Judge\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{54.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.0}}\makebox[20.00003pt][r]{54.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{43.3}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.9}}
Pessimistic voting\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{1.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 11.8}}\makebox[20.00003pt][r]{45.6}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 38.8}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}
Optimistic voting\makebox[20.00003pt][r]{0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0}}\makebox[20.00003pt][r]{19.7}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.6}}\makebox[20.00003pt][r]{19.7}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.6}}\makebox[20.00003pt][r]{33.3}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.4}}\makebox[20.00003pt][r]{33.3}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.4}}\makebox[20.00003pt][r]{33.3}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.4}}\makebox[20.00003pt][r]{26.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 9.1}}\makebox[20.00003pt][r]{29.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.9}}\makebox[20.00003pt][r]{29.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 1.7}}
Voting\makebox[20.00003pt][r]{0.0}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 0.0}}\makebox[20.00003pt][r]{38.4}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.9}}\makebox[20.00003pt][r]{45.5}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.8}}\makebox[20.00003pt][r]{60.9}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 4.2}}\makebox[20.00003pt][r]{69.2}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.6}}\makebox[20.00003pt][r]{75.6}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 3.7}}\makebox[20.00003pt][r]{29.5}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 10.2}}\makebox[20.00003pt][r]{35.9}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.5}}\makebox[20.00003pt][r]{48.8}_{\makebox[18.33618pt][l]{\scriptscriptstyle\pm 2.2}}
Appendix
Table 2: Additional training-free comparisons. Empirical TNR (%) under empirical FNR constraints of 0%, 5%, and 10%, reported as mean and standard deviation over 2,000 resampling trials. LLM-as-a-Judge , Voting , and OSVerify entries are reproduced from Table 1 . Pessimistic and optimistic voting use 16 judgments drawn without replacement from each recorded 64-judgment pool; an infeasible rule is replaced by the always-pass rule within each trial.
Figure 5: Approximate inference cost comparison. OSVerify never falls below Voting at any displayed budget; arrows mark its gain at 128k tokens per solution. Each point reports the best empirical TNR attainable within an input-plus-output token budget. Voting varies from 1 to 64 judgments; OSVerify jointly varies the proposal budget and 1–16 validation votes per proposed error. The displayed range starts at 30k tokens per solution, and curves omit leading zero-TNR operating points that accept every solution. Lines connect evaluated budget points; vertical axes vary across panels.
ComponentRequestsInput tokensOutput tokensTotal tokensTokens/solution
OSVerify : decomposition4.51K7.08M3.26M10.34M51.7K
OSVerify : proposer1.19K6.47M2.28M8.75M43.7K
OSVerify : validator ( n=16n=16 )6723.11M15.22M†15.22\mathrm{M}^{\dagger}18.33M†18.33\mathrm{M}^{\dagger}91.6K
Appendix
Table 3: Component-wise H2V inference cost with oss , projected to 16 validator judgments per proposed error. Decomposition and proposer usage are provider-reported; validator inputs are reconstructed with the model tokenizer. Outputs marked †\dagger are estimated from previously sampled completion lengths. Requests include every internal decomposition request and one batched-equivalent validation request per proposed error. K=103K=10^{3} and M=106M=10^{6} .
Figure 6: Scaling the whole-solution Voting baseline to 64 judgments. Each curve reports the best empirical TNR at the FNR constraint of 0%, 5%, or 10%, using the first nn judgments per solution. No subset resampling or smoothing is applied.
Target FNRMethodReject if scoreTest FNRTest TNR
5%Voting>16>160.0% (0/50)0.0% (0/58)
 OSVerify≥16\geq 164.0% (2/50)77.6% (45/58)
10%Voting≥15\geq 152.0% (1/50)29.3% (17/58)
 OSVerify≥15\geq 156.0% (3/50)81.0% (47/58)
Appendix
Table 4: Diagnostic comparison using calibration-selected thresholds on Putnam. All scores use 16 votes.
Figure 7: Auditing proposed errors for validator training. We compare the full training setup with a variant that omits auditing proposed errors from invalid solutions. Bars show mean TNR under a 5% empirical FNR constraint; error bars show standard deviations. The vertical axis starts at 40%.
BenchmarkInvalidValidTotal
Putnam1005850108
H2V16337200
FirstProof-Eval593594
Appendix
Table 5: Evaluation datasets.
SourceNegativePositiveTotal
Putnam (training split)7742701,044
IMO-GradingBench71165236
ProofBench45223268
Open Proof Corpus13126139
Total9037841,687
Appendix
Table 6: Composition of the online GRPO training pool. Negative examples are proposed errors on valid solutions; positive examples are substantial proposed errors on invalid solutions.
ModelProviderReasoning effortTemperature
ossTogether AIMedium0.7
5miniOpenAIMedium1.0
oss + LoRA (trained validator)TinkerMedium0.7
Appendix
Table 7: Backbones and inference settings. 5mini only supports its default temperature of 1. The trained validator is oss with the LoRA adapter described in Appendix D.5 ; it validates the proposed errors of the training-free oss proposer.
SettingValue
Base modeloss , medium reasoning effort
AdapterLoRA, rank 32
OptimizerAdam, β1=0.9\beta_{1}=0.9 , β2=0.95\beta_{2}=0.95 , ϵ=10−8\epsilon=10^{-8}
Learning rate10−510^{-5} , linear warmup for 10 steps, then constant
Examples per step32
Rollouts per example8
Appendix
Table 8: Online GRPO hyperparameters.
More in this topic

Reasoning Verification

Reasoning verification checks whether intermediate steps and final answers are logically valid and supported by evidence.

See all 57 papers

Explore similar work

CardsList
Jun 26, 2026

Towards Automating Scientific Review with Google's Paper Assistant Tool

Artificial intelligence is driving a revolution in scientific discovery, accelerating everything from hypothesis generation to mathematical theorem proving. However, this rapid acceleration is creating a systemic challenge: traditional human peer review cannot scale to match the influx of AI-assisted science. Ultimately, to resolve this tension, we must also deploy AI to accelerate the verification and review process itself. To frame the discussion around this transition, we propose a taxonomy consisting of four progressive levels of AI-human collaboration in scientific evaluation, and discuss various trade-offs involved with each. As a step toward this future, we introduce the Paper Assistant Tool (PAT), an agentic AI framework built for deep scientific review and verification. PAT ingests full scientific manuscripts and produces a comprehensive evaluation, checking theoretical results, validating experiments, suggesting improvements, and identifying potential flaws. By utilizing inference scaling techniques, PAT is able to identify deeper issues than a single model call alone, achieving a 34% improvement over zero-shot recall on mathematical errors in the SPOT benchmark. Pilot deployments of PAT as a pre-submission tool for authors at two major Computer Science conferences -- STOC and ICML -- demonstrate its ability to identify critical errors and suggest substantive improvements to research papers. By catching errors early, PAT eases the cognitive burden placed on referees, while preserving their control over the outcomes of the review process.

Rajesh Jayaram, Drew Tyler, David Woodruff +4

Google Research, USA · Google Research & Carnegie Mellon University, USA · Carnegie Mellon University, USA

Automated Peer ReviewHuman-AI Collaboration
AskPDF
May 25, 2026

ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence

Autonomous research agents produce competitive solutions and professional-looking manuscripts, yet their outputs contain verifiability failures undetectable by surface-level evaluation: fabricated citations, unreproducible scores, and method descriptions that diverge from the implementation. We address this through three contributions. First, Chain-of-Evidence (CoE), a verifiability framework requiring every claim to be traceable to its evidence source. Second, ScientistOne, an end-to-end autonomous research system that maintains evidence chains by construction throughout literature review, solution discovery, and paper writing. Third, CoE Audit, a post-hoc audit whose four integrity checks -- score verification, specification violation, reference verification, and method-code alignment -- apply uniformly to all systems. Across 75 papers spanning five systems and five frontier research tasks, every baseline exhibits at least one systematic failure mode: hallucinated reference rates reach 21%, score verification passes in as few as 42% of papers, and method-code alignment ranges from 20% to 80%. ScientistOne achieves zero hallucinated references (0/337), perfect score verification (12/12), and the highest method-code alignment (14/15), while matching or exceeding human expert performance on all five tasks. ScientistOne further generalizes to six additional tasks spanning medical imaging, fine-grained recognition, 3D perception, and language modeling, achieving state-of-the-art on Parameter Golf and gold medals on MLE-Bench tasks where baselines fail entirely.

Rui Meng, Bhavana Dalvi Mishra, Jiefeng Chen +10

Google Cloud AI Research

Data ProvenanceCitation VerificationEvidence-Grounded GenerationScientific ReproducibilityAutonomous Research AgentsScientific Claim Verification
AskPDF
Oct 22, 2026

Reliable AI-Assisted Decisions via Selective Auditing

We consider online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score estimating the trustworthiness of the decision. The decision and confidence score need not be produced by the same model: for example, one agent may propose a decision, while a separate verifier agent evaluates it and outputs a correctness or safety score. Based on this information, we must decide whether to accept the AI decision automatically or consult an expert, such as a human auditor, who reviews the proposal and either accepts or corrects it. We formulate this task as an online constrained optimization problem in which the goal is to minimize the frequency of expert auditing while guaranteeing that the fraction of AI errors that go unaudited stays below a user-defined threshold. We derive a family of algorithms with provable error-control guarantees with no assumptions, including a method that is optimally efficient: it asymptotically attains the best achievable auditing rate for both adversarial and i.i.d. data streams. Experiments on screening harmful requests to large language models, qualitative text annotation, and data labeling show that our methods achieve valid error control in practice while substantially reducing the need for human auditing compared to baselines.

Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani +1

Departments of Statistics and Management Science & Engineering, Stanford University

AI-Assisted Decision MakingHuman Oversight of AIAlgorithmic AuditingConstrained Optimization
AskPDF