AutoVerifier: An Agentic Automated Verification Framework Using Large Language Models

Yuntao Du; Minh Dinh; Kaiyuan Zhang; Ninghui Li

cs.AIApr 3, 2026

AutoVerifier: An Agentic Automated Verification Framework Using Large Language Models

Authors: Yuntao Du, Minh Dinh, Kaiyuan Zhang, Ninghui Li

Organizations: Purdue University

Abstract

Scientific and Technical Intelligence (S&TI) analysis requires verifying complex technical claims across rapidly growing literature, where existing approaches fail to bridge the verification gap between surface-level accuracy and deeper methodological validity. We present AutoVerifier, an LLM-based agentic framework that automates end-to-end verification of technical claims without requiring domain expertise. AutoVerifier decomposes every technical assertion into structured claim triples of the form (Subject, Predicate, Object), constructing knowledge graphs that enable structured reasoning across six progressively enriching layers: corpus construction and ingestion, entity and claim extraction, intra-document verification, cross-source verification, external signal corroboration, and final hypothesis matrix generation. We demonstrate AutoVerifier on a contested quantum computing claim, where the framework, operated by analysts with no quantum expertise, automatically identified overclaims and metric inconsistencies within the target paper, traced cross-source contradictions, uncovered undisclosed commercial conflicts of interest, and produced a final assessment. These results show that structured LLM verification can reliably evaluate the validity and maturity of emerging technologies, turning raw technical documents into traceable, evidence-backed intelligence assessments.

Figures & tables

Subject	Predicate	Object	Provenance	Verdict	Source
BF-DCQO	executed-on	IBM Heron (Marrakesh/Kingston)	Lv. 1 Exp.	Supports	Sec. IV, p.5
BF-DCQO	outperforms (time-to-result)	SA ( $>$ 3.5 $\times$ ) / CPLEX (up to 80 $\times$ )	Lv. 1 Exp.	Partial	Sec. V
BF-DCQO	projected-to-achieve	“several orders of magnitude”	Lv. 5 Assert.	Overclaim	Abstract, p.1
BF-DCQO	requires	classical SA pre- and post-processing	Lv. 1 Exp.	Supports	Sec. III.C, Fig. 4

Table 1: Examples of extracted claim triples with provenance classification and intra-document verification.

Metric	Quantum (BF-DCQO)	Classical (SA/CPLEX)	Comparable?
Runtime definition	$T_{\text{CPU}}+T_{\text{QPU}}$ (partial)	$T_{\text{sweep}}$ formula (partial)	✗
Included overhead	CPU pre-/post-processing, QPU gates	Sweep computation only	✗
Excluded overhead	Transpilation ( ${\sim}2\,$ s), readout, queuing	Initialization ( ${\sim}1.65\,$ s), solver setup	✗

Table 2: Normalized metric comparison: quantum vs. classical runtime definitions.

Issue	Severity	Claim	Evidence
Overgeneralization	Moderate	“achieving runtime quantum advantage” (Abstract)	Demonstrated only on “hardest instance out of a pool of 250” (Sec. IV.C)
Extreme-value reporting	Moderate	“up to a factor of 80” speedup (Sec. V)	Single maximum outlier; median enhancement ${\sim}5$ – $7\times$ across instances
Projection as result	Severe	“several orders of magnitude” (Abstract)	Prefaced with “we expect”; no supporting data or analysis in body
Scope inflation	Severe	“industrial-scale optimization problems” (Sec. V)	Only synthetic HUBO tested; $N{\leq}156$ ; no industrial problem formulated

Table 3: Overclaim detection: claims vs. supporting evidence in the target paper.

Dimension	Target Paper [ 6 ]	Rebuttal [ 31 ]	BF-Null Control [ 9 ]
Advantage claimed?	Yes	No	No (quantum component minimal)
QPU execution?	Confirmed	Confirmed (not disputed)	N/A (tests contribution, not execution)
Runtime definition	$T_{\text{CPU}}+T_{\text{QPU}}$ (partial)	End-to-end wall-clock	Full pipeline wall-clock
Classical baselines	SA (fixed sweeps), CPLEX (1 thread)	SBM, optimized SA	SA pre/post-processing only (QPU removed)
Independence	Low (commercially interested)	High (no affiliation)	Medium (competitor, but reproducible control)

Table 4: Cross-source contradiction: target paper vs. two independent evaluations.

Hypothesis	Evidence	Cross-Source	Confidence	Alternatives	Status
Runtime advantage is genuine	[ 6 , 5 ]	Contradicted [ 31 , 9 ]	High entropy	Measurement artifact	Likely Hallucination
QPU contributes to performance	[ 6 ]	Contradicted [ 9 ]	High entropy	Classical iteration suffices	Needs Review
Advantage is a measurement artifact	[ 31 , 7 ]	Supported	Low entropy	Hybrid accounting	Supported
Real but incremental progress	[ 6 , 31 , 7 ]	Consensus	Low entropy	Next-gen up-scaling	Supported

Table 5: Example hypothesis matrix for the runtime quantum advantage claim.

Claim Verification

Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.

See all 59 papers

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation
May 26, 2026Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari +1Selective Prediction Citation Verification
FirstPrinciples · School of Computing Science, Simon Fraser University
AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs
Apr 24, 2026Pouya Pezeshkpour, Estevam HruschkaLLM Evaluation LLM Answer Verification
Megagon Labs
ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning
Jul 17, 2026Binglin Zhou, Peng Shi, Ryo Kamoi +2Tool Use in VLMs Scientific Claim Verification
1The Pennsylvania State University · University of Waterloo

May 26, 2026J/K move · Enter open · S save

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation

Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari, Alexander Tessier

FirstPrinciples · School of Computing Science, Simon Fraser University

Selective Prediction Citation Verification Evidence Retrieval Scientific Claim Verification

Open paper AskPDF arXiv

Misalignment between claims and their cited evidence is a common failure mode in reports generated by large language models, limiting their reliability in scientific and other high-stakes settings. We present DeepSciVerify, a two-stage pipeline for scientific claim-citation verification that combines abstract-level reasoning with selective escalation to passage-level evidence. The system first verifies claims using the abstract and defers uncertain cases, retrieving and analyzing full-text passages only when necessary. This design leverages complementary behaviors across LLMs, as some models are more conservative while others are more decisive under uncertainty. On the SCitance benchmark, DeepSciVerify achieves 86.7 Micro-F1, outperforming strong abstract-only baselines by +4.5 points while resolving 67% of instances without full-text retrieval. These results suggest that selective evidence escalation improves both accuracy and efficiency in claim-citation verification.

Abstract

Claim Verification

Explore similar work

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation

AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs

ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning