PRIME: Towards Verifiable Evidence-Grounded Reasoning in LLM Agents

Hamed Khosravi; Xiaoming Huo

cs.AIOct 15, 2026

PRIME: Towards Verifiable Evidence-Grounded Reasoning in LLM Agents

Authors: Hamed Khosravi, Xiaoming Huo

Organizations: H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology

Abstract

LLM agents are used to check records against an organization's requirements, for example whether a vulnerability report is ready for investigation. Requirements often allow several ways to be met, such as a severity rating with a curator-reviewed advisory or a published advisory link. Agents fail by accepting a report after a required check has failed, or by settling for a weaker conclusion without trying another way that could still succeed. We introduce PRIME (Precommitted Rules for Inference from Minimal Evidence). Before checking, it fixes evidence routes specifying how each conclusion may be justified. A separate program records each check result and refuses any conclusion the results do not justify. It also points the agent to checks that could justify a stronger conclusion, or runs them itself. PRIME reports only conclusions authorized by its rules and ledger. With correct requirements, exact checks at known locations, and sufficient budget at stopping, completion reaches the strongest supported conclusion. On 480 public-record tasks, an agent shown the requirements reported unjustified conclusions on 26.7% of tasks, while PRIME reported none. Its directions and added checks raised the average score for justified conclusions from 30.1, with refusal alone, to the maximum of 59.2 out of 100.

Figures & tables

Panel of Figure 1 — Figure 1: PRIME in a hypothetical security-report review. (A) The independent runtime records check outcomes, uses precommitted evidence routes to guide the agent, and authorizes conclusions. (B) Acceptance requires a description plus either a severity rating and curator-reviewed advisory ( $p_{1}$ ) or a published advisory reference with a URL ( $p_{2}$ ). A failed curator-review check blocks $p_{1}$ , while the reference checks of $p_{2}$ remain open. PRIME recommends checking the reference but cannot yet authorize acceptance.

Approach	Controls	Next check
Answer verification, citation ( Dhuliawala et al., 2024 ; Gao et al., 2023 )	nothing; claims rechecked afterward	not chosen
Conformal factuality ( Mohri and Hashimoto, 2024 )	claims, at calibrated risk	not chosen
Action rules ( Wang et al., 2026 ; Doshi et al., 2026 )	agent actions	not chosen
Evidence-gated control ( Xia and Wang, 2026 ; Huang et al., 2026 ; Ren, 2026 )	claims or transitions	repair or stop
Evidence acquisition ( Zhang et al., 2026b ; Liu and Wang, 2026 )	stopping rule or verifier acceptance	uncertainty or information gain
PRIME	graded conclusions, every policy	open route of the strongest reachable conclusion

Table 1: What each family enforces and how it chooses further evidence. Among these families, only PRIME uses alternative routes both to authorize a conclusion and to choose the next check.

Agent	False auth. % $\downarrow$	Prem. % $\downarrow$	Against ledger $\downarrow$	Util. $\uparrow$	Emit % $\uparrow$
ReAct	13.0	56.9	6	17.9	59.0
+verify	19.9	47.6	3	10.0	49.6
+spec+verify	22.3	33.8	55	26.9	89.9
+program+verify	22.3	25.2	56	32.7	98.5
Fully informed agent	26.7	28.3	67	38.0	99.0
PRIME, no guidance	0.0	0.0	0	30.1	99.0

Table 2: Information versus enforcement (480 tasks). +verify adds an instruction to verify, +spec the specification view (each atom and its field), +program the routes, and the fully informed agent its ledger. Up to four incomplete episodes were excluded per row. Against ledger counts claims with a recorded failed check on every route. Prem. is premature reporting, and Emit is reporting among answerable tasks.

	Fully informed		FA removed	Util. with gate $\uparrow$
Model	FA % $\downarrow$	Util. $\uparrow$	[95% CI]	no guid.	PRIME
Qwen2.5-32B	26.7	38.0	28.0 [22.7, 33.5]	30.1	45.5
gpt-oss-20b	17.7	30.5	18.0 [13.5, 22.7]	26.9	58.5
Mistral-24B	19.4	35.2	19.5 [14.9, 24.2]	29.3	36.6
Qwen2.5-7B	0.4	1.0	0.5 [0.0, 2.1]	24.7	26.1

Table 3: Other models (480 tasks, 20 calls). Mistral-24B is Mistral-Small-24B. FA is false authorization, and FA removed is the family-mean reduction from the gate. With the gate, every model had 0% false authorization.

Figure 2: Supported utility by budget with known (left) or uncertain (right) evidence locations. Dashed curves denote scripted checking policies (Appendix C ). Right-panel completion replays logged episodes without model calls.

b_{\min}

is the cost of the cheapest committed route to the strongest conclusion before checking.

Agent	FA % $\downarrow$	Util. $\uparrow$	Adm. att.	Res. err. % $\downarrow$	Ep. err. % $\downarrow$
Fully informed agent	4.2	58.5	210	9.5	8.3
PRIME, no guidance	1.7	47.4	151	13.2	8.3
PRIME	4.6	57.5	298	10.1	11.7
PRIME, verified	0.8	55.1	222	2.7	2.5

Table 4: Text judgments (240 tasks). FA is false authorization, Adm. att. counts accepted judgments, Res. err. is the share of them that are wrong, and Ep. err. is the share of episodes with such an error.

System	False auth. % $\downarrow$	Prem. % $\downarrow$	Util. $\uparrow$	Emit % $\uparrow$	Reads $\downarrow$
Fully informed agent	26.7	28.3	38.0	99.0	3.1
AgentSpec ( 2026 )
stop	0.0	0.0	38.7	87.4	3.0
self-examine	0.0	0.0	40.1	92.9	3.1
ResearchLoop ( 2026 )	0.0	0.0	47.7	97.0	3.2
Proof-or-Stop ( 2026 )	0.0	0.0	52.0	99.2	4.2

Table 5: Adapted systems, sharing tools and requirements, on 480 tasks. Parentheses in Util. show scores supported by each agent’s recorded checks (HESP and PRIME with completion). Reads counts fields per task.

Policy	$n$	Unsupported % [95% CI] $\downarrow$	Util. $\uparrow$	Cov. % $\uparrow$
Assistant-built program (526 instances)
Ceiling-targeting	1,999	1.8 [1.3, 2.5]	18.4	26.3
Ceiling-targeting	600	1.7 [0.9, 3.0]	19.0	26.8
Fully informed agent	600	0.2 [0.0, 0.9]	13.1	26.8
PRIME	600	0.2 [0.0, 0.9]	9.9	26.8
PRIME, completion	600	1.7 [0.9, 3.0]	19.0	26.8

Table 6: Held-out regulatory evaluation. Cov. is the share of registrations with a stated conclusion. Tracker levels are 3 for late, 2 for due, and 1 otherwise. Scripted checks use 1,999 registrations, agents 600, and missing instances count as abstentions. Full-evidence agreement is 22.8% (assistant-built) and 81.6% (protocolized).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Mask violations	Conflict fail-open	Premature conclusion	Renderer strengthening	Retraction failure
PRIME runtime	0	0	0	0	0
Templates: 52 of 52 round trips exact; strengthening mutations: 9 of 9 rejected; slot injections: 8 of 8 blocked.
Commitment input	Share of the 70 pairs with different committed programs
Specification view only (production)	0.000
Plus each atom’s authored description	0.429
Plus the true values and the answer key	0.786

Appendix

Table 7: Runtime and renderer tests include 52 round trips, 9 mutations, 8 injections, and 27 renderer cases. Violation counts come from a scripted episode with 3 conclusions, 4 atoms, 1 conflict, and 1 retraction. Commitment tests compare 70 hidden-evidence pairs across 5 seeds and 14 perturbation types. Deliberately leaky inputs confirm detection.

Study	Question	Tasks or records	Agents and policies	What varies	Exhibits
Information vs. enforcement (§ 7.1 )	Does an agent shown the requirements and its ledger follow them?	Balanced split, 480 tasks	Qwen2.5-32B-Instruct; three other models	Information in the prompt; authorization gate	Table 2 ; Figure 3 ; Table 3
Guidance (§ 7.2 )	Does guidance recover the conclusions that enforcement alone misses?	Balanced split	Qwen2.5-32B-Instruct; scripted policies	Guidance, completion, budget	Figure 2 , left; Figures 4 and 5
Evidence location (§ 7.2 )	Does the agent’s choice of field beat fixed-order checking?	Locator-uncertain copy	Qwen2.5-32B-Instruct, gpt-oss-20b, Mistral-Small-24B; scripted policies	Field choice, completion, budget	Figure 2 , right; Table 11
Text judgments (§ 7.3 )	What happens when the model judges text?	Text-evidence split, 240 tasks	Qwen2.5-32B-Instruct	Guidance; verification of attestations	Table 4
Closest systems (§ 7.4 )	What does PRIME add over the closest systems given the same requirements?	Balanced split; locator-uncertain copy	AgentSpec, ResearchLoop, Proof-or-Stop, HESP, GLEAN; Qwen2.5-32B-Instruct	System; budget	Table 5 ; Tables 12 and 13
Regulation (App. F )	Can PRIME execute requirements generated from a published regulation?	1,999 held-out registrations; 600 for agents	Two generated programs; ceiling-targeting policy; Qwen2.5-32B-Instruct agents	Construction workflow; policy	Tables 14 to 16

Appendix

Table 8: Study overview for Section 7 and Appendices F – G . Pilot results are not pooled.

Atom	Field	Predicate	Value
$e_{1}$	details	exists	true
$e_{2}$	database_specific.	equals True-alt	false
	github_reviewed
$e_{3}$	database_specific.severity	exists	true
$e_{4}$	references.0.type	equals ADVISORY-alt	false
$e_{5}$	references.0.url	exists	true

Appendix

Table 9: Routes to

c_{1}

c_{2}

, and

c_{3}

are

\{e_{1}\}

\{e_{1},e_{2},e_{3}\}

\{e_{1},e_{4},e_{5}\}

, and

\{e_{1},e_{2},e_{3},e_{6},e_{7}\}

, respectively. Constructed failures at

e_{2}

and

e_{4}

leave

c_{1}

strongest.

Agent	What it sees or does	Enforced	Guided
ReAct	the request, scope, conclusions, and tools ( Yao et al., 2023 )	no	no
+verify	plus an instruction to verify before concluding	no	no
+spec	ReAct plus the specification view: each atom and the field it reads	no	no
+spec+verify	both	no	no
+program	+spec plus the committed routes	no	no
+program+verify	both	no	no

Appendix

Table 10: Section 7.1 information ladder. Agents share model, tools, budget, and scorer.

Figure 3: Information ladder on 480 tasks at 20 calls. Top shows false authorization and bottom supported utility. Annotations are family-mean paired differences with 95% intervals.

Figure 4: Guidance gains by budget and route structure (480 tasks). Points show family-mean paired utility differences with 95% intervals. Dashed line marks the minimum effect of 6.16.

Figure 5: Utility on 480 tasks at 20 calls, comparing direct answering, unguided PRIME, the ideal filter on the fully informed agent’s evidence, PRIME, and scripted ceiling targeting.

		FA (%)	Utility, balanced split			Guidance gain	Locator split, $b_{\min}$
Model	Agent	at 20	$b_{\min}$	$2b_{\min}$	20	at $b_{\min}$ [95% CI]	Utility	vs. blind [95% CI]
Qwen2.5-32B	Fully informed agent	26.7	38.1	38.6	38.0		37.2	$-$ 2.0 [ $-$ 4.7, 0.8]
	PRIME without guidance	0.0	30.1	30.1	30.1		22.0	$-$ 16.5 [ $-$ 19.9, $-$ 13.1]
	PRIME	0.0	52.6	47.3	45.5	23.0 [19.5, 26.3]	51.5	12.5 [8.9, 16.3]
	PRIME with completion	0.0	58.8	59.2	59.2		53.7	14.7 [11.5, 18.0]
gpt-oss-20b	Fully informed agent	17.7	30.6	30.5	30.5		31.5	$-$ 7.5 [ $-$ 9.7, $-$ 5.2]

Appendix

Table 11: Three-model results on balanced and locator-uncertain splits. FA is measured at 20 calls and utility at three budgets. Differences are family-mean contrasts, with blind enumeration as the locator baseline (40.3). Locator completion is live, unlike the Qwen replay (53.6). The guidance gain is defined only for PRIME.

System	FA $\downarrow$	Prem. $\downarrow$	Util. $\uparrow$	Emit $\uparrow$	Reads $\downarrow$	Calls $\downarrow$
Fully informed agent	26.7	28.3	38.0	99.0	3.09	4.4
AgentSpec, stop	0.0	0.0	38.7	87.4	2.96	4.2
AgentSpec, self-examine	0.0	0.0	40.1	92.9	3.06	5.8
ResearchLoop	0.0	0.0	47.7	97.0	3.22	6.0
Proof-or-Stop	0.0	0.0	52.0	99.2	4.21	12.3
GLEAN	14.4	14.6	35.0	83.3	3.09	9.2

Appendix

Table 12: Closest-system results on 480 tasks at 20 calls. FA and Prem. are episode percentages, Emit is among answerable tasks, and Reads counts fields. Calls counts model turns, planning, and reminders as applicable, and for GLEAN only its judge calls on the fully informed agent’s runs.

	$b_{\min}$	$b_{\min}{+}1$	$2b_{\min}$	20
Balanced split: utility $\uparrow$
Ceiling-targeting (no model)	59.2	59.2	59.2	59.2
HESP controller	58.0	59.2	59.2	59.2
Locator-uncertain copy: utility $\uparrow$
PRIME	51.5	50.7	48.8	48.2
PRIME with completion	53.7	54.6 r	59.2 r	59.2 r

Appendix

Table 13: Budgeted PRIME–HESP comparison (480 tasks). The HESP and no-model controllers use scripted checks. The marker r denotes completion replay from PRIME stopping points. The balanced-split premature-conclusion rate is 0.0% for ceiling targeting and 11.7% for HESP at every budget.

Held-out stratum	Value
Sampled; fetched	2,000; 1,999
Not an applicable clinical trial (11.22(b))	370
Not yet due at the snapshot date	750
Applicable and due	879
applicability field unreadable	353
classified	526

Appendix

Table 14: Assistant-built held-out sample. Program exclusions precede tracker labels. The first two tracker-agreement rates have 95% intervals [97.0, 99.2] and [98.6, 99.9].

Quantity	Value
Late or overdue under the assistant-built program	52.0%
Tracker agreement, development pool (317/319)	99.4%
Tracker agreement, held out (518/526)	98.5%
Held-out sample classified (526/1,999)	26.3%
Stated conclusions without a route	0.0%
Filed on time (153): strongest conclusion stated	$c_{2}$

Appendix

Table 15: Assistant-built program on 319 development and 526 held-out registrations.

	Tracker status			Stated
Verdict	rep.	late	overdue	$c_{2}$	$c_{3}$
compliant (153)	153	0	0	153	0
late (86)	2	84	0	13	73
overdue (80)	0	0	80	80	0

Appendix

Table 16: Development registrations (319). Program verdicts versus tracker status (left) and runtime-reported conclusions (right).

		ReAct		Static checklist		PRIME without guidance
Domain	$n$	FA	Utility	FA	Utility	FA	Utility
Software review	21	47.6	24.6	0.0	20.6	0.0	37.3
Research review	8	100.0	4.2	0.0	10.4	0.0	37.5
Security	8	75.0	25.0	0.0	100.0	0.0	35.4
Financial	8	25.0	75.0	0.0	100.0	0.0	83.3
Drug safety	7	85.7	14.3	0.0	81.0	0.0	45.2

Appendix

Table 17: Hand-built exploratory pilot with gpt-4.1 at 20 calls.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 148 papers

Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement
Jun 22, 2026Pierre Dantas, Lucas Cordeiro, Ehsan Nowroozi +1LLM Agent Safety LLM Agent Reliability
The University of Manchester, Manchester, UK · The University of Greenwich, London, UK · Technology Innovation Institute, Abu Dhabi, UAE
When Is Enough Not Enough? Illusory Completion in Search Agents
Feb 7, 2026Dayoon Ko, Jihyuk Kim, Sohyeon Kim +5AI Agent Reliability LLM Answer Verification
Seoul National University · LG AI Research · University of Illinois Chicago +1
When Can You Trust an Agent's Own Verifier? Stress-Testing Counterfactual Rationale Verification Under Search Pressure and Oracle Access
Oct 16, 2026Jonghyun Lee, Moonsu Kim, Sunghwan ParkLLM Agent Verification Automated Agent Evaluation
Yonsei University

Jun 22, 2026J/K move · Enter open · S save

Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement

Pierre Dantas, Lucas Cordeiro, Ehsan Nowroozi, Norbert Tihanyi

The University of Manchester, Manchester, UK · The University of Greenwich, London, UK · Technology Innovation Institute, Abu Dhabi, UAE

LLM Agent Safety LLM Agent Reliability

Open paper AskPDF arXiv

Large Language Model (LLM) agents take irreversible real-world actions, yet no system gives formally grounded, task-level safety guarantees for their plans. In a PRISMA 2020 systematic review, we searched six databases (20-29 May 2026) for work on LLM agent specification, verification, and enforcement published in 2022-2026. We narratively synthesized 38 studies, assessed risk of bias per study, and graded evidence certainty with GRADE. Three authors independently screened a repeated search to validate the single-reviewer screening (agreement: Fleiss'

κ= 0.81

). First, specification is the bottleneck: on one benchmark, only 24.8-34.2% of LLM translations from natural language into Planning Domain Definition Language (PDDL) are semantically correct. Second, runtime monitoring is the most mature enforcement strategy; it reduced unsafe actions by 65.37% in one household-agent study, but its guarantees cover only single steps or short horizons. Third, in one

τ

-Bench study of transactional tool-use agents, blocking 94% of unsafe actions still left fewer than 5% of tasks completed both safely and correctly in most settings, because agents found other unsafe routes (the verifier tax). Fourth, no approach simultaneously achieves soundness, completeness, scalability, semantic grounding, and utility preservation. For seven of the nine principal claims, GRADE gives the lowest certainty rating (Very Low), meaning further studies are likely to produce substantially different figures; each rests on one or two studies in narrow domains. Two of the open problems we rank highest are translating requirements into formal specifications that keep their meaning and enforcing safety over whole plans rather than single actions. We contribute a three-level taxonomy and an agenda of ten open problems.

Abstract

LLM Agent Verification

Explore similar work

Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement

When Is Enough Not Enough? Illusory Completion in Search Agents

When Can You Trust an Agent's Own Verifier? Stress-Testing Counterfactual Rationale Verification Under Search Pressure and Oracle Access