VReAct: Verification as an Architectural Layer for LLM Agents: A V-Model Design, and a Pilot Study of Its Deterministic Core

Ali Afoud; Jie JW Wu

cs.SESep 25, 2026

VReAct: Verification as an Architectural Layer for LLM Agents: A V-Model Design, and a Pilot Study of Its Deterministic Core

Authors: Ali Afoud, Jie JW Wu

Organizations: Independent Researcher Canada · Michigan Technological University Houghton, USA

Abstract

Large language model (LLM) agents built on the ReAct pattern concentrate four responsibilities in one model: selecting a strategy, choosing each action, formatting it, and judging whether the result is adequate. Nothing outside the generative loop can reject its output, so an agent that cannot progress does not report failure; it runs until an external budget terminates it. We propose VReAct, which treats verification as an architectural layer for ReAct agents by adapting the software-engineering V-model: specification levels descend from requirements to individual steps, each paired with a dedicated verifier; a deterministic controller enforces every verdict; and only verification outcomes write memory. A rejection thus localizes the level that introduced the fault, and an agent halts by declining rather than by exhaustion. Each verifier separates a zero-cost deterministic gate from an optional LLM judge, so the contribution and cost of each can be measured independently. A pilot study implements the acceptance- and unit-level verifier pairs, comparing five configurations that share one executor, tool set, and scorer and differ only in verification components, on the four-hop stratum of MuSiQue with an 8B-parameter backbone. Across 47 executions, the two unverified configurations answered none of ten questions, every run ending at a step cap or provider token limit; the verified configuration without a planner answered eight of ten and abstained on the rest. Deterministic gates produced eight of nine observed corrections at zero marginal cost, and planning degraded performance once verification was present. These results characterize termination behavior, not accuracy at scale; we outline a twelve-month plan to complete and evaluate the full architecture, including the integration-level pair the pilot omits.

Figures & tables

	What the literature does	The gap	Our fix
G1	The model that produced the output also checks it ( 11 , 12 ) .	No agent has a checker outside the loop whose verdict it must obey. Self-checking without external feedback degrades accuracy ( 14 , 15 ) .	Verifiers are separate components. The controller enforces their verdicts, so the agent cannot proceed past a rejection.
G2	External checks exist, but for one artifact type: a reasoning step ( 16 ) , a program ( 17 ) .	None pairs checks to the levels of a specification, so a rejection cannot say which stage broke, and recovery means starting over.	One verifier per level. A rejection names the level that failed, and only that level is redone.
G3	Systems mix free structural checks with LLM judgement and report the combined effect.	Nobody has measured what each half contributes, so the layer cannot be priced.	Gates and judges are separate stages, and every correction is attributed to one of them.
G4	Context is trimmed to fit a budget ( 7 , 8 ) .	Nothing decides what deserves to be carried forward: verified and failed work are resubmitted alike.	Only verdicts write memory. Accepted results go to a ledger, rejections to an index keyed by component and cause.

Table 1. The four gaps this work addresses.

Panel of Figure 1 — Figure 1. The motivating frame. Solid arrows show the flow of work; dotted lines show which verifier checks which specification. The pilot implements the outer pair (L1/R3) and the inner pair (L3/R1) only. A V-shaped diagram. Three specification levels descend on the left, requirements to architecture to detailed design, meeting an execution box at the bottom; three verification levels ascend on the right, unit to integration to acceptance. Dotted horizontal lines pair each specification level with the verifier of matching scope.

Panel of Figure 2 — Figure 2. The architecture. Dotted lines are the controller’s mediation: no artifact reaches a verifier and no verdict reaches a generator except through it, and it alone reads and writes the context store (§ 3.3 ). Each verifier is a free deterministic gate followed by an optional LLM judge invoked only on what the gate cleared; a rejection routes to its paired level, and an exhausted budget yields an abstention naming that level. The pilot implements the R3 and R1 rows only. A block diagram. Specification levels L1 to L3 form a column on the left, verifiers R1 to R3 a column on the right, and a vertical deterministic controller sits between them. Dotted double-headed arrows connect every specification and verifier to the controller. An executor box sits below, and a context store above. Output at top right is an answer or an abstention naming the failing level.

Role	Reads	Never reads
Planner (L2)	L1; inventory of tools and components	any execution output
Designer (L3)	L2; this component’s slot and contract	sibling implementations
Executor (one subtask)	its objective; the contract it must honor; its checks; its own failure entries	L1, L2, sibling code, every earlier transcript
Gates	the artifact and the one condition	everything else
Judges R1–R3	the artifact and its paired specification level	every other level, and all transcripts
Controller	the whole state	—

Table 2. The context schema. The executor row is what makes a subtask a new input rather than a continuation.

Panel of Figure 3 — Figure 3. The write path. A verdict is the only event that writes memory, so a rejected attempt survives as its verdict and not as a transcript, and the assembler reads back only what Table 2 permits. A flow diagram. An assembler builds the context for one invocation, which goes to an agent, whose artifact goes to a verifier. The verdict branches: accept writes to a success ledger, reject writes to a failure index. Dotted arrows return from both stores to the assembler.

Panel of Figure 4 — Figure 4. The implemented pipeline. Diamonds are deterministic checks (no model cost); green rounded boxes are LLM judges, invoked only on gate-cleared artifacts; dashed arrows are bounded rejection paths. When plan regeneration is exhausted, the system falls back to executing the question as a single task (§ 5 ). A left-to-right pipeline. A question passes through a planner, plan gates, a plan judge, an executor loop, answer gates and an answer judge to an answer or abstention. Dashed arrows loop backwards from each check to the component it rejects. A deterministic controller spans the width beneath.

Config.	Composition	Plan	Guards	Gates	Judges
N (A)	plain ReAct	–	–	–	–
NP (B)	ReAct + planner	yes	–	–	–
VG (C1)	verified, no planner	–	yes	yes	–
VPG (C0)	+ planner + plan gates	yes	yes	yes	–
VPGJ (C)	+ LLM judges	yes	yes	yes	yes

Table 3. The five configurations; internal labels in parentheses match the released logs. All share one executor, tool set, retriever, scorer and backbone, so adjacent pairs differ by one component.

Config.	exec.	score.	crash	corr.	abst.	cov.
N (plain ReAct)	10	0 5	5	0	5	0.00
NP (+ planner)	10	0 5	5	0	5	0.00
VG (verified)	10	10	0	8	2	0.80
VPG (+ planner)	10	10	0	1	9	0.10
VPGJ (+ judges)	0 7	0 7	0	4	3	0.57
Total	47	37	10	13	24	—

Table 4. Execution accounting over all 47 runs. No configuration produced a silent error, so that column is omitted.

Config.	LLM calls	exec. steps	prompt tok.	compl. tok.
N	12.0	12.0 (all at cap)	18,891	1, 598
NP	13.0	12.0 (all at cap)	11,180	1, 752
VG	11.8	11.8 (3–21)	19,511	1,150
VPG	15.2	14.2 (8–20)	17,226	1, 840
VPGJ	14.8	10.0 (4–18)	20,943	1,210
judge disabled	23.0	20.0	28,609	1,464

Table 5. Mean cost per scoreable run over distinct logged traces. Executor steps are the architecture-neutral comparison; total calls include planner and judge overhead.

Source	rejections	corrections	rate	marginal cost
Deterministic gates	24	8	0.33	none
LLM judges	0 3	0 1	0.33	one call each

Table 6. Rejections and corrections across the logged traces; identical repeats are not double-counted. A correction required the retry to produce a different answer that was then accepted.

Months	Work and rationale
1–2	Consolidation, and the confound. One codebase with the configuration as a parameter, removing the divergence risk of § 7 ; a dense retriever ( 27 ) to raise the 67% ceiling. First ablation : guards without gates, gates without guards, and both ends of the ladder—until those cells exist, the difference in § 5 cannot be assigned to verification rather than loop control.
3–4	Scale, and verifier configurations. Roughly 500 items stratified by hop count, three seeds, three to five model tiers, every repeat retained; paired per-item analysis with effect sizes ( 38 ) , hypotheses registered beforehand. Asymmetric and ground-truth-constrained verification (§ 6.4 ) as separate conditions. Second ablation : full accumulation vs. ledger only vs. both stores, with tokens read per step reported alongside accuracy.
5–7	Transfer to code. Port to SWE-bench ( 32 ) and its agent interfaces ( 33 ) , with function-level suites under rigorous test augmentation ( 34 , 35 ) as a cheaper inner loop. In code the gate layer becomes a compiler, type checker, and test runner, with ground truth no question-answering gate can have; we expect the gate/judge balance to shift and will re-run Table 6 .
8–10	Debugging, and the integration layer. Fault localization and repair, where a failing test supplies both trigger and oracle ( 17 ) . The L2/R2 pair becomes testable at last: individually correct units forming a broken assembly is a real repository failure class invisible to unit checks (§ 3.5 ). Repository length makes the memory schema consequential, so its ablation is repeated with per-component isolation on and off.
11–12	Synthesis. Cross-domain comparison of where each half contributes, a replication package of code, prompts, items, seeds and logs, and submission.

Table 7. Twelve-month plan; half the schedule goes to code, where deterministic verifiers are rich and the L2/R2 row can be exercised.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 131 papers

LLM-as-a-Verifier: A General-Purpose Verification Framework
Jul 6, 2026Jacky Kwok, Shulu Li, Pranav Atreya +6LLM Answer Verification LLM Agent Evaluation
Stanford University · UC Berkeley · NVIDIA Research
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Oct 1, 2026Caiqi Zhang, Rujun Han, Zifeng Wang +4LLM Agent Verification Long-Horizon LLM Agents
University of Cambridge · Google Cloud AI Research
Multi-Aspect Runtime Verification for Simulation-Based V&V of LLM-Enabled Autonomous Agents
Oct 6, 2026Nikolaos Kekatos, Dimitrios Nikou, Anastasios Temperekidis +4LLM Agent Verification Runtime Enforcement for AI Agents
Clone Systems, Cyprus · International Hellenic University, Greece · University of Thessaly, Greece +2

Jul 6, 2026J/K move · Enter open · S save

LLM-as-a-Verifier: A General-Purpose Verification Framework

Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu+5

Stanford University · UC Berkeley · NVIDIA Research

LLM Answer Verification LLM Agent Evaluation Test-Time Scaling LLM Agent Verification

Open paper AskPDF arXiv

Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of LLMs. In this work, we identify verification, the ability to determine the correctness of a solution, as a new scaling axis. To unlock this and demonstrate its effectiveness, we introduce LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that prompt LLMs to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring token logits to generate continuous scores. This probabilistic formulation enables verification to scale along multiple dimensions: (1) score granularity, (2) repeated evaluation, and (3) criteria decomposition. In particular, we show that scaling the scoring granularity leads to better separation between positive and negative solutions, resulting in more calibrated comparisons. Moreover, scaling repeated evaluation and criteria decomposition consistently lead to additional gains in verification accuracy through variance and complexity reduction. We further introduce a cost-efficient ranking algorithm for selecting the best solution among candidates using the verifier's continuous scores. LLM-as-a-Verifier achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). Beyond verification, the fine-grained signals from LLM-as-a-Verifier can also serve as a proxy for estimating task progress. We build an extension for Claude Code, enabling developers to monitor and improve their own agentic systems. Finally, we show that LLM-as-a-Verifier can provide dense feedback for RL, improving the sample efficiency of SAC and GRPO on robotics and mathematical reasoning benchmarks.

Abstract

LLM Agent Verification

Explore similar work

LLM-as-a-Verifier: A General-Purpose Verification Framework

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

Multi-Aspect Runtime Verification for Simulation-Based V&V of LLM-Enabled Autonomous Agents