Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance

S M Abdulla Al Mamun; Akif Islam; Mohd. Ruhul Ameen

cs.SEOct 27, 2026

Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance

Authors: S M Abdulla Al Mamun, Akif Islam, Mohd. Ruhul Ameen

Organizations: Department of Computer Science, The University of Manchester, United Kingdom · Department of Computer Science and Engineering, University of Rajshahi, Bangladesh · Department of Computer Science and Engineering, Varendra University, Bangladesh

Abstract

Can additional specialist language-model contexts improve software assurance when deterministic tool evidence is already available? We study this question through five experiments spanning authored Java warning controls, OWASP Benchmark assessment, Defects4J test generation, SV-COMP assertion verdicts, and deliberately misleading evidence packages. The evaluated systems use Qwen2.5-Coder-7B-Instruct, with a 1.5B sensitivity analysis, and complete 13,638 model requests. On eight related development warnings, increasing the context count from 1 to 50 increases token use 49-fold without improving unsafe clearance. On 160 OWASP cases, model configurations trade clearance against unresolved outcomes; the ten-context arm assesses only 96 cases. On 32 SV-COMP tasks, raw verifier feedback raises mean correct model verdicts from 12.7 to 22.0, but also raises incorrect verdicts from one to seven; a JBMC-plus-JVM validation baseline returns 26 correct verdicts and six unresolved outcomes. On two sets of 120 evidence packages, models reject 70-100% of applicable counterexamples, while construction-specific scope checks classify every package correctly. No target defect is discovered in three Defects4J bugs, one unsupported by the test adapter. These descriptive results support deterministic evidence admission for the tested settings, while exposing limits of verdict aggregation, feedback interpretation, and the available evaluation coverage.

Figures & tables

Work	Principal mechanism	Relation to this study
QASecClaw [ 1 ]	LLM filtering of SAST reports	Warning disposition
Evident [ 4 ]	Validated harness; analyser decides	Deterministic admission
MAGS [ 5 ]	Dafny translation and repair	Specification scope
VERDICT [ 6 ]	Assumed contracts and reconciliation	Modular obligations
Kim et al. [ 8 ]	Controlled agent architectures	Scaling and task fit
This study	Fixed-evidence verdict aggregation	Clearance, abstention and applicability

TABLE I: Closest systems and the boundary they investigate. Prior results are contextual comparisons, not reproduced baselines.

Panel of Figure 2 — Fig. 1: Overview of the experimental interfaces. Each experiment uses the relevant tool adapter; the tools are not fused into a single end-to-end system. Deterministic baselines bypass the model. E5 outputs accept/reject/unresolved instead of warning dispositions.

Exp.	Data (stage)	Req.	Compared arms
E1	8 authored NPE controls (dev)	144	1 vs 5 ctx; $\pm$ gate
E4	same controls, 7B + 1.5B (dev)	4,608	1/5/10/25/50; adaptive
E1/E4	160 OWASP cases (held-out)	3,840	$k\!\in\!\{1,5,10\}$
E2	Defects4J Cli-1/2/3 (dev)	54	1 vs 5 ctx; contract tests
E3	32 SV-COMP tasks (frozen)	192	$\pm$ feedback; validated backend
E5	20 progs $\times$ 6 pkgs (dev)	3,600	$\pm$ feedback, $\pm$ gate

TABLE II: Experiments, data and scale. Req. = model requests.

Method	Unsafe clearance /12	Benign clearance /12	Unresolved /24
Scanner retain-all	0	0	0
Verifier only (no model)	0	12	0
1 context	11	12	1
5 specialist contexts	11	12	1
1 context + scope gate	0	12	1
5 contexts + scope gate	0	12	1

TABLE III: E1 development controls, Qwen2.5-Coder-7B, three seeds (4 defects and 4 benign warnings per seed).

Fig. 2: E4: unsafe clearance (ungated, mean of 3 seeds; whiskers show the seed range) against total tokens. A 49

\times

budget increase leaves unsafe clearance near 100%; the deterministic gate holds it at zero for every

k

Method	Unsafe /100	Benign /60	Unresolved /160
Retain-all	0	0	0
Clear-without-signal	40	20	0
1 context	33/36	22/18	72/74
5 contexts	39/41	19/23	85/81
10 contexts (96 assessed)	31/28	16/19	111/113

TABLE IV: OWASP outcomes on 160 selected cases. Paired counts are seeds 17/43; denominators are fixed across rows.

Fig. 3: E3 verdicts on 32 frozen tasks (means of three seeds). Raw feedback raises both correct and incorrect counts. The deterministic comparator additionally uses JVM validation.

Fig. 4: E5 misleading-evidence challenge (100 corrupted and 20 clean packages per seed). The gate removes erroneous acceptance but cannot repair the model’s rejection of valid evidence; the scope validator alone makes neither error.

Setting	Executed	Pass both	Fail both	Buggy/fixed differential
1 context, strict parser	0	0	0	0
5 contexts, strict parser	16	13	3	0
1 context, secondary parser	28	25	3	0
5 contexts, secondary parser	125	104	21	0
Contract-test baseline (no model)	180	171	9	0

TABLE V: E2 independent discovery on Defects4J Cli-1/2/3 (3 seeds). Counts are executed test specifications, not independent bugs.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 147 papers

Evidence-First Agentic Verification
Oct 13, 2026Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou +7LLM Agent Verification AI Agent Evaluation
Scale AI · University of Arizona
ClaimScope: Label-Blind Witness Verification for Mathematical Claim Auditing
Oct 15, 2026Mushir AkhtarLLM Mathematical Reasoning Agent Benchmarks
Agentic Interpretation: Harness-Governed Evidence Construction for Software Agents
Oct 15, 2026Yifeng He, Yanjun Wang, Siwei Cui +2Software Engineering Agents LLM Agent Harnesses
University of California, Davis, Davis, California, USA · AWS AI Labs, USA

Oct 13, 2026J/K move · Enter open · S save

Evidence-First Agentic Verification

Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou, MohammadHossein Rezaei+6

Scale AI · University of Arizona

LLM Agent Verification AI Agent Evaluation

Open paper AskPDF arXiv

Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the execution trace. A judge may overlook this evidence or accept unsupported claims, including an agent's own reports of success. When rubric verdicts serve as reinforcement-learning rewards, such errors can reinforce undesirable behavior. We introduce Sauron, an evidence-first agentic verifier that investigates both state and trace and relies on a claim only after independently validating it against its source. Bounded queries and joint investigation of related criteria reduce context use and redundant work, and each verdict carries an auditable receipt of its supporting evidence. Across 2,586 human-labeled criteria from 100 personal-assistant episodes, Sauron achieves the highest F1 of five verifiers on all eight backbones tested while costing less than Gandalf, the strongest baseline. With GPT-5.6 Sol, it reaches 0.96 F1 versus Gandalf's 0.92 at 23% lower cost, with a wider margin on process criteria. On a paired fault-injection suite covering 13 families, several of which simulate reward hacking, Sauron detects 97% of injected failures while flagging only 2% of clean controls, outperforming baselines even when they are given the full trace. On TRAIL, a benchmark for detecting and localizing real agent errors, Sauron leads both Gandalf and TRAIL's own judge on all three quality metrics.

Abstract

LLM Agent Verification

Explore similar work

Evidence-First Agentic Verification

ClaimScope: Label-Blind Witness Verification for Mathematical Claim Auditing

Agentic Interpretation: Harness-Governed Evidence Construction for Software Agents