AgentStripes: An Evidence-Graded Maturity Model for AI Agents

Nutan Kumar Naik; Vijay Prasad Poudel; Aditya Kumar Saroj; Saurav Samantray; Abhishek Patel

cs.AIOct 13, 2026

AgentStripes: An Evidence-Graded Maturity Model for AI Agents

Authors: Nutan Kumar Naik, Vijay Prasad Poudel, Aditya Kumar Saroj, Saurav Samantray, Abhishek Patel

Organizations: Algen.AI, Bengaluru, Karnataka, India · Department of Earth and Atmospheric Sciences, National Institute of Technology Rourkela, Odisha, India

Abstract

Benchmarks report what an Artificial Intelligence (AI) agent can do on a task suite, but not whether one released version has been engineered with the controls that make it reliable, safe and observable, or how strongly that is shown. This paper proposes AgentStripes, an evidence-graded maturity model for AI agents. It defines levels (0 to 7 stripes), process areas (12 areas holding 40 capabilities), three gates (basic, safety, proof) that hold an agent at a level however high its score, and evidence grades E0 to E5. Each applicable capability receives a rating from 0 to 3 and a separate evidence level; these are combined into a credit, aggregated into a severity-weighted score and converted into stripes that the gates limit. Instruments gather evidence, language-model judges interpret it under explicit criteria, and the arithmetic is deterministic. The model is compared with standards at topic level without a claim of conformance, and an unexecuted validation protocol is specified. An empirical evaluation (exploratory, non-confirmatory) of 30 public Python agent repositories, with static evidence only, is a demonstration and not a validation: static evidence caps every final stripe at 2, 26 repositories end at one stripe and 4 at two, and this distinction is unreliable, since re-judging changed the final stripe of 4 of 22 repositories (engine, dossier and panel changed), whereas none of 8 panel-only repeats, all at stripe 1, changed. The parameters are expert conventions without outcome based calibration, and the stripes have not been shown to predict reliability or safety.

Figures & tables

Precedent	Unit	Levels or scale	Evidence or assurance	Assessor	How AgentStripes differs
CMMI; ISO/IEC 33020 ( CMMI Institute, 2023 ; ISO/IEC, 2019 )	Organisational process	Capability or maturity levels; rated process attributes	Evidence collected in an assessment	Assessment team	Rates one released product, not a process; grades the evidence behind each rating
OWASP SAMM; BSIMM ( OWASP SAMM Project, 2020 ; Black Duck Software, 2026 )	Organisation’s security programme	Three levels per practice; observed activities	Self- or expert assessment; interviews	Self, expert or consultant	Product, not programme; agent-specific capabilities
ISO/IEC 25010, 25059 ( ISO/IEC, 2023d ; ISO/IEC, 2023c )	Product (AI system)	Characteristics, no levels	None defined	–	Rates instances of several characteristics at a pinned commit (Section 5 )
OpenSSF Scorecard ( Zahan et al., 2023 )	Repository	0–10 per check, risk-weighted mean	Automated heuristics	Tool	Agent-specific capabilities; ratings discounted by evidence; gates instead of a mean only; judge agreement reported
Best Practices badge ( Open Source Security Foundation, 2026a )	Open-source project	Passing, silver, gold	Self-certification	Maintainer	Assessor-run; graded evidence instead of self-attestation
SLSA ( SLSA Community, 2025 )	Build or source of an artefact	Levels per track	Signed provenance	Consumer	Covers agent capabilities beyond supply-chain integrity; shares the idea that higher levels need stronger evidence

Table 1: Software-engineering precedents and how AgentStripes differs from each.

Class	What it assesses	Items verified
A. Individual-agent maturity	The agent itself	Agent Maturity Compass (AMC, an open-source industry tool, evidence-weighted L0–L5 over eight areas; no independent validation in the corpus) ( AgentMaturity, 2026 ) ; MLCommons Agentic Product Maturity Ladder V0.1 (a prototype white paper: levels R0–R5 per supported task, measured by benchmarks that are built level by level; outcomes, not engineering mechanisms) ( McGregor et al., 2025 )
B. Organisational/enterprise readiness	Whether an organisation or its development process is prepared to adopt or govern agents	ACMM (a six-level, CMMI-inspired model of how a codebase supports AI-assisted development, defined by its feedback loops and supported by one practitioner’s four-month experience report) ( Anderson, 2026 ) ; Microsoft Agentic AI Adoption Maturity Model ( Microsoft, 2026b ) ; TDWI readiness assessment ( TDWI, 2026 ) ; AdversLLM (0–5 practice scores for LLM applications, two companies compared) ( Belmoukadam et al., 2024 ) ; the Agentic AI Governance Maturity Model (AAGMM; five levels, twelve domains, validated by simulation only) ( Acharya, 2026 ) ; interview study placing 12 companies on a six-level scale ( Alvanakis Apostolou et al., 2026 ) ; CMMI as the staged-maturity precedent ( CMMI Institute, 2023 )
C. AI governance frameworks	Principles, processes and risk-management practice, not agent-level scores	National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF) and its generative-AI profile ( National Institute of Standards and Technology, 2023 ; National Institute of Standards and Technology, 2024 ) ; International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 23894 and 42001 ( ISO/IEC, 2023a ; ISO/IEC, 2023b ) ; the European Union (EU) AI Act ( European Parliament and Council of the European Union, 2024 ) ; AURA, a risk score for an agent action in context ( Satta Chiris and Mishra, 2025 ) ; AGENTSAFE, a governance framework whose validation is a plan ( Khan et al., 2025 ) ; Microsoft’s Agent Governance Toolkit, a runtime policy-enforcement toolkit ( Microsoft, 2026a )
D. Security maturity/controls	Control posture and threat knowledge	SANS AI security maturity model ( Cochran, 2026 ) ; AI-ZTMM, a zero-trust maturity model for an organisation’s control over the agents it runs (four levels taken from the CISA Zero Trust Maturity Model, self-assessed; reviewed by experts and checked for coverage against MITRE ATLAS case studies, not yet applied in an organisation) ( Mok et al., 2026 ) ; Open Worldwide Application Security Project (OWASP) Top 10 for LLM ( OWASP GenAI Security Project, 2024 ) and for Agentic Applications ( OWASP GenAI Security Project, 2025 ) ; MITRE ATLAS ( The MITRE Corporation, 2026 )
E. Agent capability/autonomy taxonomies	Levels of capability or autonomy, not engineering quality	Levels of AI Agents (L0–L5, conceptual, no assessment method) ( Huang, 2024 )
F. Agent evaluation frameworks	Agent behaviour or performance	CLEAR ( Mehta, 2025 ) , HAL ( Kapoor et al., 2026 ) , CORE ( Michelakis et al., 2025 ) , MAESTRO ( Ma et al., 2026 ) , Trace-Based Assurance ( Păduraru et al., 2026 ) and the other works of Section 2.5

Table 2: Maturity, readiness, assurance and governance artefacts by unit of assessment and function (items verified for this paper).

Panel of Figure 1 — Figure 1: From an assessed artefact to the final stripes. Orange steps involve language-model judgement, green steps are deterministic code, and the blue step is the result.

Stripe	Level	Score from	Requirement (basics, safety and demonstration gates)	Step to the next level
0	Not demonstrated	0	A basic capability is rated below 1, or could not be checked (lower bound)	Document the agent’s purpose and how to run it, describe every tool, keep secrets out of the release
1	Assessable	12.5	Basics gate: Purpose documented, Runnable, Tool interfaces clear and Secrets protected rated 1 or more	Bring the six safety-gate capabilities to rating 2 (adequate) as shipped
2	Safeguarded as built	25	Safety gate: Untrusted input separated, Consequential actions controlled, Runs bounded, Failures surfaced, Execution isolated (if the agent runs code) and Secrets protected rated 2 or more; documentation or code suffice	Demonstrate the safety behaviours in the developer’s integrity-checked runs (E3) and raise the score
3	Demonstrated	37.5	Demonstration gate: every safety behaviour shown at E3 or higher	Broaden maturity: more capabilities adequate or strong, and better evidenced
4	Demonstrated, broad	50	As stripe 3	Show the safety behaviours in a published re-runnable suite (E4) or the assessor’s sandbox (E5a)
5	Reproducible	62.5	Every safety behaviour shown at E4 or E5a	Broaden maturity further

Table 3: The eight levels of AgentStripes. Each level requires everything required below it and the score shown; “safety behaviours” are runs bounded, failures surfaced, untrusted input separated and, where they apply, consequential actions controlled and execution isolated. The last column is the improvement step to the next level.

Area	Capability	Sev.	Gate	Definition
Task contract	Purpose documented	L	B	Someone new can tell from the agent’s documentation what it does, who it is for, what it takes in and what it produces.
	Limits documented	M	–	The documentation states what the agent must not be used for, or its known limitations.
	Instructions owned and explicit	L	–	The agent’s instructions can be found in one clear place, maintained by the developers, and state its rules plainly.
Tool design	Tool interfaces clear	L	B	Each tool tells the model what it does and when to use it.
	Tool inputs specified	M	–	Tool inputs have declared types or a schema the framework can validate.
	Tools fail gracefully	M	–	When a tool’s call to a network, database or file fails, the agent gets a useful message rather than a crash or a raw traceback, and large results are bounded.

Table 4: The 40 AgentStripes capabilities in 12 areas with their definitions, severity (H, weight 3; M, weight 2; L, weight 1) and gate role (B basics gate; S safety gate; P demonstration set). † Emerging: counted only when it raises the score. ∗ High severity but not a gate capability. The 0–3 anchors, example mechanisms, the failure each capability prevents, its sources and its improvement measure are published with the model (Data Availability).

Level	Where it comes from	Credit
E0	Could not be assessed	0.00
E1	Documentation	0.50
E2	Code or configuration (read-only capabilities: 0.90)	0.70
E3	The developer’s runs, checked for integrity	0.85
E4	A published suite anyone can re-run	0.95
E5a	The assessor’s sandbox	1.00

Table 5: Evidence levels and their credits. The credits are uncalibrated ordinal conventions (Section 4.1 ); the assessor’s party type is recorded with every result (Section 3.1 ).

Gate capability	Holds at	Rationale	Failure it is meant to prevent (illustrative)
The basics (assessability): any of these below 1
Purpose documented	0	No one can judge an agent if they cannot tell what it is for	An “email assistant” that also posts to Slack and edits the Customer Relationship Management (CRM) system is adopted
Runnable	0	If no one else can run it, no one else can check it	A reviewer cannot start the agent, so every claim about it goes untested
Tool interfaces clear	0	The model chooses tools from their descriptions; with none, the choice cannot be checked	A tool called process(x) is called with a customer’s address instead of an order number
Secrets protected	0	A committed live key makes the published release unassessable as it stands	An API key in the repository lets anyone who clones it incur charges
Safety (safeguard baseline for the reference context): any non-starred high-severity capability below 2

Table 6: Every gate capability, the stripe at which it holds an agent, why it exists, and an illustrative failure it prevents. For Untrusted input separated and Failures surfaced the rationale is stated for the anchors of method 1.1.0, under which a rating of 2 shows the mechanism, not resistance to the failure (Section 3.8 ).

Row	Sev.	Status	$r$	Evid.	$\epsilon$	$c$
Capability A	H	assessed	2	E2	0.70	0.4667
Capability B	H	assessed	3	E2	0.70	0.7000
Capability C	M	assessed	2	E1	0.50	0.3333
Capability D	M	assessed	1	E2	0.70	0.2333
Capability E	M	assessed	3	E2	0.70	0.7000
Capability F ∗	L	assessed	2	E2	0.90	0.6000

Table 7: A synthetic worked example (Stage 1–3): rating

r

, evidence level, credit

\epsilon

and

c=\tfrac{r}{3}\epsilon

. † emerging capability; ∗ a read-only capability (code credit 0.90).

Parameter	Value	Origin	Empirical status in this paper
Rating and aggregation
Rating scale and anchors	0–3; opt-in safeguards at most 1	Authors’ judgement	Agreement measured within and across panels of one model family (Section 7.4 ); $\alpha<0.667$ for 10 capabilities, 3 of them gate capabilities; not compared with human raters; opt-in rule not applied by judges (Table 10 )
Severity weights $w_{i}$	1, 2, 3	Authors’ convention	Not calibrated; joint perturbation changes at most 1 final stripe (Section 7.3 )
Evidence credits $\epsilon(L)$	0, 0.50, 0.70 (read-only class 0.90), 0.85, 0.95, 1.00 (E5a, E5b)	Authors’ ordinal convention	Not calibrated; the E2 credit has the largest first-order share of score-stripe changes under joint perturbation, final stripes almost none
Stripe width	12.5 points	Convention ( $100/8$ )	Varied over 11–14 points in the joint perturbation
Gate thresholds	basics $<1$ ; non-starred high severity $<2$	Authors’ judgement (assessability; harm in the reference context)	Decisive: a safety threshold of $<1$ changes 13 of 30 final stripes; in the joint perturbation the thresholds account for nearly all variation of the final stripes; not validated against harm

Table 8: Every parameter that decides a level: value, origin and empirical status (method 1.1.0). “Authors’ judgement”: set by the authors during development, without external experts or structured elicitation (Section 3.2 ).

Inference	Evidence in this paper	Defeaters identified	Tested by
Scoring	Quotes verified to exist in the cited files; instrument ceilings and floors (fixture test run, Table S2 ); acceptance rules	Quotes that exist but do not support the rating (audit not run); instruction-like repository text; opt-in or prompt-only safeguards credited (Table 10 ); absence of a hazard credited (A1)	Quote audit; fixture tests
Generalisation	Agreement of ratings, gate decisions and levels across panels (Section 7.4 )	One model family and prompt; dossier composition; prevalence of rating 2; 3 gate capabilities below the reliability threshold	H3
Extrapolation	Undetermined states and the cause of every unassessed gate capability (Section 7.2 )	Material outside the dossier; guards outside a library target; documentation describing absent mechanisms; framework releases (A8); a level reachable on documentation alone (A4)	H1, H2, H5
Decision	None	A level read as a certification or a ranking; a stripe shown without its range	User study (not planned)

Table 9: Interpretation and use argument for a level: inferences, evidence in this paper, known defeaters and the hypotheses that test them.

Requirement	What the evaluation did	Consequence and treatment
Judges see only the dossier (Section 3.5 )	The primary panel ran in a harness with file access; non-access cannot be shown from its files	A confined rerun of all 30 repositories, with confinement verified from the transcripts, changed 3 final stripes ( $\alpha=0.887$ ; Section 7.4 )
Judge model identifiers recorded (Section 3.3 )	The study files record only the harness alias	Identifiers recovered from the harness transcripts: 87 of 90 primary answers from claude-sonnet-5-5 and 3 (one repository, in every panel that judged it) from claude-sonnet-5
Judges from at least two providers (Section 10 )	One model family	Not addressed; agreement is reported as within one family
Opt-in and prompt-only safeguards rated at most 1 (Section 3.4 )	The rules were added with method 1.1.0 after the primary judging; no judge request, including those of the confined rerun, contained them	The stored ratings were not re-judged, so the rule did not shape them; ratings of opt-in safeguards reflect the judges’ own reading
E0 means not assessable (Table 5 )	Two ratings supported only by an instrument floor, with no quote, were recorded by the engine at E0 with credit 0	Both scores are lower than with the cells unassessed (by less than one point); no stripe changed
Per-capability $\alpha$ and the experimental flag (Section 3.7 )	Computed in the revision (Table AgentStripes: An Evidence-Graded Maturity Model for AI Agents ): 10 capabilities, 3 of them gate capabilities, fall below 0.667	Flagged as experimental in the results; they are still scored and gate as specified, and Table 16 shows the effect of taking them out of the gates

Table 10: Where the empirical evaluation departed from the specification and the planned study, and how each departure was handled.

Panel of Figure 2 — Figure 2: Which panel judged which dossier (solid arrows), and the comparisons of Section 7.4 (dashed). Every panel had three judges; all answer sets are scored with engine 0.3.0.

Quantity	Value ( $n=30$ )
Final stripes, headline lower bound	0: 9 (all undetermined, range 0–1); 1: 20 (3 undetermined, range 1–2); 2: 1; 3–7: none
Final stripes, rules of engine 0.2.2	0: 0; 1: 26; 2: 4 (before the rule for unassessed gate capabilities, adopted after these results were seen)
Final-stripe range	0–1: 9; 1–1: 17; 1–2: 3; 2–2: 1
Score stripes (headline score)	0: 1; 1: 1; 2: 9; 3: 18; 4: 1
Binding limit	safety 20; basics 8; demonstration 1; score 1
Undetermined gate	basics 8; safety 3; none 19

Table 11: Results of the empirical evaluation (engine 0.3.0, primary panel). The headline final stripe is the lower end of its range; SD is the population standard deviation.

Gate capability	Gate	Cells	One rating	Cause	Shortest judge statement
Tool interfaces clear	basics	5	0	file not in dossier	“Tool definition code (tool.py, function_schema.py) is not shown.”
Execution isolated	safety	4	2	file not in dossier	“README claims sandboxed execution and a sandbox package exists, but the executor code is not shown.”
Runs bounded	safety	4	1	file not in dossier	“Only per-command timeouts are visible; the main agent loop (step limit) is in files not shown.”
Untrusted input separated	safety	4	0	file not in dossier	“Prompt construction code is not shown.”
Runnable	basics	3	0	README section beyond the 6,000-character cut	“README contents list Installation and Running Your Crew sections but their steps are not in the shown portion.”
Consequential actions controlled	safety	1	0	file not in dossier	“Shell and code interpreter tools exist in the file list but their approval or constraint logic is not shown.”

Table 12: Why gate capabilities were unassessed in the primary panel: every one of the 22 unassessed gate cells (15 repositories), by capability. “One rating”: cells where one judge’s rating was accepted and the other two judges abstained, so the two accepted ratings required were not reached. Cause: what the abstaining judges stated, checked against the dossier; in every cell the material they named exists in the release but was not in the dossier. The full list is in Table AgentStripes: An Evidence-Graded Maturity Model for AI Agents .

Figure 3: Repositories by score stripes (headline score) and by headline final stripes (lower bound), with the flow between them (counts only). All 9 results at zero stripes and 3 of those at one are undetermined lower bounds; under the rules of engine 0.2.2 the distribution would be 0/26/4 (Table 11 ).

Figure 4: The stripe at which each gate holds an agent, on the stripe axis of Figure 3 (basics: 0; safety: 1; demonstration, by the weakest evidence level of the safety behaviours: 2 below E3, 4 at E3, 6 at E4 or E5a, 7 at E5b), and the number of repositories for which each gate is the binding limit; the score binds for the remaining 1.

Alternative rule or parameter	Score	Final
Code evidence at 0.90 for every E2 rating (the blanket rule of engine 0.2.2)	mean shift +9.81 (range +2.62 to +14.07); score stripes change for 26; $\rho=0.998$	8/21/1; changed for 1
No read-only exception (every E2 rating at 0.70)	mean shift $-$ 0.37; score stripes change for 1; $\rho=0.999$	9/20/1; changed for 0
Gates ignore unassessed capabilities (upper bound)	unchanged	1/25/4; changed for 11
Excluded score as headline, also under low confidence	mean shift +2.03; $\rho=0.884$	8/21/1; changed for 1
Unassessed capabilities counted as 0 for every repository	mean shift $-$ 4.61; $\rho=0.941$	9/20/1; changed for 0
All rules of engine 0.2.2 (blanket 0.90, excluded score, unassessed gate capabilities ignored, Secrets protected rated 1 in the basics gate)	mean shift +12.34; $\rho=0.856$	0/26/4; reproduces the stored 0.2.2 results for 30 of 30

Table 13: Sensitivity to alternative rules and parameters, recomputed on the stored ratings of the primary panel (no repository re-judged). “Final”: repositories at 0/1/2 stripes (headline, lower bound) and the number whose headline final stripe changes. The joint perturbations draw every listed parameter independently and uniformly.

Measure	22 pairs (dossier and panel differ)	8 repeats (panel only)	30, confined rerun (panel and confinement)
Capability ratings
Rating cells that differ	219 of 871 (25.1%)	42 of 318 (13.2%)	164 of 1188 (13.8%)
raised, lowered, to or from unassessed	50, 53, 116	9, 17, 16	38, 45, 81
Ordinal $\alpha$ on cells rated by both [95% CI]	0.812 [0.755, 0.857]	0.879 [0.825, 0.923]	0.887 [0.852, 0.915]
cells rated by both panels	647 of 871	263 of 318	910 of 1,188
Nominal $\alpha$ , unassessed as a category [95% CI]	0.63 [0.57–0.69]	0.80 [0.73–0.86]	0.80 [0.75–0.84]

Table 14: Agreement between judge panels on capability ratings, gate decisions and levels, all answer sets scored with engine 0.3.0. The ordinal

\alpha

is computed over the two panels’ aggregated (median) ratings on the cells that both panels rated, so cells that one panel leaves unassessed are missing data for it; the nominal

\alpha

and the assessed-or-not

\kappa

count them. Intervals are 95% bootstrap intervals over repositories (2,000 resamples), except for the share of changed levels (exact binomial). Score changes use the headline score and, in brackets, the excluded score. A

\kappa

is undefined (n/a) when both panels use a single category.

Measure	Value
$\alpha$ , all capabilities (primary panel)	0.854 [0.828, 0.877]
$\alpha$ , safety-gate capabilities (primary)	0.758 [0.699, 0.812]
$\alpha$ , all capabilities (confined rerun)	0.842 [0.797, 0.879]
$\alpha$ , safety-gate capabilities (confined)	0.754 [0.652, 0.839]
Exact agreement, all judge pairs (abstain counted as a value)	2,974 of 3,600 (82.6%)
Both judges rated, exact agreement	2,439 of 2,765 (88.2%)

Table 15: Agreement among the three judges of one panel, and quote verification.

\alpha

is Krippendorff’s ordinal

\alpha

with judges as raters and abstentions missing, with a 95% bootstrap interval clustered by repository.

Item	Rule applied to the stored ratings	Final	Changed	Upper	Shift
A1	Consequential actions controlled not applicable when the agent cannot take consequential actions	9/19/2	1	1	0.00
Context	Read-only reference context: safety gate without Consequential actions controlled and Execution isolated	9/10/11	10	16	0.00
Context	Action-taking reference context: ratings and exclusions that rest on hazard absence become unassessed	9/21/0	1	1	$-$ 0.10
A2	Bound: Untrusted input separated capped at 1	9/21/0	1	3	$-$ 1.03
A3	Bound: Failures surfaced capped at 1	9/21/0	1	4	$-$ 1.19
A3	Failures surfaced gates below 3	9/21/0	1	4	0.00

Table 16: The amendments of Section 3.8 and alternative reference contexts, applied to the stored ratings of the primary panel (no repository re-judged; the reported results keep method 1.1.0). “Bound”: the largest effect the amendment could have on the stored ratings, since the stored rationales do not identify the mechanism behind a rating; “Rule”: the reliability rule of Section 3.7 applied to the gates. Final: repositories at 0/1/2 stripes (headline lower bound; 9/20/1 under method 1.1.0 and 0/26/4 under the rules of engine 0.2.2, before the rule for unassessed gate capabilities); changed: headline final stripes that change; upper: upper ends of the range that change; shift: change of the mean headline score.

Figure S1: Corpus overview. (a) Records per publication year by relevance tier (Tiers 1–3 only; Tier 4 and pre-2020 records excluded; tiers are heuristic labels; 2026 is a partial year). (b) Number of the 103 registered benchmarks flagged for each dimension (darker bars: authorisation, consequential action, human oversight, memory security and observability). Flags are mention-mined and heuristic, and “unknown” values count as not flagged.

Emerging capability	Sev.	Applies	Rated	Rated $\geq 2$
Delegation controlled	M	25	22	15
Users informed	M	30	22	8
Actions recoverable	M	27	22	9
Least privilege	H	30	28	14
Personal data protected	H	30	18	6
Outputs safe for their destination	H	30	26	10

Table S1: Adoption of the nine emerging capabilities in the sample: repositories where the capability applies, where it was rated, and where it was rated 2 or higher (primary panel). For comparison, the median share rated 2 or higher among the 31 other capabilities is 57%.

Threat	Mitigation in the method	Test protocol (fixtures)	Status
Instruction-like text in the repository	Judges instructed to treat text as evidence; quotes must exist	Seed READMEs and comments that claim controls or order a rating; check ratings against instrument findings	Pending
Relevant files outside the judge context	Instruments read all files; judges leave capabilities unrated rather than absent	Move a mechanism into an unshown file; verify unrated and not 0. Guard files read first (run; Section 7.4 )	Partly run
Judge reads files outside the dossier	Judges given only the request file; tool calls checked in the transcripts	Confined rerun of all repositories with verified confinement (Section 7.4 )	Run
Judge and instrument disagree	Ratings are clamped to instrument ceilings and floors	Plant a committed key and a claim of protection; verify the ceiling. Run with a synthetic judge answer (ceiling 0; claimed rating 3 lowered to 0); with real judges pending	Partly run
Forged quotes and line numbers	Quote and line verification within three lines	Judge outputs with altered quotes; verify rejection	Pending
Prompt injection against the process	Dossier sanitisation, quote rule, deterministic arithmetic after judging	Injection strings in code and docs; compare ratings with and without	Pending

Table S2: Judge threat model and the status of the corresponding tests. “Pending” means not yet executed.

Field group	Content
Artefact	repository Uniform Resource Locator (URL), commit or release digest, framework and dependency versions of the assessed agent
Method	method and engine versions, instrument identifiers, parameter table (credits, read-only class, weights, stripe width), rules for unassessed and emerging capabilities
Judges	provider, model identifier as resolved for each answer (not only an alias) and any version information, prompts, parameters, number of judges, tools available to the judges
Run	assessment date, assessor and the assessor’s party type (first, second or third party), execution environment, random seeds
Dossier	files read, accepted quotes with line numbers, instrument findings, rejected judgements
Result	ratings, evidence levels, applicability, score with its two bounds, score stripes, gate limits with any undetermined gate and the cause of each unassessed gate capability (not shown or not found), final stripes with their range

Table S3: Assessment-report schema.

Stratum	$n$	Repositories	Final	Undet.	Safety	Score	Cov.
Framework or SDK	7	openai-agents-python , smolagents , crewai , langgraph , pydantic-ai , agno , ag2	2/5/0	3	6 (4)	35.7	73
Coding, computer-use or browser agent	6	browser-use , swe-agent , gptme , agent-s , magentic-ui , mobile-use	2/4/0	3	5 (4)	36.0	87
Task application	12	tradingagents , deer-flow , genericagent , hermes-agent , nanobot , vibe-trading , deepagents , openmanus , cowagent , daily_stock_analysis , mirothinker , local-deep-research	4/7/1	5	10 (5)	42.0	84
Enterprise or vertical agent	5	redamon , strix , holmesgpt , db-gpt , finrobot	1/4/0	1	5 (2)	36.7	77

Table S4: Classification of the 30 targets into strata, assigned before assessment, with the results by stratum: repositories at 0/1/2 stripes (headline lower bound), how many of them are undetermined, how many fail the safety gate (in brackets: with a shortfall in Consequential actions controlled), the mean headline score and the mean coverage (%). With 5 to 12 repositories per stratum no comparison between strata is drawn. deepagents , classified as a task application, could also be counted as a framework; the framework stratum would then be at 2/6/0.

File	Secure Hash Algorithm 256 (SHA-256), prefix
agentstripes_v2/init.py	e3b0c44298fc1c14
agentstripes_v2/main.py	13a1a5b340cdcfc1
agentstripes_v2/build_model.py	10f0b06218d7afcd
agentstripes_v2/cli.py	93d8114ba3bb6d29
agentstripes_v2/evaluate.py	214121ebfa58c729
agentstripes_v2/facts.py	240ec12516b224ad

Table S5: Frozen engine files of version 0.3.0 (first 16 of 64 hexadecimal characters of the SHA-256; full values in workflow1_claude/engine_freeze_0.3.0.json ). Every result of the paper was scored with these files.

Row	Sev.	Rated $\geq 2$	Rated $\leq 1$	Unassessed	n/a
Task contract
Purpose documented	L	27	3	0	0
Limits documented	M	5	11	14	0
Instructions owned and explicit	L	27	0	3	0
Tool design
Tool interfaces clear	L	17	8	5	0

Table S6: Aggregate counts per capability over the 30 repositories (primary panel), in the order of the capability inventory. “Rated

\geq 2

” and “rated

\leq 1

” count repositories whose median accepted rating is at least 2 or at most 1; “unassessed” counts repositories with fewer than two accepted ratings; “n/a” counts repositories where the capability does not apply. † emerging capability.

Predictor	Odds ratio	95% CI	$p$
Intercept (Task contract, application target)	0.29	0.17–0.50	0.000
Log target files (SD)	1.20	0.70–2.04	0.503
Share of target files shown (SD)	0.91	0.50–1.65	0.754
Files truncated in the dossier (SD)	0.76	0.54–1.07	0.115
Load cap hit	0.47	0.27–0.84	0.010
Framework target	1.92	1.23–3.00	0.004

Table S7: Logistic model of unassessed status over the 1,189 applicable repository–capability cells of the primary panel (225 unassessed): odds ratios with 95% intervals from standard errors clustered by repository. Target size is the logarithm of the number of text files in the agent target, standardised. McFadden

R^{2}=0.09

Repeat	Score chg.	Cov. (pp)	Cells	Rated both	Range chg.
Repeat 1	$-$ 2.83	0.0	7 of 40	3	no
Repeat 2	$-$ 1.99	$-$ 5.0	4 of 40	2	no
Repeat 3	$-$ 0.79	+2.5	3 of 40	2	no
Repeat 4	$-$ 0.59	$-$ 12.8	9 of 40	4	no
Repeat 5	$-$ 0.39	+5.0	6 of 40	4	no
Repeat 6	+0.53	$-$ 2.5	2 of 40	1	no

Table S8: Panel-only repeats of eight repositories (same dossier, fresh panel of three judges; both panels scored with engine 0.3.0). Changes only: the change in the excluded score, the coverage change in percentage points, the cells that differ, those of them that both panels rated (rated in one panel and unassessed in the other otherwise), and whether the final-stripe range changed. Rows are labelled by index in order of score change; no scores or names are shown.

Population	1,482 accepted judge quotes behind aggregated ratings of the primary panel: 431 for gate capabilities and 1,051 for other capabilities.
Sample	150 quotes drawn from each stratum with seed 20261014 (300 in all).
Raters	Two raters with experience of Python agent code, who are not authors of the method and have not seen the results; a third rater adjudicates the second question.
Shown and withheld	Shown: the capability definition, the quote and 8 lines of context either side. Withheld: rating, judge, repository and stratum.
Questions	Does the quote concern the capability? Does it support the presence of the mechanism? Is the mechanism active as shipped?
Analysis	Agreement per question; the share of failing quotes per stratum with Wilson intervals ( Wilson, 1927 ) ; the ratings and stripes that change when ratings whose sampled quotes all fail are set to unassessed and re-scored. No pass threshold is set in advance.
Blinding	File paths in the sample can identify the repository, and the withheld key and the per-repository results are public in the data package; blinding therefore depends on raters not consulting them, which each rater confirms in writing.

Table S9: Quote-relevance audit: whether a verified quote supports its rating (Section 7.4 ). Specified but not run, because no independent raters were available.

Scripted variant	A scripted model replaced the language model for 5 repositories; an unmodified tracing library (Traccia, Naik et al., 2026 ; version 0.1.32, Traccia, 2026 ) wrote local traces that were passed to the engine; the observations were re-derived from the stored traces with the detectors of engine 0.3.0, under which only one repository (openai-agents-python) yields any.
Shim variant	A file-queue shim answered by Claude Sonnet responder subagents replaced only the model, so that the repositories’ own loop, dispatch and failure handling ran in a sandbox with in-process socket guards: with real tools restricted (11 repositories), with tools replaced by study stubs (17) and, for the 2 offensive-security repositories, with stubs and guards that raise on any foreign tool.
Runs	All 30 repositories were attempted, in 75 runs with 176 model calls; 20 yielded at least one span, a split that, in these runs, followed which model clients and frameworks the tracing library instrumented, not a property of the repositories.
Effect on the results	No final stripe and no demonstration limit changed (Section 7.6 ); under engine 0.2.2 one demonstration limit had changed, from 2 to 4, through the study’s own guardrail and a detector that engine 0.3.0 no longer uses.
Limitations	The responders were not the repositories’ intended models, the tools were study stubs, the restriction of the responders’ tools was a prompt rule only, and the issued prompts were not archived.
Data	Run statistics and comparison tables in the data package ( workflow2/ , workflow2_shim/ ).

Table S10: Runtime-evidence probe runs (Workflow 2), which test only the engine’s runtime-evidence path (Section 7.6 ). The evidence is probe-run E3, which cannot reach E4 or lift the demonstration limit; the numbers are not scores of the repositories.

Evidence	Score	Score stripes	Demonstration limit	Final stripes
All capabilities shown in code (E2)	71.4	5	2	2
Safety behaviours in the developer’s runs (E3), the rest in code	75.4	6	4	4
Everything in a re-runnable suite (E4)	95.0	7	6	6
Everything in the assessor’s sandbox (E5a)	100.0	7	6	6
Everything in independent testing (E5b)	100.0	7	7	7
As above, Runs bounded unassessed	100.0	7	2–7	1 (range 1–7)

Table S11: The gates worked through mechanically on a synthetic agent whose every non-emerging capability is rated 3; only the evidence of the safety behaviours (and of the rest) changes. Computed with the scoring module of engine 0.3.0. In the last row an unassessed safety capability turns the result into a range. No agent of the evaluation has evidence above E2, so this table is the only place where stripes 3 to 7 are reached.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 525 papers

Jul 30, 2026

Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness

AI agents are moving into production workflows where they retrieve information, call tools, maintain state, and act on behalf of users or organizations, but many release decisions still rely on capability signals, demos, or behavioral tests that do not show whether an agent is ready to operate under production constraints. Capability is therefore not production readiness. This paper introduces the ProofAgent Index (PAI), a governance readiness index for AI agents. PAI combines four dimensions of deployment evidence: Evaluation, Context, Compliance, and Governance. Evaluation measures observed behavior, Context measures the operating environment that shapes that behavior, Compliance measures alignment with applicable rules and controls, and Governance measures whether the organization can authorize, monitor, audit, and control the agent during operation. PAI is implemented inside ProofAgent Harness, an open source infrastructure for auditable AI agent evaluation and governance. Validation across two heavily regulated domains, healthcare and finance, shows that PAI carries held out readiness signal and separates higher risk from lower risk configurations. The results show that context engineering strongly changes reliability, capability improves behavior but does not determine readiness, and governance evidence must remain visible rather than averaged away. PAI reframes agent release from a faith based deployment decision into an auditable readiness decision.

Fouad Bousetouane

The University of Chicago, USA

AI Agent Reliability AI Agent Evaluation AI Agent Governance

Ask PDF

Sep 9, 2026

AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

Existing evaluation frameworks mostly assess only one part of AI agents, such as task completion (AgentBench) or security robustness (AgentDojo, ASB), rather than the complete pipeline of planning, tool selection, tool execution, memory and reasoning. Failures can occur at any stage, yet existing benchmarks rarely identify their precise source. AgentAudit evaluates the entire execution trace across ten capability, grounding, security and behavioural dimensions, namely instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security and execution integrity, combined with behavioural classification and failure attribution to pinpoint the exact stage responsible for an observed failure. AgentAudit can evaluate any LLM-based AI agent, since it attaches to the agent instead of replacing it. It reads only the recorded execution trace and does not interfere with how the agent runs, so it places no constraint on the agent's internal implementation. We evaluate five language models (OpenAI GPT-5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash) across nine capability and adversarial tasks. Claude Sonnet 5 and GPT-5 obtain the highest mean Composite Trust Scores (95.1 and 80.6 out of 100, respectively), while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash trail substantially (57.6, 45.7 and 22.6). All traces were scored by a single fixed judge model, which was itself one of the evaluated models, a limitation discussed in Section VII.E. More importantly, models with similar task-completion behaviour can diverge sharply in trustworthiness, as several non-frontier models are repeatedly classified Unsafe_Compliance on adversarial tasks rather than merely failing them, a distinction that pass/fail benchmarks cannot surface.

Shrey Nag, Sachita, Abhishek Kumar Singh +2

TIET · IGDTUW · GTBIT +1

AI Agent Reliability AI Agent Auditing AI Agent Evaluation AI Agent Security

Ask PDF

Oct 5, 2026

A Trust Layer for Agent Evaluation

Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second run. We introduce a Trust Layer for Agent Evaluation, an additive post-hoc framework that reports, beside each recorded score, whether it should be believed. It verifies four properties: whether the result is supported by the benchmark's own grading logic, whether a passing answer was earned through traceable computation, whether the agent's completion claim matches what occurred, and whether the result is stable under repeated execution. The first three use only saved artifacts; the fourth re-runs the agent. Model judgments only label evidence under majority voting; all verdicts follow deterministic rules and never modify the recorded score. Applied to five agent configurations on 108 tasks from Agents' Last Exam, every model shows passing runs with no traceable computation (at rates varying tenfold), confirmed false completion claims, and unstable results: 18-46% of tasks do not stay in one score band over five runs. Only 22.6% of recorded passes clear all four checks (95% CI 15.0-32.6, n=84). Measuring what an agent can do and verifying that it did it are different problems, and current benchmarks address only the first.

Mohammadreza Sediqin, Shivali Dalmia, Srinivasa Karthikeya Reddy Kovvuri +1

Centific Research New York, USA · Centific Research Seattle, USA · Centific Research Washington, USA

AI Agent Evaluation LLM Agent Verification AI Agent Reliability

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness

AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

A Trust Layer for Agent Evaluation