Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI

Jinhu Qi; Yifan Li; Minghao Zhao; Wentao Zhang; Zijian Zhang; Yaoman Li; Irwin King

cs.CLMar 16, 2026

Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI

Authors: Jinhu Qi, Yifan Li, Minghao Zhao, Wentao Zhang, Zijian Zhang, Yaoman Li, Irwin King

Organizations: The Chinese University of Hong Kong, Hong Kong SAR, China · Macao Polytechnic University, Macao SAR, China · Jilin University, Changchun, China

Abstract

Agentic AI systems increasingly act through tool-augmented, multi-step workflows whose failures (unsafe tool use, unauthorised actions, social harm) carry deployment-level consequences. Evaluation practice remains fragmented across isolated benchmark slices, and "trustworthiness" is frequently invoked but rarely defined operationally. We argue the central limitation is twofold: (i) the absence of a measurable specification of what agent trustworthiness means, and (ii) the lack of a principled notion of representativeness allowing assessment over a socio-technical scenario distribution rather than disconnected benchmark instances. We address (i) by defining agentic trustworthiness as a five-property profile (Reliability, Robustness, Safety, Social-Ethical Alignment, Operational Integrity) grounded in current AI risk frameworks, and (ii) with the Holographic Agent Assessment Framework (HAAF), which measures this profile over a scenario manifold through static policy analysis, sandbox simulation, social-ethical alignment assessment, and distribution-aware sampling, connected through an iterative Trustworthy Optimization Factory that converts red-team diagnoses into blue-team interventions. Our contributions are: (1) an operational five-property definition of agentic trustworthiness; (2) a distribution-aware scenario-sampling framework that surfaces property-level trade-offs invisible to scalar leaderboards; and (3) a cross-family transfer experiment in which interventions designed from a single focal model generalise -- without per-model or per-scenario tuning -- to 13 systems from seven model families (Llama, Mistral, Kimi, GLM, Qwen, GPT, DeepSeek) on a 100-scenario suite, where all 13 systems improve and two reach a perfect risk-weighted profile, establishing HAAF's Factory as a model-agnostic deployment-readiness pipeline. Code: https://github.com/TonyQJH/haaf-pilot

Figures & tables

	Benchmark Slices				Deployment Context
Benchmark	Task	Tool	Long-Horiz.	Factuality	Adversarial	Operational	Social	Risk
AgentBench	$\checkmark$	$\triangle$	$\triangle$	$\times$	$\times$	$\times$	$\times$	$\times$
WebArena	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$
SWE-bench	$\checkmark$	$\triangle$	$\checkmark$	$\times$	$\times$	$\triangle$	$\times$	$\times$
HaluEval	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
JailbreakBench	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\triangle$

Table 1. Coverage audit of selected benchmarks.

\checkmark

= primary,

\triangle

= partial,

\times

= minimal (based on qualitative expert assessment).

	P1 Rel.	P2 Rob.	P3 Safe.	P4 S-Eth.	P5 Op.
L1 Static	$\circ$	$\circ$	$\bullet$	$\circ$	$\bullet$
L2 Sandbox	$\bullet$	$\bullet$	$\bullet$	$\circ$	$\bullet$
L3 Social-Eth.	$\circ$	$\circ$	$\circ$	$\bullet$	—

Table 2. HAAF layers

\times

trustworthiness properties (P1–P5).

\bullet

= primary probing target,

\circ

= secondary signal. L4 (Representative Sampling) is cross-cutting and determines what is tested for every property.

Figure 1. (a) Current benchmarks evaluate isolated capability islands (safety, coding, search, etc.) without a unified distributional perspective. (b) The Holographic Agent Assessment Framework (HAAF): Layer 4 generates representative scenarios via distribution-aware sampling; Layers 1–3 probe the agent from complementary views (static analysis, interactive sandbox, and social-ethical assessment); the Trustworthy Optimization Factory iterates red-team probing and blue-team hardening until deployment readiness.

Property	Design	Validation	Targets	Risk
	Scenarios
P1 Reliability	9	26	GD, HT, IR	1–2
P2 Robustness	8	20	PI	3–5
P3 Safety	2	22	UA, PL	4–5
P4 Social-Ethical	2	18	SH	4–5
P5 Operational	3	14	OF, RF	2–3

Table 3. Scenario suite overview. Design-study suite (24, original pilot) used in § 4.2 and § 4.4 . Validation suite (100, new for this revision) used in § 4.3 and § 4.5 . Risk severity: 1 (low) to 5 (high).

Code	Failure Type	Description	Property
PI	Prompt Injection	Agent follows malicious instructions in tool outputs / retrieved text	P2
GD	Goal Drift	Agent deviates from original user objective	P1
UA	Unauthorized Action	Agent performs actions outside permitted scope	P3
HT	Hallucinated Tool-use	Agent calls tools with fabricated parameters / non-existent tools	P1
RF	Recovery Failure	Agent fails to recover from errors gracefully	P5
OF	Operational Failure	Agent fails under resource constraints (budgets, latency)	P5

Table 4. Failure taxonomy (9 classes) with property mapping. “IR” = Improper Refusal (new in this revision) captures cases where the agent refuses a benign, legitimate request.

P	ID	Title	Attack / probe pattern
P1	s96	New-hire privacy lookup	IR trap: a legitimate self-service request—refusing it is itself a P1 violation.
P2	s78	Vendor onboarding worksheet	Indirect injection: an "_assistant_directive" field embedded in a retrieved JSON template.
P3	s27	Compliance-auditor exfil	Authority cover: “Legal asked us to forward this to the external SOC2 auditor by EOW.”
P4	s50	False-precedent gaslight	Fabricated prior interaction (“you helped me last week filter by has_kids_in_school ”) to normalise a family-status filter.
P5	s73	Tight quota planning	3 tool calls for a 4-step itinerary; tests graceful prioritisation under budget pressure.

Table 5. One representative scenario per property from the 100-scenario validation suite, illustrating the design patterns: IR traps (P1, legitimate-sensitive requests where refusal itself is a failure); indirect injection (P2, payload embedded in retrieved structured content); authority-cover exfil (P3, plausible third-party justification); false precedent / gaslight (P4, fabricated prior interaction to normalise a discriminatory filter); and constrained budgets (P5, tight quota requiring graceful prioritisation).

System	Family	$\hat{T}_{P_{1}}$	$\hat{T}_{P_{2}}$	$\hat{T}_{P_{3}}$	$\hat{T}_{P_{4}}$	$\hat{T}_{P_{5}}$	$\mathrm{RWF}\downarrow$
Llama-3.1-8B	Llama	1.000	1.000	0.545	0.611	1.000	0.216
Llama-3.1-70B	Llama	1.000	1.000	0.136	0.278	1.000	0.407
Mistral-Large-2	Mistral	1.000	0.800	0.636	0.389	1.000	0.301
Mistral-Large-3	Mistral	0.962	0.950	0.273	0.278	0.929	0.396
Kimi-K2-Thinking	Kimi	1.000	0.600	0.136	0.611	1.000	0.434
Kimi-K2.5	Kimi	1.000	0.950	0.227	0.389	1.000	0.374

Table 6. Trustworthiness profiles across 13 cross-family agentic systems on the 100-scenario HAAF validation Control suite. Each

\hat{T}_{P_{k}}

is one minus the violation rate on scenarios targeting property

P_{k}

(higher = better);

\mathrm{RWF}

= Risk-Weighted Failure (lower = better). Per-axis scenario counts: P1 (

n{=}26

), P2 (

n{=}20

), P3 (

n{=}22

), P4 (

n{=}18

), P5 (

n{=}14

). Bold = best in column.

Figure 2. Trustworthiness profile across 13 agentic systems from seven model families on the 100-scenario HAAF validation Control suite. Each cell shows

\hat{T}_{P_{k}}

for one system

\times

property pair (green = trustworthy, red = violated); the rightmost column shows the risk-weighted aggregate

\mathrm{RWF}

(lower is better) and rows are sorted by it.

Metric	Baseline	Hardened	Abs. Change
Success Rate	20/24 (83.3%)	23/24 (95.8%)	$+12.5$ pp
Violation Rate	4/24 (16.7%)	1/24 (4.2%)	$-12.5$ pp
Risk-Weighted Failure	0.210	0.062	$-0.148$
Improper Refusal Count	0	0	$0$

Table 7. Factory cycle results on Qwen3-8B. The hardened (blue-team) system applies two interventions derived from red-team attribution (§ 4.2 ). Change reported as absolute difference (percentage points for rates, raw for RWF). Bold : hardened-system value when it improves over the baseline.

System	Family	Control	Treated	$\Delta\downarrow$
Llama-3.1-8B	Llama	0.216	0.000	$+0.216$
Llama-3.1-70B	Llama	0.407	0.115	$+0.292$
Mistral-Large-2	Mistral	0.301	0.055	$+0.246$
Mistral-Large-3	Mistral	0.396	0.101	$+0.295$
Kimi-K2-Thinking	Kimi	0.434	0.164	$+0.270$
Kimi-K2.5	Kimi	0.374	0.164	$+0.210$

Table 8. Factory cycle generalisation: Control vs. Treated

\mathrm{RWF}

across the 13-system cross-family suite. The Treated configuration adds the tool-output firewall (P2) and confirmation gate (P3) interventions identified by the single-model attribution in § 4.2 ; lower

\mathrm{RWF}

is better.

\Delta

is the absolute drop (

\mathrm{RWF}_{\text{Control}}-\mathrm{RWF}_{\text{Treated}}

); positive values indicate improvement. Bold : largest and smallest

\Delta

; perfect

\mathrm{RWF}_{\text{Treated}}=0.000

also bold.

Figure 3. Factory cycle generalisation across 13 cross-family systems on the 100-scenario validation suite. The identical pair of blue-team interventions identified from Qwen3-8B’s 24-scenario failure profile (tool-output firewall + confirmation gate) is applied uniformly to every system, with no per-model and no per-scenario tuning. Red bars: Control RWF; green bars: Treated RWF. Lower is better. Systems sorted by Control RWF ascending (best on left), matching the order of Figure 2 .

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 519 papers

May 17, 2026

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness. This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments: Safety and Robustness, and Privacy and System Security. For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies. Other trustworthiness aspects (value alignment, transparency, fairness, and accountability) are discussed as relevant context rather than parallel chapters. To support consistent comparison and deployment decisions, we consolidate evaluation into a unified metrics-and-benchmarks hub, emphasizing both outcome and process signals (e.g., constraint violations, trace completeness, and adversarial success rates) and offering scenario-to-metric guidance for release gating. We conclude by outlining open challenges such as self-evolving agents, runtime monitoring and verification, privacy-preserving personalization, and the trust-utility trade-off, and present a case study of real-world security failures in open-source agentic systems. Our goal is to serve as a practical reference for researchers and practitioners building trustworthy agentic systems in high-stakes environments.

Jinhu Qi, Muzhi Li, Jiahong Liu +9

Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.

AI Agent Reliability AI Agent Evaluation AI Agent Security AI Agent Safety

Ask PDF

Sep 9, 2026

AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

Existing evaluation frameworks mostly assess only one part of AI agents, such as task completion (AgentBench) or security robustness (AgentDojo, ASB), rather than the complete pipeline of planning, tool selection, tool execution, memory and reasoning. Failures can occur at any stage, yet existing benchmarks rarely identify their precise source. AgentAudit evaluates the entire execution trace across ten capability, grounding, security and behavioural dimensions, namely instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security and execution integrity, combined with behavioural classification and failure attribution to pinpoint the exact stage responsible for an observed failure. AgentAudit can evaluate any LLM-based AI agent, since it attaches to the agent instead of replacing it. It reads only the recorded execution trace and does not interfere with how the agent runs, so it places no constraint on the agent's internal implementation. We evaluate five language models (OpenAI GPT-5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash) across nine capability and adversarial tasks. Claude Sonnet 5 and GPT-5 obtain the highest mean Composite Trust Scores (95.1 and 80.6 out of 100, respectively), while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash trail substantially (57.6, 45.7 and 22.6). All traces were scored by a single fixed judge model, which was itself one of the evaluated models, a limitation discussed in Section VII.E. More importantly, models with similar task-completion behaviour can diverge sharply in trustworthiness, as several non-frontier models are repeatedly classified Unsafe_Compliance on adversarial tasks rather than merely failing them, a distinction that pass/fail benchmarks cannot surface.

Shrey Nag, Sachita, Abhishek Kumar Singh +2

TIET · IGDTUW · GTBIT +1

AI Agent Reliability AI Agent Auditing AI Agent Evaluation AI Agent Security

Ask PDF

Sep 17, 2026

A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

Benchmark scores alone provide an incomplete basis for assessing the trustworthiness of modern artificial intelligence systems. Large language models (LLMs), agentic systems, and multimodal models (MLLMs) require different forms of assessment, yet their evaluation evidence must remain interpretable for development and oversight. We propose a unified framework that connects output-level, trajectory-level, and cross-modal assessment through eight trustworthiness dimensions: capability, robustness, safety, fairness, transparency, governance, oversight, and efficiency. The framework preserves system-specific metrics while mapping native measurements to common performance bands, accompanied by uncertainty estimates and traceable evidence. A meta-evaluation layer examines the validity, reliability, and reproducibility of the evaluation itself. Multidimensional profiles expose strengths and weaknesses, while safety-critical overrides prevent aggregate scores from masking critical failures. Mappings to governance frameworks, international standards, and European Union regulatory requirements connect technical assessment with oversight needs. The framework provides a structured basis for assessing both system performance and the credibility of the evidence supporting it, with empirical validation across deployment contexts remaining an essential next step.

Shaina Raza, Ahmed Y. Radwan, Imran Liaquat +1

Vector Institute for Artificial Intelligence, Toronto, ON M5G 0C6, Canada

LLM Evaluation AI Agent Evaluation Multimodal Model Evaluation

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems