AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

Xun Wang; Bihe Zhao; Michael Backes; Franziska Boenisch; Adam Dziedzic

cs.CRAug 30, 2026

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

Authors: Xun Wang, Bihe Zhao, Michael Backes, Franziska Boenisch, Adam Dziedzic

Organizations: CISPA Helmholtz Center for Information Security

Abstract

Commercial LLM APIs advertise a specific foundation model, but the served backbone may be silently substituted, quantized, or wrapped, for example to save deployment costs. All existing audits decide backbone identity from the text-output channel, which is structurally fragile for agentic APIs because modern serving stacks (OpenAI, Anthropic, Gemini, Cloudflare Workers AI, LangGraph) discard text and expose only structured actions when the model calls a tool, and provider-injected system prompts can distort text distributions enough that text-channel tests falsely accuse honest providers of substituting the claimed model. We observe that recent agentic post-training internalizes tool-use directly into the weights, opening a new audit channel that the serving stack still exposes and that is largely invariant to deployment context. We introduce Agentic Provenance (AgentProv), the first action-based identity audit for agentic LLM APIs: AgentProv fingerprints a deployed model through its categorical tool-call distribution and decides identity via an MMD permutation test. AgentProv catches every substituted model (100% on 630 evaluated checkpoint pairs), while holding the false-positive rate under system-prompt injection at 7% (vs. 67% for MET and 53% for RUT). On third-party API endpoints, AgentProv's disagreements with MET are consistent with an independent token-count side-channel that detects provider-injected system prompts.

Figures & tables

Figure 1: An auditor queries a suspect API endpoint with a probe state $s$ ; $a$ denotes the resulting tool-call action and $\pi(a\mid s)$ its conditional action policy. AgentProv compares this policy against a trusted reference and detects deviations from the claimed model, exposing a silent backbone substitution.

Panel of Figure 2 — Figure 2: Our AgentProv pipeline. (a) Each template offers the model functionally-equivalent tools (e.g., gmail/outlook/yahoo) and we query $N$ times. (b) Each query becomes a one-hot row marking which tool was chosen and the column average is the fingerprint vector $\varphi$ . (c) The test statistic $T$ is the squared $L_{2}$ distance between reference and suspect fingerprints. (d) A permutation null calibrates the test. We reject if the observed $T$ exceeds the $\alpha{=}0.05$ threshold.

Category	$n$	MET	RUT	AgentProv
Cross-pairs	$630$	$100\%$	$100\%$	$100\%$
Self-pairs	$36$	$0/36$	$2/36$	$0/36$

Table 1: Identity tests on

36

checkpoints at

\alpha=0.05

. AgentProv uses the tool-selection probe at

K=20

templates,

N=50

samples per template. All methods reject every cross-pair; on self-pairs, MET and AgentProv never falsely reject while RUT does so on

2

models.

Family	$T_{\text{intra}}$	$T_{\text{cross}}$	$T_{\text{c}}/T_{\text{i}}$
R1-Distill	$5.43$	$15.32$	$2.8\times$
DeepSeek	$5.57$	$16.62$	$3.0\times$
Qwen3	$6.49$	$17.28$	$2.7\times$
Hammer-2.1	$6.80$	$15.99$	$2.4\times$
Qwen2.5	$7.29$	$15.13$	$2.1\times$
Llama	$7.40$	$15.43$	$2.1\times$

Table 2: Family cohesion in tool-selection action space on the

36

-model set (§ 4.2 ), AgentProv’s tool-selection probe at

K{=}20

N{=}50

T_{\text{intra}}

: mean MMD 2 across within-family pairs;

T_{\text{cross}}

: mean across between-family pairs (both in units of

10^{-4}

). Higher ratio

=

tighter internal clustering. Qwen3-Think has a single member ( qwen3-4b-thinking-2507 ) so

T_{\text{intra}}

is undefined.

	MET			RUT			Adapted LLMmap			AgentProv
Model	empty	short	long	empty	short	long	empty	short	long	empty	short	long
Llama-3.2-1B-Instruct	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Qwen3-1.7B	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Qwen2.5-1.5B-Instruct	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Gemma-3-1B-it	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$
Hammer2.1-1.5B	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 3: Per-model false-positive rate on the hidden-prompt control experiment (§ 4.3 ). Each cell reports the per-model decision at

\alpha=0.05

when the system prompt is replaced by the column variant; weights are unchanged, so every

\times

is a false positive on identity. The empty condition is a single whitespace (

1

token); short is “You are a helpful assistant.” (

\sim

5

tokens); long is a

\sim

50

-token provider-style customer-service prompt. AgentProv uses the tool-selection probe at

K=20

templates,

N=50

samples per template.

With detected hidden prompt
Endpoint	MET	RUT	AgentProv
Llama- $3.2$ - $3$ B	rej	rej	rej
Gemma- $3$ n-E $4$ B	rej	acc	acc
Llama- $3.2$ - $1$ B	rej	rej	acc
Qwen $3$ - $14$ B	rej	rej	acc
Llama- $3$ - $8$ B	rej	rej	acc

Table 4: Third-party API verification on

9

OpenRouter endpoints (§ 4.4 ), grouped by whether the OpenRouter token-count scan (Appendix A.5 ) detected a provider-injected hidden system prompt. Across these endpoints, AgentProv accepts

7

, MET accepts

1

, and RUT accepts

2

. Cells report decisions at

\alpha=0.05

. AgentProv uses the tool-selection probe with

N=50

samples per template; MET and RUT consume the same matched-provider API responses.

	$N$ samples per template
$K$ templates	$5$	$10$	$25$	$50$
$5$	$36.67\%$	$66.67\%$	$92.70\%$	$97.46\%$
$10$	$60.48\%$	$91.90\%$	$98.89\%$	$99.84\%$
$20$	$65.56\%$	$94.13\%$	$99.84\%$	$100.00\%$
$30$	$64.44\%$	$95.24\%$	$100.00\%$	$100.00\%$
$40$	$64.76\%$	$95.71\%$	$100.00\%$	$100.00\%$

Table 5: Template (

K

)

\times

sample-count (

N

) ablation for AgentProv (tool-selection probe,

B=1{,}000

\alpha=0.05

). Cells report the rejection rate (accuracy) on the

630

distinct-checkpoint pairs of the

36

-model set (§ 4.2 ). Accuracy generally increases along both axes, with small finite-sample fluctuations at low

N

, and saturates along a frontier: the

K=20

N=50

configuration reaches

100\%

, as does

K\geq 30

N\geq 25

. At fixed small

N

, adding templates beyond

K=20

yields only marginal gains (e.g.

N=5

rises from

65.6\%

K=20

68.6\%

K=60

); more samples per template is the more effective margin.

Probe surface	Self (acc)	Distinct (rej)
Original schema	$12/12$	$66/66$
Random neutral tool names	$12/12$	$66/66$
Paraphrased tool/parameter descriptions	$12/12$	$66/66$

Table 6: Tool-name and schema robustness.

Prompt knowledge	GPT-4o-mini $\rightarrow$ GPT-4o	GPT-5.4-mini $\rightarrow$ GPT-5.4
No mimicry	rej, $6.4{\times}10^{-5}$	rej, $5.8{\times}10^{-5}$
Identity	rej, $6.4{\times}10^{-5}$	rej, $5.7{\times}10^{-5}$
+Profile	rej, $7.5{\times}10^{-5}$	rej, $8.7{\times}10^{-5}$
+Few-shot	rej, $8.2{\times}10^{-5}$	rej, $1.05{\times}10^{-4}$

Table 7: Prompt-based mimicry.

Probe set	Decision	$p$ -value
Training	acc	$1.00$
Held-out	rej	$0.013$

Table 8: LoRA fine-tuning mimicry.

$q$	GPT-4o-mini $\rightarrow$ GPT-4o	GPT-5.4-mini $\rightarrow$ GPT-5.4
$0.0$	rej, $6.7{\times}10^{-5}$	rej, $6.0{\times}10^{-5}$
$0.2$	rej, $4.5{\times}10^{-5}$	rej, $4.1{\times}10^{-5}$
$0.3$	rej, $3.6{\times}10^{-5}$	rej, $3.1{\times}10^{-5}$
$0.4$	rej, $2.9{\times}10^{-5}$	acc, $2.4{\times}10^{-5}$
$0.5$	acc, $2.5{\times}10^{-5}$	acc, $2.0{\times}10^{-5}$
$0.7$	acc, $1.2{\times}10^{-5}$	acc, $8.4{\times}10^{-6}$

Table 9: Probabilistic substitution.

q

is the fraction of requests served by the genuine reference.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Probe setup	Option set
General tools	“Schedule a team sync for Thursday at 2pm.” Tools: create_event , schedule_meeting , add_event , book_calendar_event (identical descriptions).	{tool names, _no_call , _malformed , _other }
MCP servers	“Send a short greeting to [email protected].” Tools: mcp__gmail__send_email , mcp__outlook__send_email , mcp__smtp__send_mail (identical descriptions).	{tool names, _no_call , _malformed , _other }

Appendix

Table 10: Two representative templates of the tool-selection probe, showing the probe setup and the option set into which the model’s response is classified. The first uses general-tool naming variants; the second uses MCP server naming variants. The full template pool contains

K=60

such templates spanning both domains.

Family	Members evaluated
Qwen $2.5$	$0.5$ B, $1.5$ B, $3$ B, $7$ B; DistilQwen $2.5$ - $3$ B
Qwen $3$	$0.6$ B, $1.7$ B, $4$ B, $4$ B-Instruct- $2507$
Qwen $3$ -Think	$4$ B-Thinking- $2507$
Llama- $3.1$ / $3.2$	Llama- $3.1$ - $8$ B; Llama- $3.2$ - $1$ B, $3.2$ - $3$ B; SuperNova-Lite
Mistral	Mistral- $7$ B-Instruct, Mistral-Nemo, Ministral- $8$ B
DeepSeek base	DeepSeek-LLM- $7$ B, DeepSeek-Coder- $7$ B

Appendix

Table 11: The

36

open-weight checkpoints and nine closed-source OpenAI models evaluated. The five small checkpoints revisited under injected system prompts in § 4.3 are a subset (Llama-

3.2

1

B, Qwen

3

1.7

B, Qwen

2.5

1.5

B, Gemma-

3

1

B, Hammer

2.1

1.5

B). Third-party API endpoints audited in § 4.4 are listed in Table 4 .

Figure 4: Full $36\times 36$ pairwise MMD 2 matrix on the AgentProv $\cap$ MET $\cap$ RUT common set (§ 4.2 ), computed with AgentProv’s tool-selection probe at $K=20$ templates and $N=50$ samples per template. Models are grouped by family; black lines mark family boundaries. All $630$ distinct-checkpoint pairs reject at $\alpha=0.05$ .

Comparison	Pairs	Decision
Matching model: official OpenAI vs. OpenRouter	$9$	$9/9$ acc
Distinct official OpenAI models	$36$	$36/36$ rej
Official OpenAI models vs. local checkpoints	$324$	$324/324$ rej

Appendix

Table 12: Closed-model reseller control and controlled separation checks. The official OpenAI endpoint is the trusted reference for the route-matched comparisons.

	$N$ samples per template
$K$ templates	$5$	$10$	$25$	$50$
$5$	$0/36$	$0/36$	$0/36$	$0/36$
$10$	$0/36$	$1/36$	$0/36$	$0/36$
$20$	$0/36$	$0/36$	$0/36$	$0/36$
$30$	$0/36$	$0/36$	$0/36$	$0/36$
$40$	$0/36$	$0/36$	$0/36$	$0/36$

Appendix

Table 13: Self-test Type-I error (split-sample FPR) on the

36

-model set, for the same

K\times N

grid as Table 5 . Each cell reports the number of models for which the test rejects when both halves come from the same model (any rejection is a false positive on identity). The test is conservative: FPR is at or below the nominal

\alpha=0.05

rate at every cell, with the single false positive at

K{=}10

N{=}10

within the expected rate by chance across the

24

-cell grid.

Quantized deployment	AgentProv	MET	RUT
Qwen2.5-7B, 8-bit	acc	acc	acc
Llama-3.1-8B, 8-bit	acc	acc	acc
Qwen2.5-7B, 4-bit	acc	rej	rej
Llama-3.1-8B, 4-bit	rej	rej	rej

Appendix

Table 14: Decisions for quantized deployments against their full-precision references.

System-prompt condition	AgentProv	MET	RUT
Safety policy	$0/6$	$4/6$	$6/6$
Agent wrapper	$2/6$	$4/6$	$6/6$

Appendix

Table 15: False positives under safety-policy and agent-wrapper system prompts.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 621 papers

Agent Audit: A Security Analysis System for LLM Agent Applications
Mar 24, 2026Haiyue Zhang, Yi Nian, Yue ZhaoAI Agent Auditing LLM Agent Security
University of Southern California Los Angeles, California, USA
Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs
Aug 17, 2026Xiangfan Wu, Zonghao Ying, Huiyu Wu +4LLM Auditing Black-Box Model Auditing
Tencent Zhuque Lab
PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents
Oct 15, 2026Kshitij Mishra, Abhijith Sharma, Nils Lukas +1AI Agent Auditing LLM Agent Evaluation
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

Mar 24, 2026J/K move · Enter open · S save

Agent Audit: A Security Analysis System for LLM Agent Applications

Haiyue Zhang, Yi Nian, Yue Zhao

University of Southern California Los Angeles, California, USA

AI Agent Auditing LLM Agent Security Cybersecurity Static Code Analysis

Open paper AskPDF arXiv

What should a developer inspect before deploying an LLM agent: the model, the tool code, the deployment configuration, or all three? In practice, many security failures in agent systems arise not from model weights alone, but from the surrounding software stack: tool functions that pass untrusted inputs to dangerous operations, exposed credentials in deployment artifacts, and over-privileged Model Context Protocol (MCP) configurations. We present Agent Audit, a security analysis system for LLM agent applications. Agent Audit analyzes Python agent code and deployment artifacts through an agent-aware pipeline that combines dataflow analysis, credential detection, structured configuration parsing, and privilege-risk checks. The system reports findings in terminal, JSON, and SARIF formats, enabling direct integration with local development workflows and CI/CD pipelines. On a benchmark of 22 samples with 42 annotated vulnerabilities, Agent Audit detects 40 vulnerabilities with 6 false positives, substantially improving recall over common SAST baselines while maintaining sub-second scan times. Agent Audit is open source and installable via pip, making security auditing accessible for agent systems. In the live demonstration, attendees scan vulnerable agent repositories and observe how Agent Audit identifies security risks in tool functions, prompts, and more. Findings are linked to source locations and configuration paths, and can be exported into VS Code and GitHub Code Scanning for interactive inspection.

Abstract

LLM Auditing

Explore similar work

Agent Audit: A Security Analysis System for LLM Agent Applications

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents