Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

Xiangfan Wu; Zonghao Ying; Huiyu Wu; Xing Zheng; Huangsheng Cheng; Xiaorong Shi; Jing Guo

cs.CRAug 17, 2026

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

Authors: Xiangfan Wu, Zonghao Ying, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo

Organizations: Tencent Zhuque Lab

Abstract

As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \mbox{Ventor-QTest}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports average fidelity loss (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report extreme fidelity loss (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.

Figures & tables

Object	Meaning	Directly estimated here?
$A_{r}$	Cross-window mean latent coarsened KL	No; most routes have one repeated-request window
$S_{r,b}$	AFL: within-window, across-context bias-corrected coarsened-KL statistic	Yes
$X_{r}(m)$	Expected excess maximum of latent window KL	No
$D_{r,b}$ , $\widehat{F}_{r}^{D}$	EFL: run-level centered-surprisal statistic and its observed empirical upper tail	Yes; neither is KL

Table 1: Latent objects and observed Ventor-QTest statistics. AFL and EFL name the two reported observables; they are not interchangeable estimators of a common loss.

Panel of Figure 1 — Figure 1: Composite Ventor-QTest pipeline. Repeated requests to fixed contexts report AFL as a within-window mean coarsened-KL statistic, while independent long-sequence runs report EFL from the empirical upper tail of all observed centered-surprisal deviations. Both components require probabilities only from the trusted reference. Their outputs have different units and are reported jointly rather than summed into a scalar.

Figure 2: Auxiliary consistency check for AFL. The left panel contains 36 context–route cells from three logprob-capable route conditions. Text-count AFL statistics and the logprob-derived coarsened-KL comparator show descriptive agreement after context centring; with only

3!=6

route permutations, the exact route-level test is not confirmatory. The right panel shows the seven-route audit, in which the official control remains at the route-level null.

Route	$S_{r}$ [95% CrI]	Logprob comparator	Holm $p$
Official self-check	$-0.0007$ [ $-0.0144$ , $0.0193$ ]	$0.0001$	0.515
Aliyun 0731	$0.5704$ [ $0.4710$ , $0.6765$ ]	–	0.00035
Ark 0731	$0.1591$ [ $0.1129$ , $0.2071$ ]	–	0.00035
Baidu Qianfan 0731	$0.1875$ [ $0.1456$ , $0.2365$ ]	–	0.00035
StreamLake	$0.2950$ [ $0.2364$ , $0.3533$ ]	$0.3075$	0.00035
DeepInfra FP8	$0.1185$ [ $0.0782$ , $0.1611$ ]	–	0.00035

Table 2: AFL audit over seven routes.

S_{r}

is the mean bias-corrected coarsened-KL statistic over 12 held-out contexts; intervals are 95% posterior credible intervals after subtracting the null baseline. The final column reports Holm-adjusted

p

values.

Route	Median $D_{500}$	SD	Empirical $q_{0.95}$	Maximum
Official self-check	0.00545	0.00835	0.02484	0.03498
Aliyun 0731	0.00754	0.00819	0.02190	0.02924
Ark 0731	0.00679	0.00881	0.02633	0.02901
Baidu Qianfan 0731	0.00612	0.00534	0.01693	0.01975
StreamLake	0.00409	0.01390	0.04817	0.04969
DeepInfra FP8	0.00517	0.00806	0.01996	0.03497

Table 3: EFL summaries across seven route snapshots. All quantities summarize

D_{r,b}(500)

over 20 paired runs. Smaller values indicate less observed absolute centered-surprise drift under this probe; the statistic is not a KL estimator and does not provide a universal ordering of model fidelity. The empirical

q_{0.95}

uses NumPy’s linear interpolation and, with 20 runs, is descriptive.

Method	Target calls	Reference/setup	Audit objective and limitation
Gao et al. [ 17 ]	200–250 per single test; 10 tests/verdict	Matched 200–250 reference samples per test	Distributional equality/distance; adaptive routing not explicit
RUT [ 58 ]	100	10,000 local generations and logits	Rank uniformity on user-like prompts; probabilistic substitution model
LLMmap [ 39 ]	3–8 online	About 1,200 sourcing calls per reference model over train/test; trained classifier	Closed-set version identity; analyzes query-informed defenses
FLIPS [ 42 ]	8	40 extraction calls; trained classifier	Closed/open-set instance identity; assumes a non-adaptive target
KVV [ 34 , 36 ]	4,000	Official responses and labels	Functional behavior; a finite suite can be recognized
Ventor-QTest	Mean: 600 one-token; run level: 20 sequences	Mean: 12 shared probability calls; run level: 10,000 prefix-rescoring calls/route	Finite-probe mean statistic plus empirical run tail; no guarantee against perfect recognition

Table 4: Representative operating points for black-box verification methods. Counts are not matched-power comparisons. Ventor-QTest’s mean component is target-side inexpensive, whereas its run-level component is reference-intensive.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Route snapshot	Access path	Precision status	Target logprobs
Official self-check	Official API	not declared	yes
Aliyun 0731	Aliyun Model Studio	not declared	no
Ark 0731	Volcengine Ark	not declared	no
Baidu Qianfan 0731	Baidu Qianfan	not declared	no
StreamLake	OpenRouter	not declared	yes
DeepInfra FP8	OpenRouter	provider-declared FP8	no

Appendix

Table 5: Configuration fields preserved for the repeated-request audit. The requested revision is DeepSeek V4 Flash 0731 for every row. Optional target log probabilities are provider-controlled and never enter the text-count statistic.

Figure 3: GPQA-Diamond accuracy versus median run-level

D_{500}

. Error bars are Wilson 95% intervals; the route-level association is not statistically supported.

Route	$S_{r}$	Passes	Pass rate $\uparrow$	Errors $\downarrow$
Official	$-0.0007$	48	53.9%	22
Aliyun 0731	$0.5704$	45	50.6%	26
Ark 0731	$0.1591$	53	59.6%	14
Baidu Qianfan 0731	$0.1875$	38	42.7%	32
StreamLake	$0.2950$	52	58.4%	22
DeepInfra FP8	$0.1185$	46	51.7%	23

Appendix

Table 6: Complete Terminal-Bench 2.1 route results after targeted API-error replacement. Every denominator is 89 tasks.

Exposure quartile	Tasks	DO pass rate	Comparator pass rate	DO agent timeouts
Q1 (shortest)	23	82.6%	76.5%	4
Q2	22	31.8%	62.7%	10
Q3	22	18.2%	37.3%	16
Q4 (longest)	22	13.6%	28.2%	17

Appendix

Table 7: DigitalOcean success across task-exposure quartiles. The comparator is the same-task mean over the five matched routes used to construct exposure.

Minimum mean coarsened KL, $\delta$	Repeats/context, $M$	Target calls/route	$\nu(M)$
$0.010$	159	1,908	26
$0.020$	79	948	25
$0.030$	52	624	24
$0.050$	30	360	21
$0.100$	15	180	19

Appendix

Table 8: Fixed-sample planning for

J=12

, 90% power, and family-wise level 0.05 across

R=6

route comparisons. Calls are one-token target requests per route; 12 reference distributions are shared across routes.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 621 papers

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes
Aug 30, 2026Xun Wang, Bihe Zhao, Michael Backes +2LLM Auditing Language Model Fingerprinting
CISPA Helmholtz Center for Information Security
IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation
Feb 26, 2026Yanpei Guo, Wenjie Qu, Linyu Wu +7Verifiable Outsourced Computation LLM Auditing
National University of Singapore · Nanyang Technological University · Independent Researcher +1
KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing
May 28, 2026Yijia Fang, Yiqing Feng, Bingyu Li +1LLM Auditing Language Model Fingerprinting
Beihang University China · Xidian University China · HKUST Hong Kong SAR, China

Aug 30, 2026J/K move · Enter open · S save

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

Xun Wang, Bihe Zhao, Michael Backes, Franziska Boenisch+1

CISPA Helmholtz Center for Information Security

LLM Auditing Language Model Fingerprinting

Open paper AskPDF arXiv

Commercial LLM APIs advertise a specific foundation model, but the served backbone may be silently substituted, quantized, or wrapped, for example to save deployment costs. All existing audits decide backbone identity from the text-output channel, which is structurally fragile for agentic APIs because modern serving stacks (OpenAI, Anthropic, Gemini, Cloudflare Workers AI, LangGraph) discard text and expose only structured actions when the model calls a tool, and provider-injected system prompts can distort text distributions enough that text-channel tests falsely accuse honest providers of substituting the claimed model. We observe that recent agentic post-training internalizes tool-use directly into the weights, opening a new audit channel that the serving stack still exposes and that is largely invariant to deployment context. We introduce Agentic Provenance (AgentProv), the first action-based identity audit for agentic LLM APIs: AgentProv fingerprints a deployed model through its categorical tool-call distribution and decides identity via an MMD permutation test. AgentProv catches every substituted model (100% on 630 evaluated checkpoint pairs), while holding the false-positive rate under system-prompt injection at 7% (vs. 67% for MET and 53% for RUT). On third-party API endpoints, AgentProv's disagreements with MET are consistent with an independent token-count side-channel that detects provider-injected system prompts.

Abstract

LLM Auditing

Explore similar work

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing