As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \mbox{Ventor-QTest}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports average fidelity loss (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report extreme fidelity loss (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.
EFL: run-level centered-surprisal statistic and its observed empirical upper tail
Yes; neither is KL
Table 1: Latent objects and observed Ventor-QTest statistics. AFL and EFL name the two reported observables; they are not interchangeable estimators of a common loss.
Figure 1: Composite Ventor-QTest pipeline. Repeated requests to fixed contexts report AFL as a within-window mean coarsened-KL statistic, while independent long-sequence runs report EFL from the empirical upper tail of all observed centered-surprisal deviations. Both components require probabilities only from the trusted reference. Their outputs have different units and are reported jointly rather than summed into a scalar.
Figure 2: Auxiliary consistency check for AFL. The left panel contains 36 context–route cells from three logprob-capable route conditions. Text-count AFL statistics and the logprob-derived coarsened-KL comparator show descriptive agreement after context centring; with only 3!=6 route permutations, the exact route-level test is not confirmatory. The right panel shows the seven-route audit, in which the official control remains at the route-level null.
Route
Sr [95% CrI]
Logprob comparator
Holm p
Official self-check
−0.0007 [ −0.0144 , 0.0193 ]
0.0001
0.515
Aliyun 0731
0.5704 [ 0.4710 , 0.6765 ]
–
0.00035
Ark 0731
0.1591 [ 0.1129 , 0.2071 ]
–
0.00035
Baidu Qianfan 0731
0.1875 [ 0.1456 , 0.2365 ]
–
0.00035
StreamLake
0.2950 [ 0.2364 , 0.3533 ]
0.3075
0.00035
DeepInfra FP8
0.1185 [ 0.0782 , 0.1611 ]
–
0.00035
Table 2: AFL audit over seven routes. Sr is the mean bias-corrected coarsened-KL statistic over 12 held-out contexts; intervals are 95% posterior credible intervals after subtracting the null baseline. The final column reports Holm-adjusted p values.
Route
Median D500
SD
Empirical q0.95
Maximum
Official self-check
0.00545
0.00835
0.02484
0.03498
Aliyun 0731
0.00754
0.00819
0.02190
0.02924
Ark 0731
0.00679
0.00881
0.02633
0.02901
Baidu Qianfan 0731
0.00612
0.00534
0.01693
0.01975
StreamLake
0.00409
0.01390
0.04817
0.04969
DeepInfra FP8
0.00517
0.00806
0.01996
0.03497
Table 3: EFL summaries across seven route snapshots. All quantities summarize Dr,b(500) over 20 paired runs. Smaller values indicate less observed absolute centered-surprise drift under this probe; the statistic is not a KL estimator and does not provide a universal ordering of model fidelity. The empirical q0.95 uses NumPy’s linear interpolation and, with 20 runs, is descriptive.
Method
Target calls
Reference/setup
Audit objective and limitation
Gao et al. [ 17 ]
200–250 per single test; 10 tests/verdict
Matched 200–250 reference samples per test
Distributional equality/distance; adaptive routing not explicit
RUT [ 58 ]
100
10,000 local generations and logits
Rank uniformity on user-like prompts; probabilistic substitution model
LLMmap [ 39 ]
3–8 online
About 1,200 sourcing calls per reference model over train/test; trained classifier
Closed-set version identity; analyzes query-informed defenses
FLIPS [ 42 ]
8
40 extraction calls; trained classifier
Closed/open-set instance identity; assumes a non-adaptive target
KVV [ 34 , 36 ]
4,000
Official responses and labels
Functional behavior; a finite suite can be recognized
Ventor-QTest
Mean: 600 one-token; run level: 20 sequences
Mean: 12 shared probability calls; run level: 10,000 prefix-rescoring calls/route
Finite-probe mean statistic plus empirical run tail; no guarantee against perfect recognition
Table 4: Representative operating points for black-box verification methods. Counts are not matched-power comparisons. Ventor-QTest’s mean component is target-side inexpensive, whereas its run-level component is reference-intensive.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Route snapshot
Access path
Precision status
Target logprobs
Official self-check
Official API
not declared
yes
Aliyun 0731
Aliyun Model Studio
not declared
no
Ark 0731
Volcengine Ark
not declared
no
Baidu Qianfan 0731
Baidu Qianfan
not declared
no
StreamLake
OpenRouter
not declared
yes
DeepInfra FP8
OpenRouter
provider-declared FP8
no
Appendix
Table 5: Configuration fields preserved for the repeated-request audit. The requested revision is DeepSeek V4 Flash 0731 for every row. Optional target log probabilities are provider-controlled and never enter the text-count statistic.
Figure 3: GPQA-Diamond accuracy versus median run-level D500 . Error bars are Wilson 95% intervals; the route-level association is not statistically supported.
Route
Sr
Passes
Pass rate ↑
Errors ↓
Official
−0.0007
48
53.9%
22
Aliyun 0731
0.5704
45
50.6%
26
Ark 0731
0.1591
53
59.6%
14
Baidu Qianfan 0731
0.1875
38
42.7%
32
StreamLake
0.2950
52
58.4%
22
DeepInfra FP8
0.1185
46
51.7%
23
Appendix
Table 6: Complete Terminal-Bench 2.1 route results after targeted API-error replacement. Every denominator is 89 tasks.
Exposure quartile
Tasks
DO pass rate
Comparator pass rate
DO agent timeouts
Q1 (shortest)
23
82.6%
76.5%
4
Q2
22
31.8%
62.7%
10
Q3
22
18.2%
37.3%
16
Q4 (longest)
22
13.6%
28.2%
17
Appendix
Table 7: DigitalOcean success across task-exposure quartiles. The comparator is the same-task mean over the five matched routes used to construct exposure.
Minimum mean coarsened KL, δ
Repeats/context, M
Target calls/route
ν(M)
0.010
159
1,908
26
0.020
79
948
25
0.030
52
624
24
0.050
30
360
21
0.100
15
180
19
Appendix
Table 8: Fixed-sample planning for J=12 , 90% power, and family-wise level 0.05 across R=6 route comparisons. Calls are one-token target requests per route; 12 reference distributions are shared across routes.
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.
Commercial LLM APIs advertise a specific foundation model, but the served backbone may be silently substituted, quantized, or wrapped, for example to save deployment costs. All existing audits decide backbone identity from the text-output channel, which is structurally fragile for agentic APIs because modern serving stacks (OpenAI, Anthropic, Gemini, Cloudflare Workers AI, LangGraph) discard text and expose only structured actions when the model calls a tool, and provider-injected system prompts can distort text distributions enough that text-channel tests falsely accuse honest providers of substituting the claimed model. We observe that recent agentic post-training internalizes tool-use directly into the weights, opening a new audit channel that the serving stack still exposes and that is largely invariant to deployment context. We introduce Agentic Provenance (AgentProv), the first action-based identity audit for agentic LLM APIs: AgentProv fingerprints a deployed model through its categorical tool-call distribution and decides identity via an MMD permutation test. AgentProv catches every substituted model (100% on 630 evaluated checkpoint pairs), while holding the false-positive rate under system-prompt injection at 7% (vs. 67% for MET and 53% for RUT). On third-party API endpoints, AgentProv's disagreements with MET are consistent with an independent token-count side-channel that detects provider-injected system prompts.
Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly. We present IMMACULATE, a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals. IMMACULATE selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead. Experiments on dense and MoE models show that IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead. Our code is published at https://github.com/guo-yanpei/Immaculate.
Yanpei Guo, Wenjie Qu, Linyu Wu +7
National University of Singapore · Nanyang Technological University · Independent Researcher +1
Relay and reseller APIs mediate access to large language models (LLMs), but users cannot directly verify which model serves them. We introduce \name, a black-box auditing protocol based on stable factual recall near the knowledge boundary, including repeatable wrong answers. KBF generates benign, renewable probes and calibrates audit decisions against reference self-variation. Across 16 production endpoints, KBF detects all 155 economically relevant substitutions without rejecting any of the 16 same-reference controls. KBF remains robust to deployment variation and reaches 95% TPR at a substitution rate as low as 15% in mixed-routing simulations. Field audits flag 7 of 28 endpoints across six platforms as statistically inconsistent with their references. After reference enrollment, even GPT-6 Astra costs only approximately $0.67 per online audit at the recorded API prices.
Yijia Fang, Yiqing Feng, Bingyu Li +1
Beihang University China · Xidian University China · HKUST Hong Kong SAR, China