IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

Yanpei Guo; Wenjie Qu; Linyu Wu; Shengfang Zhai; Lionel Z. Wang; Ming Xu; Yue Liu; Binhang Yuan

cs.CRFeb 26, 2026

IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

Authors: Yanpei Guo, Wenjie Qu, Linyu Wu, Shengfang Zhai, Lionel Z. Wang, Ming Xu, Yue Liu, Binhang Yuan, +2 more

Organizations: National University of Singapore · Nanyang Technological University · Independent Researcher · University of California, Berkeley

Abstract

Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly. We present IMMACULATE, a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals. IMMACULATE selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead. Experiments on dense and MoE models show that IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead. Our code is published at https://github.com/guo-yanpei/Immaculate.

Figures & tables

Figure 1: The auditor sends random requests like other users and requires the model owner to prove the responses after they are received.

Figure 2: When fixing discrete selections

{d_{i}}

, the entire inference (green workflow) can be viewed as a continuous computation, as discrete selection no longer introduces branching uncertainty.

Figure 3: Logit TV-distance distribution of LLaMA3-70B. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.

	Thp.	Request VC
LLaMA3-70B	$0.3\%$	$400\times$
Qwen3-32B	$0.3\%$	$400\times$
Qwen3-30B-A3B	$0.9\%$	$900\times$
DeepSeek-V2-Lite	$1.0\%$	$800\times$

Table 5: Overhead Evaluation. ‘Thp.’ denotes the inference throughput loss. ‘Request VC’ denotes the runtime of VC on CPU-TEE compared with GPU inference

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Global logit TV-distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.

Panel of Figure 5 — Figure 5: Global logit KL divergence distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.

Panel of Figure 6 — Figure 6: Global logit Top- $K$ distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.

Panel of Figure 7 — Figure 7: Global logit Top- $K$ distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.

Explore similar work

Token Inflation: How Dishonest Providers Can Overcharge for Large Language Model Usage
May 28, 2026Shahinul Hoque, Jinghuai Zhang, Jinyuan Sun +1LLM Auditing Black-Box Model Auditing
University of Tennessee, Knoxville · University of California, Los Angeles
Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI
Mar 8, 2026Heng Jin, Chaoyu Zhang, Hexuan Yu +4LLM Auditing LLM Inference
Virginia Tech · Washington University in St. Louis
Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs
Feb 19, 2026Arka Pal, Louai Zahran, William Gvozdjak +2LLM Inference LLM Auditing
Ritual · MIT · Columbia University

May 28, 2026J/K move · Enter open · S save

Token Inflation: How Dishonest Providers Can Overcharge for Large Language Model Usage

Shahinul Hoque, Jinghuai Zhang, Jinyuan Sun, Fnu Suya

University of Tennessee, Knoxville · University of California, Los Angeles

LLM Auditing Black-Box Model Auditing

Open paper AskPDF arXiv

Per-token billing is now the standard pricing model for commercial large language models (LLMs), so the honesty of reported token counts directly affects what users pay. We show that this kind of billing is hard to audit by design: providers hide the model, the tokenizer, and the execution to protect their IP, mitigate jailbreaks, and preserve user privacy, which means an auditor can only inspect proofs the provider supplies. The audit therefore reduces to a consistency check on the provider's own reports. We call this a trust paradox: every audit must trust some artifact, but current frameworks trust exactly the ones a provider has the strongest reason to manipulate. We study three recent token auditing frameworks and show that a provider with ordinary commercial capabilities can systematically inflate billed token counts. In the most permissive setting, hidden reasoning usage can be inflated by 1,469% on average without detection. At current frontier reasoning prices, that turns a $100 honest bill into roughly a $1,569 bill on the same query. Even when the user can see the full reasoning string, tokenization ambiguity alone still allows 50.85% over-reporting below the detection threshold. These results suggest the problem is not in any specific auditor but in any audit whose evidence comes from the audited party. Restoring honest billing will require verification that ties reported token counts to evidence the provider does not control, such as trusted execution attestation, cryptographic proofs of inference, or third-party re-execution.

Abstract

Explore similar work

Token Inflation: How Dishonest Providers Can Overcharge for Large Language Model Usage

Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI

Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs