Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly. We present IMMACULATE, a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals. IMMACULATE selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead. Experiments on dense and MoE models show that IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead. Our code is published at https://github.com/guo-yanpei/Immaculate.
Figures & tables
Figure 1: The auditor sends random requests like other users and requires the model owner to prove the responses after they are received.
Figure 2: When fixing discrete selections di , the entire inference (green workflow) can be viewed as a continuous computation, as discrete selection no longer introduces branching uncertainty.
Figure 3: Logit TV-distance distribution of LLaMA3-70B. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.
Thp.
Request VC
LLaMA3-70B
0.3%
400×
Qwen3-32B
0.3%
400×
Qwen3-30B-A3B
0.9%
900×
DeepSeek-V2-Lite
1.0%
800×
Table 5: Overhead Evaluation. ‘Thp.’ denotes the inference throughput loss. ‘Request VC’ denotes the runtime of VC on CPU-TEE compared with GPU inference
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Global logit TV-distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.
Figure 5: Global logit KL divergence distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.
Figure 6: Global logit Top- K distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.
Figure 7: Global logit Top- K distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.
Per-token billing is now the standard pricing model for commercial large language models (LLMs), so the honesty of reported token counts directly affects what users pay. We show that this kind of billing is hard to audit by design: providers hide the model, the tokenizer, and the execution to protect their IP, mitigate jailbreaks, and preserve user privacy, which means an auditor can only inspect proofs the provider supplies. The audit therefore reduces to a consistency check on the provider's own reports. We call this a trust paradox: every audit must trust some artifact, but current frameworks trust exactly the ones a provider has the strongest reason to manipulate. We study three recent token auditing frameworks and show that a provider with ordinary commercial capabilities can systematically inflate billed token counts. In the most permissive setting, hidden reasoning usage can be inflated by 1,469% on average without detection. At current frontier reasoning prices, that turns a $100 honest bill into roughly a $1,569 bill on the same query. Even when the user can see the full reasoning string, tokenization ambiguity alone still allows 50.85% over-reporting below the detection threshold. These results suggest the problem is not in any specific auditor but in any audit whose evidence comes from the audited party. Restoring honest billing will require verification that ties reported token counts to evidence the provider does not control, such as trusted execution attestation, cryptographic proofs of inference, or third-party re-execution.
Shahinul Hoque, Jinghuai Zhang, Jinyuan Sun +1
University of Tennessee, Knoxville · University of California, Los Angeles
Cloud-based infrastructure has become the dominant platform for deploying large models, particularly large language models (LLMs). Fine-tuning and inference are increasingly delegated to cloud providers for simplified deployment and access to proprietary models, yet this creates a fundamental trust gap. Although cryptographic and TEE-based verification approaches exist, prohibitive proving costs and limited TEE memory prevent them from scaling to modern LLMs, leaving clients unable to practically audit these processes. This lack of transparency creates concrete security risks that can silently compromise service integrity. We present AFTUNE, an auditable and verifiable framework that ensures the computational integrity of cloud-based fine-tuning and inference. AFTUNE incorporates a lightweight recording and spot-check mechanism that produces verifiable traces of execution. These traces enable clients to later audit whether the fine-tuning and inference processes followed the agreed configurations, by verifying sampled execution blocks inside a TEE, each covering only a small portion of the model and the execution trace. Our evaluation shows that AFTUNE adds modest overhead and makes auditing practical for clients.
Heng Jin, Chaoyu Zhang, Hexuan Yu +4
Virginia Tech · Washington University in St. Louis
As large language models (LLMs) continue to grow in size, fewer users are able to host and run models locally. This has led to increased use of third-party hosting services. However, in this setting, there is a lack of guarantees on the computation performed by the inference provider. For example, a dishonest provider may replace an expensive large model with a cheaper-to-run weaker model and return the results from the weaker model to the user. Existing tools to verify inference typically rely on methods from cryptography such as zero-knowledge proofs (ZKPs), but these add significant computational overhead, and remain infeasible for use for large models. In this work, we develop a new insight -- that given a method for performing private LLM inference, one can obtain forms of verified inference at marginal extra cost. Specifically, we propose two new protocols which leverage privacy-preserving LLM inference in order to provide guarantees over the inference that was carried out. Our approaches are cheap, requiring the addition of a few extra tokens of computation, and have little to no downstream impact. As the fastest privacy-preserving inference methods are typically faster than ZK methods, the proposed protocols also improve verification runtime. Our work provides novel insights into the connections between privacy and verifiability in LLM inference.