cs.LGAug 2, 2026

Caliber: Cross-Architecture Extraction-Cost Control for Score-Returning APIs

Authors: Chi Wang, Hanwen Wang, Yu Xia, Zihan Wang, Guangdong Bai

Abstract

We present Caliber, an output-perturbation defense against model extraction that formulates noise selection as a calibration problem: how much the defense degrades the supervision signal used to train a surrogate, and the provable per-input query cost of recovering the clean logits. To defend against an attacker that uses returned scores for knowledge distillation, Caliber adds independent and identically distributed Gaussian noise to the internal logits. We establish two properties of the resulting perturbed predictions. Monotone agreement degradation: When the clean logits have a unique maximizer, agreement with the clean prediction decreases strictly with the noise scale, so every target in (1/K,1)(1/K,1) corresponds to a unique positive scale; task accuracy is bounded by computable lower and upper envelopes. Per-input recovery cost: We derive a closed-form minimax lower bound on the repeated queries needed to recover the clean logits for a fixed input. Caliber normalizes noise variance by the squared median top-two logit margin and fits the resulting noise-utility relationship with a logistic curve, either per model or shared within a task. Across more than thirty model-dataset combinations, per-model calibration achieves mean absolute relative errors of 0.6-1.4%. End-to-end experiments show that surrogate performance generally tracks the configured degradation, while fixed-input averaging follows the expected variance reduction.

Explore similar work

CardsList
May 11, 2026cs.LG

Identified-Set Geometry of Distributional Model Extraction under Top-K Censored API Access

Modern LLM APIs often reveal only top-KK logit scores and censor the remaining vocabulary. We study the per-position distribution-recovery limits of this access model. For censoring threshold ττ, the compatible teacher distributions form an identified set whose total-variation diameter is exactly UK=(V−K)exp⁡(τ)/(ZA+(V−K)exp⁡(τ))U_K=(V-K)\exp(τ)/(Z_A+(V-K)\exp(τ)), where ZAZ_A is the observed partition function. For KL recovery, we give a computable binary-endpoint lower bound and an asymptotically matching small-ambiguity upper bound, with an extension to reference-aware attackers. Experiments on a Qwen3 math-reasoning teacher reveal a layered extraction hierarchy: on-task top-KK distillation recovers 12% of private capability, full-logit distillation recovers 56% despite 99% KL closure, and generation-based extraction recovers 96%. Top-KK censoring therefore limits per-position distribution recovery but does not by itself prevent capability extraction, separating fidelity from transfer in prompt-only logit distillation.
Wenhua Nie, ZiCheng Zhu, Jianan Wu +3
Top-KApplication Programming InterfacesRepresentational Geometry
AskPDF
Jun 23, 2026cs.CR

What Does It Mean to Break a Distillation Defense?

Black-box LLMs (accessible only via API) are vulnerable to distillation attacks, in which an attacker queries the model and trains a student on its outputs. A recent line of work proposes output perturbation defenses that modify the teacher's output to reduce student performance while preserving utility for legitimate users. As a relatively new family of approaches, output perturbation defenses lack a shared threat model, making it difficult to compare them, reason about composing them with other attacks, or evaluate their robustness against realistic adversaries. This underspecification matters beyond technical evaluation: when defenses are deployed to protect intellectual property or justify regulatory compliance, an imprecise threat model can create a false sense of security. We propose a threat model framework that describes attackers along three dimensions: a query budget, a data budget, and an interface profile that captures how attackers interact with the API. Using antidistillation sampling as a case study, we show that whether the defense is considered effective depends on the assumed threat model. We argue that future work on distillation defenses, along with any governance or policy frameworks built around them, should explicitly specify and stress-test attacker capabilities along our three dimensions.
Lena Libon, Pura Peetathawatchai, Michael Aerni +2
Threat ModelsGeometry-Aware Distillation
AskPDF
Jun 4, 2026cs.CR

An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic

Large language models (LLMs) are increasingly deployed through hosted APIs, making model extraction a practical threat to model ownership and service security. However, individual extraction queries often resemble benign requests, and existing evaluations often focus on single-query anomaly scoring or pure benign-versus-attacker user settings. We formulate model extraction monitoring as benign-calibrated traffic-window distribution testing and show that an embarrassingly simple detector is effective: embed incoming queries into a semantic space and test whether their aggregate distribution deviates from historical benign traffic. We instantiate the detector with maximum mean discrepancy (MMD), using only benign-vs-benign comparisons to set the decision threshold. We evaluate on fourteen attacker-normal query pairs from four extraction scenarios and compare with adapted PRADA, SEAT, CAP, DATE, and marginal Mahalanobis baselines. Across three random seeds, MMD achieves 0.3% benign FPR, 100.0% pure-attacker TPR, 90.5% average TPR over attacker fractions, and 95.1% balanced accuracy. These results show that benign-calibrated distribution testing is a strong empirical baseline for model extraction detection in both user-level and mixed multi-user LLM API traffic. Code is released at: https://github.com/LabRAI/mmd-llm-mea-detection.
Shuze Liu, Qianwen Guo, Yushun Dong
Adversarial Attack StrategiesAnomaly Score
AskPDF