The Limits of Query Monitoring against Distributed Model Extraction

Snigdha Chandan Khilar

cs.CROct 9, 2026

The Limits of Query Monitoring against Distributed Model Extraction

Authors: Snigdha Chandan Khilar

Organizations: Independent Researcher

Abstract

Model providers now face distillation campaigns run through thousands of accounts, each of which looks like an ordinary user. When network and payment metadata cannot link those accounts, the only evidence left is what they ask. We study what a provider can and cannot learn from those queries alone, using one million real chat turns for development and 17 later weeks that no design choice touched for a single frozen evaluation. The answer is set less by anything specific to extraction than by benign coordination: a campaign is visible only if it is more distinctive than the most distinctive group of ordinary users in the same week. We make this precise, show that a small but positive difference between campaign and users buys a detection time rather than detectability, and identify the campaign that is hardest to detect. Empirically, a calibrated monitor catches naive distributed extraction that per-account monitors miss (0.86-1.00 of weeks), and one variant - selecting accounts that move together after discounting how topical each one is - still catches campaigns padded with natural text (0.50-0.96 of weeks) where the other two spectral detectors catch none. Distilling each campaign into a small student shows that, in our testbed, campaigns that evaded every detector kept at most 42% of the capability of a detectable one. The limits are equally concrete: a campaign that ramps up slowly poisons every detector's calibration, benign evaluation pipelines look exactly like distillation, we found no deployable detector at colluder shares of 2.5% or below, and campaigns that relay real users' requests may be invisible to any query-only monitor.

Figures & tables

Figure 1: The paper at a glance. (a) Capability a small student gains from 3,000 queries of each campaign type (GSM8K points; bars span two seeds) against how often hollowed selection, our most robust detector, catches the campaign each week (untouched weeks for the dataset, padded and least-detectable campaigns; development weeks for the two written ones). Every campaign in the shaded band evaded detection and kept at most 42% of the dataset campaign’s gain. (b) Three frozen detectors on 17 weeks that no design choice had touched; missing bars are zero. Only hollowed selection still catches campaigns padded with natural prompts; none catches the least-detectable campaign.

Figure 2: The monitor. Each account’s queries become a topic fingerprint, compared with last month’s traffic; hollowing keeps only how accounts co-vary; the most coherent group is scored against every past benign week.

Campaign	Two-scale	Global stream	MMD	Spectral	Per-account / HC / Fraudar
MetaMathQA (generated math)	0.93	0.71	0.60	0.19	$\leq$ 0.01
Magicoder (generated code)	0.62	0.46	0.32	0.00	0.00
Alpaca (generated, broad)	0.01	0.00	0.01	0.00	0.00
Narrow cluster of real prompts	0.12	0.04	0.27	0.00	0.00
Random real prompts	0.01	0.00	0.01	0.00	0.00
Realised false alarms	0.7%	4.1%	11.6%	7.8%	5.3 / 7.5 / 2.9%

Table 1: Weekly paired detection at deployed thresholds (nominal 1%), whole weekly populations,

K=50

volume-matched colluders, 24 development weeks. The last row is the realised false-alarm rate of each detector.

Detector	Campaign	Weekly detection	First weekly alarm	Anytime-valid
Two-scale	Dataset	0.79–0.85 / 0.86	week 1 (8.8) / week 1 (8.8)	week 3 (12.7) / week 3 (12.7)
	Half padding	0.21 / 0.00	week 3 (9.5) / none	week 15 (15.5) / none
Hollowed	Dataset	1.00 / 1.00	week 1 (8.8) / week 1 (8.8)	not run
	Half padding	0.44–0.48 / 0.50–0.57	week 2 (7.9) / week 1 (5.6)	not run
Multi-outlier	Dataset	1.00 / 0.97	week 1 (8.8) / week 1 (8.8)	not run / week 3 (12.7)
	Half padding	0.36–0.42 / 0.00	week 2 (7.9) / none	not run / 25% of runs

Table 2: Capability held at first detection: GSM8K gain (points) of the extraction-regime student at the number of queries sent by the median week of first detection, at 1,640 queries a week. Each cell: development weeks / untouched weeks. “none”: no alarm within 16 weeks.

Setting		Two-scale	Hollowed	Multi-outlier	Volume scan
MiniLM, 64–252 accounts	False alarms	1.8%	1.9%	1.9%	1.5%
	Dataset (MetaMathQA)	0.86	1.00	0.97	0.61
	Half padding	0.00	0.57	0.00	0.37
	Least detectable (tilted)	0.00	0.04	0.00	0.10
	Random real prompts	0.00	0.00	0.00	0.00
	GSM8K sweep, 10 / 3 accounts	0.87 / 0.88	1.00 / 0.34	0.97 / 0.93	0.46 / 0.46

Table 3: The frozen pipeline on 17 untouched weeks: weekly paired detection at deployed thresholds, nominal 1% (conformal over all prior weeks; eight preceding weeks for the volume scan). Development-week values for comparison: dataset 0.79–0.93 / 1.00 / 1.00; half padding 0.21 / 0.44–0.48 / 0.36–0.42.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The barrier in a controlled setting. Synthetic benign users with topic clusters and trends of increasing strength; campaigns of 10,000 extraction queries. A single tight benign cluster per window erases detection power (at strength 0.25 for 100 accounts; at 1,000 accounts power falls at 0.05 and vanishes at 0.1). Details in Appendix E .

Figure 4: Calibration on real traffic. (a) False alarms on held-out weeks for three calibration schemes at a nominal 5%. Calibrating once on early weeks fails; resampling whole windows from the four preceding weeks, each whitened against its own lagged reference, restores the nominal rate in 100-account windows but not in 500-account windows. (b) Per-detector false alarms at a nominal 1% over whole weekly populations, with week-clustered intervals: only the two-scale test is calibrated under rolling calibration.

Figure 5: Scale, with the campaign fixed at 1,640 queries over 50 colluders, in populations pooled from account-weeks across the year (oracle thresholds at 1% false alarms). The detector’s own group selection fails as colluders become rarer, although the true colluder set still scores above the threshold at every size; scoring a group from every outlier eigenvector of

R_{c}

closes the gap, but only with oracle thresholds.

$A$ , $K$	Strategy	Suite	$s=0$	0.05	0.1	0.25	0.5	1
100, 20	entropy	geometry	1.00	1.00	1.00	0.00	0.00	0.00
100, 20	entropy	task-aware	1.00	1.00	1.00	1.00	0.12	0.17
100, 20	$k$ -center	geometry	1.00	1.00	1.00	0.00	0.00	0.00
100, 20	$k$ -center	task-aware	1.00	1.00	1.00	0.79	0.04	0.08
1000, 100	entropy	geometry	1.00	0.67	0.04	0.00	0.04	0.04
1000, 100	entropy	task-aware	1.00	0.96	0.67	0.33	0.04	0.04

Appendix

Table 4: Power against 10,000-query campaigns as benign structure strength

s

grows (24 windows per cell, window-resampled calibration). Geometry suite: spectral, Higher Criticism and global tests on the victim embedding. Task-aware suite: the same aggregations applied to each account’s share of low-margin queries.

$A$ , $K$	Campaign	$N$	Share	Fixed	Rolling	Rolling + lag
100, 20	random natural	20,000	0.94	0.00	0.00	0.00
100, 20	broad capability (1 of 10)	5,000	0.81	0.03	0.04	0.21
100, 20	broad capability	20,000	0.94	0.82	0.90	0.93
100, 20	narrow capability (1 of 30)	1,000	0.46	0.00	0.00	0.03
100, 20	narrow capability	5,000	0.81	0.78	0.93	0.94
100, 20	broad, 50% natural padding	20,000	0.94	0.25	0.04	0.38

Appendix

Table 5: Paired detection in small monitored populations (8 held-out weeks, 3 campaigns

\times

3 windows per week;

\alpha=5\%

). Share is the campaign’s fraction of window traffic.

	AUC / TPR at 1% / TPR at 5%
Campaign	Two-scale	Global stream	MMD (Liu et al.)	QC
MetaMathQA (generated)	0.88 / 0.80 / 0.81	0.97 / 0.70 / 0.74	0.94 / 0.58 / 0.62	0.94 / 0.64 / —
Human GSM8K questions	0.96 / 0.95 / 0.95	0.99 / 0.88 / 0.94	0.97 / 0.73 / 0.80	1.00 / 0.93 / —
Half natural, mixed into every account	0.68 / 0.24 / 0.39	0.75 / 0.05 / 0.11	0.71 / 0.07 / 0.14	0.86 / 0.31 / —
Real follow-up after every prompt	0.68 / 0.24 / 0.40	0.75 / 0.06 / 0.11	0.72 / 0.11 / 0.20	0.85 / 0.27 / —
Random natural prompts	0.48 / 0.00 / 0.07	0.39 / 0.00 / 0.04	0.39 / 0.02 / 0.02	0.52 / 0.10 / —

Appendix

Table 6: Primary comparison: detection at oracle-matched realised false-alarm rates (thresholds from the pooled held-out null windows, which a deployed system cannot set), with AUC. Weekly populations,

K=50

volume-matched colluders, about 1,640 queries, 24 test weeks. QC is the repaired query-cluster detector (per-cell lagged scoring with shrinkage; see “Query-level pooling” below). Bold: best AUC and best TPR at 1% in each row.

Campaign	Two-scale	Global	MMD
MetaMathQA, one prompt per source problem	0.88	0.68	0.57
MetaMathQA, four variants per source problem	0.80	0.71	0.60
WildChat broad cluster, random prompts	0.01	0.01	0.07
WildChat broad cluster, farthest-point deduplicated	0.04	0.02	0.07

Appendix

Table 7: Mechanism probes (weekly populations,

K=50

, volume-matched). Rephrased variants barely weaken the test, and deduplicating natural prompts does not create a signal.

Campaign	Selection on $C$	Selection on $R$	$R$ on topic cores
MetaMathQA	0.86 (0.92)	1.00 (1.00)	1.00 (1.00)
Half natural, mixed	0.21 (0.66)	0.94 (1.00)	0.95 (1.00)
Real follow-up after every prompt	0.23 (0.66)	0.93 (1.00)	0.93 (1.00)
Human GSM8K questions	0.93 (0.97)	1.00 (1.00)	1.00 (1.00)
Random natural prompts	0.02 (0.52)	0.00 (0.51)	0.00 (0.48)
GSM8K evaluation sweep, 3 accounts	0.75 (0.87)	0.34 (0.61)	0.02 (0.54)

Appendix

Table 8: Selection on the hollowed matrix

R=\Delta^{-1/2}(YY^{\top}-\Delta)\Delta^{-1/2}

(as

R_{c}

but without centring) versus

C

, both scored with the two-scale statistic. Oracle TPR at a matched 1% false-alarm rate (AUC in parentheses); weekly populations,

K=50

volume-matched colluders. Deployed false alarms: 0.8% for

C

, 8.4% for

R

, 7.6% for the core variant.

Weekly, deployed (conformal)
	MetaMathQA	Mixed	Follow-up	Random	Sweep, 3 accts	Sweep, 10 accts
Selection on $C$	0.82	0.00	0.02	0.00	0.80	0.82
Selection on $R_{c}$	1.00	0.48	0.46	0.00	0.42	0.97
Pooled, oracle TPR at 1%
	$A=250$	1,000	2,500	5,000	10,000
$C$ , unpadded	0.97	0.75	0.28	0.12	0.05

Appendix

Table 9: Selection on the centred hollowed matrix

R_{c}

against

C

. Top: weekly populations, conformal thresholds over all prior weeks (no false alarm in 960 held-out windows from 24 weeks for either; week-level 95% upper bound about 12%). Bottom: pooled populations with the campaign fixed at 1,640 queries over 50 colluders, oracle TPR at 1%, and the share of colluders in the group

R_{c}

selects. The

C

row repeats cells of Table 10 from a separate run (0.28 against 0.42 at

A=2{,}500

), a difference within run-to-run variation.

$A$	Colluder share	Null median	Null 99%	Detector TPR at 1%	True colluder set above threshold
250	20%	1.27	3.28	1.00	1.00
1,000	5%	1.21	3.16	0.80	1.00
2,500	2%	1.05	4.35	0.42	1.00
5,000	1%	0.94	3.18	0.15	1.00
10,000	0.5%	0.61	2.95	0.00	1.00

Appendix

Table 10: Scale with the campaign fixed (1,640 MetaMathQA queries over

K=50

colluders) in pooled populations of benign account-weeks, oracle thresholds. “True colluder set” scores the campaign’s own accounts with the two-scale statistic against the same null threshold.

$A$	Colluders	unpadded	padded	unpadded	padded	Colluder rank (unp./pad.)	$\rho$ vs needed
		Top eigenvector		$T_{\max}$
250	20%	1.00	1.00	1.00	1.00	0 / 0	0.49 vs 0.013
1,000	5%	1.00	0.07	1.00	1.00	0 / 1	0.58 vs 0.026
2,500	2%	0.00	0.00	1.00	0.83	1 / 2	0.58 vs 0.041
5,000	1%	0.00	0.00	0.93	0.77	2 / 3	0.60 vs 0.058
10,000	0.5%	0.03	0.03	1.00	0.17	3 / 8	0.61 vs 0.082

Appendix

Table 11: Multi-outlier selection at scale: oracle TPR at a matched 1% false-alarm rate in pooled populations, campaign fixed at 1,640 queries over 50 colluders. Diagnostics: rank of the eigenvector of

R_{c}

most aligned with the colluders (0 = top eigenvector), and their mean pairwise cosine

\rho

against the BBP requirement

\sqrt{A/r}/K

Figure 6: The price of stealth depends on whether the student already has the skill. (a) Extraction regime: SmolLM2-360M, which starts at 3.5% on GSM8K, keeps gaining with budget; padding half the campaign costs 3–4 points at every budget, and the least-detectable (tilted) campaign gains almost nothing (bars: range over two seeds). (b) Elicitation regime: Qwen2.5-0.5B on GSM8K (three seeds) saturates within 100 queries, and padding costs nothing measurable.

	Detection at 1,641 queries			GSM8K gain (points)
Campaign	Two-scale	Global	MMD	1,641 queries	7,550 queries
MetaMathQA, one prompt per source problem	0.85 [0.74, 0.94]	0.65	0.56	+11.4	+12.6
Half natural prompts mixed into every account	0.21 [0.10, 0.37]	0.12	0.08	+12.0	+11.0
Each prompt followed by a real follow-up turn	0.24 [0.11, 0.38]	0.12	0.08	+13.4	—
Natural WildChat cluster	0.01	0.01	0.07	—	+6.0

Appendix

Table 12: Detection against what the campaign extracts. Detection: paired power at 1% (weekly populations,

K=50

, 1,641 queries). Distillation: Qwen2.5-0.5B (base) fine-tuned with LoRA on the campaign’s (prompt, answer) pairs, GSM8K accuracy on the first 500 test problems (one seed; standard error about 2.2 points). Base model: 34.4%.

Queries	All math	Half natural, mixed	Camouflaged (natural-looking math)
100	$+9.8\pm 2.1$	$+9.8\pm 1.8$	$+9.3\pm 0.7$
300	$+7.5\pm 0.5$	$+10.2\pm 2.1$	$+8.7\pm 2.3$
900	$+8.6\pm 1.2$	$+10.1\pm 1.0$	$+11.2\pm 1.9$

Appendix

Table 13: GSM8K gain over the base model (points; mean

\pm

standard deviation over three seeds) for campaigns of equal query budget. All models are evaluated on the same 400 problems.

	Queries	All math	Half natural, mixed	Paired cost
SmolLM2-360M, GSM8K	1,000	$+7.5\pm 0.0$	$+3.9\pm 0.6$	$-3.6$
	3,000	$+10.4\pm 0.1$	$+7.6\pm 0.6$	$-2.8$
	10,000	$+15.9\pm 0.9$	$+12.1\pm 1.1$	$-3.8$
SmolLM2-360M, tilted (least detectable)	3,000	—	$+2.2\pm 0.2$	$-8.2$
	10,000	—	$+1.5\pm 0.2$	$-14.4$
Qwen2.5-1.5B, MATH-500	1,000	$+12.0$	$+11.4$	$-0.6$

Appendix

Table 14: The extraction regime: gain over the base model (points) for campaigns of equal query budget. SmolLM2-360M (base 3.5% on GSM8K, 400 problems), full fine-tuning, mean

\pm

standard deviation over two seeds; Qwen2.5-1.5B with LoRA on MATH-500 (base 22.8%), one seed. Tilted rows: half on-task WildChat prompts and half off-task (Proposition 4 ); the cost is against the all-math campaign of the same size.

Population	Campaign	Two-scale	Hollowed	Multi-outlier
Development weeks, 64–364 accounts	Dataset	0.79–0.85	1.00	1.00
(deployed)	Half padding	0.21	0.44–0.48	0.36–0.42
	Least detectable	0.00	0.01	0.01
Untouched weeks, 64–252 accounts	Dataset	0.86	1.00	0.97
(deployed)	Half padding	0.00	0.50–0.57	0.00
	Least detectable	0.00	0.04	0.00

Appendix

Table 15: What works where: weekly detection of each campaign type by population and detector. Deployed thresholds in weekly populations; oracle thresholds in pooled ones, where no deployed threshold reached 1% false alarms.

Embedding	CIFAR-10: victim penultimate layer, PCA-32. WildChat: all-MiniLM-L6-v2, PCA to $d\in\{32,64,128,384\}$ , rescaled to total variance 4.
Fourier modes	$m=400$ complex modes (CIFAR-10 and the small-population WildChat protocol); $m=300$ modes with augmented real whitening (realistic protocol). Bandwidths $h\in\{2,4,8\}$ with Bonferroni correction (CIFAR-10), $h=4$ (WildChat), and $h=4$ coarse with $h=0.5$ fine for the two-scale test.
Whitening floor	eigenvalues below $10^{-4}$ of the maximum discarded.
Reference	CIFAR-10: 300,000 benign queries. WildChat: up to 100,000 benign turns from the four preceding weeks (lagged).
Calibration	CIFAR-10: 1,000 null worlds per configuration, Bonferroni over bandwidths. WildChat: 40 null windows per week, thresholds from the 8 preceding weeks (realistic protocol) or 250 windows from the 4 preceding weeks (small populations); conformal thresholds over all prior weeks for hollowed and multi-outlier selection.
MMD baseline	unbiased MMD 2 on normalised MiniLM embeddings, Gaussian kernel with median-heuristic bandwidth, window sample of 2,000 queries against 20 reference batches of 500.
Fraudar baseline	account $\times$ topic count matrix (100 $k$ -means topics), column weights $1/\log(\text{degree}+5)$ , greedy peeling.

Appendix

Table 16: Main settings.

Model Extraction Attacks

Model extraction attacks infer a machine-learning model’s behavior, architecture, parameters, or hidden assets from queries and system leakage.

See all 48 papers

An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic
Jun 4, 2026Shuze Liu, Qianwen Guo, Yushun DongModel Extraction Attacks LLM Security
Santa Clara University · Florida State University
Stateful Online Monitoring Catches Distributed Agent Attacks
May 29, 2026Davis Brown, Samarth Bhargav, Arav Santhanam +7LLM Agent Security AI Agent Monitoring
University of Pennsylvania · 2Carnegie Mellon University
Prompt-Injection Probes Learn the Conversation, Not Just the Injection
Oct 13, 2026Kaan Kale, Sriram VishwanathPrompt Injection Detection Indirect Prompt Injection
Georgia Institute of Technology

Jun 4, 2026J/K move · Enter open · S save

An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic

Shuze Liu, Qianwen Guo, Yushun Dong

Santa Clara University · Florida State University

Model Extraction Attacks LLM Security

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed through hosted APIs, making model extraction a practical threat to model ownership and service security. However, individual extraction queries often resemble benign requests, and existing evaluations often focus on single-query anomaly scoring or pure benign-versus-attacker user settings. We formulate model extraction monitoring as benign-calibrated traffic-window distribution testing and show that an embarrassingly simple detector is effective: embed incoming queries into a semantic space and test whether their aggregate distribution deviates from historical benign traffic. We instantiate the detector with maximum mean discrepancy (MMD), using only benign-vs-benign comparisons to set the decision threshold. We evaluate on fourteen attacker-normal query pairs from four extraction scenarios and compare with adapted PRADA, SEAT, CAP, DATE, and marginal Mahalanobis baselines. Across three random seeds, MMD achieves 0.3% benign FPR, 100.0% pure-attacker TPR, 90.5% average TPR over attacker fractions, and 95.1% balanced accuracy. These results show that benign-calibrated distribution testing is a strong empirical baseline for model extraction detection in both user-level and mixed multi-user LLM API traffic. Code is released at: https://github.com/LabRAI/mmd-llm-mea-detection.

Abstract

Model Extraction Attacks

Explore similar work

An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic

Stateful Online Monitoring Catches Distributed Agent Attacks

Prompt-Injection Probes Learn the Conversation, Not Just the Injection