Black-box LLM fingerprinting aims to determine whether a target model was derived from a source model when only the output responses to input queries are observable. One way to reveal this relationship is to perturb each query and observe the resulting responses of both models, yielding two corresponding sets of response variations. These sets provide two complementary sources of information: the relations among response changes within each set, and the correspondence of response changes between the two sets. Existing approaches, however, have not fully exploited both. We introduce SetPrint, a training-free method that jointly captures both within-set relations and between-set correspondence. SetPrint expresses each response change relative to the mean change of its own set, and then compares changes from the two models that were induced by the same perturbation. On LeaFBench, SetPrint consistently outperforms existing training-free black-box fingerprinting methods. Further analyses attribute these gains to both within-set relations and between-set correspondence.
Figures & tables
Figure 1: Concept of SetPrint. The same query perturbations yield two corresponding sets of response changes. SetPrint uses both (1) within-set relations and (2) between-set correspondence, which pairs the changes induced by the same perturbation in the two models.
Method
Access
Supervised
AUROC ↑
TPR@1%FPR ↑
pAUC ↑
REEF †
White box
No
0.896±0.021
0.634±0.021
0.832±0.019
LLMmap †
Black box
Yes
0.632±0.009
0.253±0.032
0.635±0.009
LLMDNA + SVM ‡
Black box
Yes
0.806±0.011
0.292±0.066
0.678±0.023
MET †
Black box
No
0.661±0.003
0.329±0.004
0.658±0.003
SEF †
Black box
No
0.581±0.008
0.282±0.012
0.646±0.004
ZeroPrint †
Black box
No
0.720±0.013
0.366±0.009
0.683±0.005
Table 1: Fingerprinting results on LeaFBench (mean ± std). Methods are grouped by required model access and supervision. † : reported in [ 17 ] ; ‡ : our runs. Bold: best among black-box, unsupervised methods.
Measure
AUROC ↑
TPR@1%FPR ↑
MMD (Linear)
0.711±0.008
0.323±0.010
MMD (RBF)
0.713±0.007
0.330±0.011
Wasserstein
0.719±0.011
0.331±0.011
CKA (Linear)
0.620±0.027
0.089±0.015
CKA (RBF)
0.610±0.025
0.104±0.019
RSA (Pearson)
0.623±0.025
0.095±0.014
Table 2: Alternative set comparison measures on the same response change sets. Bold: proposed method. Last block: kernel variants of SetPrint.
Permutation
AUROC ↑
TPR@1%FPR ↑
None (matched)
0.776±0.008
0.403±0.002
Within-base
0.746±0.006
0.335±0.013
Full-set
0.494±0.004
0.050±0.004
Table 3: Contribution of between-set correspondence in SetPrint. The target changes are left in their original order (None), shuffled among perturbations of each base query (Within-base), or shuffled across the full set (Full-set).
Figure 2: Contribution of within-set relations in SetPrint. In the products (riS)⊤rjT with i=j , target changes are progressively replaced by changes at the same indices from other models sharing the target’s base model. The products with i=j remain fixed. The line shows the mean AUROC across the five runs. The shaded band shows sample standard deviation across these runs.
As Large Language Model (LLM) APIs become ubiquitous, users increasingly rely on black-box fingerprinting to verify that providers are serving the advertised premium models. However, these methods may overlook adversarial providers who manipulate model weights to cheat the fingerprint process. We introduce a novel threat termed fingerprint spoofing, where a malicious provider stealthily serves a weaker model that has been parameter-efficiently fine-tuned to mimic a stronger model, thereby evading user-side fingerprinting. We first formally prove that user-side resource constraints (i.e., finite query budgets and weak fingerprinting classifiers) make current fingerprinting vulnerable to fingerprint spoofing. Guided by this theoretical analysis, we propose GhostPrint, a cost-effective attack framework leveraging surrogate modeling, reward-ranked fine-tuning, and knowledge distillation. Extensive evaluations in both static and continual fingerprinting settings demonstrate that GhostPrint allows weak models to consistently bypass representative fingerprint methods while maintaining utility at a low fine-tuning cost, exposing a critical vulnerability in current LLM fingerprinting pipelines.
Large language models (LLMs) are high-value assets that can be derived through redeployment, fine-tuning, quantization, or further alignment. Because deployed LLMs are commonly exposed only through query APIs, ownership verification must often rely on black-box text responses. This setting is difficult: generations are open-ended and can vary across repeated queries, while existing black-box fingerprints rely on signals that are fragile under a final-response interface, including full-text matching, soft behavioral features, or model-specific prompts designed not to transfer. We propose TCF (Targeted Counterfactual Fingerprinting), a black-box LLM fingerprinting framework that converts open-ended generation comparison into constrained-answer targeted counterfactual transfer. TCF restricts each verification query to a finite answer space, reducing the surface-form ambiguity that enters the verification score, and optimizes a prompt perturbation toward a counterfactual target different from the protected model's clean answer on the original prompt. Verification reduces to checking whether the suspect model's parsed final answer matches the recorded target. We introduce the source-model counterfactual margin (SCM), a protected-model-only quantity that certifies the target is unlikely before the perturbation and likely after it; SCM controls target selection, perturbation stopping, and fingerprint filtering. Under explicit derived-preservation and independent-transfer budgets motivated by local behavioral closeness, we derive a target-accuracy gap between derived and independent models. Across four LLM families, TCF achieves an average AUC of 0.9861, improving over TRAP, ProFLingo, and ZeroPrint by 0.07 to 0.19.
Yutong Wu, Xiaofan Bai, Shixin Li +10
Huazhong University of Science and Technology · Alibaba Group · Microsoft Corporation
Large language models (LLMs) are increasingly released under restricted licenses, creating a growing need for robust model ownership verification. Existing fingerprinting methods are often fragile under downstream finetuning, require invasive training modifications, or fail in black-box settings. We introduce RAFP, a robust framework for identifying LLM lineages via rare-region fingerprints. Our key insight is that downstream finetuning primarily updates common high-density language behaviors, while low-probability prompt regions receive weak optimization signal and limited gradient alignment under finetuned distribution. As a result, rare prompt-response behaviors remain stable across common model adaptations. RAFP is non-invasive, constructing fingerprints via discrete gradient-based optimization over rare prompts without modifying model weights. We provide a theoretical analysis showing that the likelihood change of rare-region fingerprints under finetuning remains bounded. Experiments across four LLM families and multiple downstream adaptations, including supervised finetuning, LoRA, quantization, prompt-template variation, and decoding changes, show that RAFP achieves strong fingerprint persistence and substantially outperforms prior fingerprinting baselines in black-box settings.
Yun-Yun Tsai, Jia Hao Liang, Chuan Guo +2
Columbia University / Meta · New York / Menlo Park, USA · MIT +5