Strict anonymity of model responses is a key for the reliability of voting-based leaderboards, such as LM Arena. While prior studies have attempted to compromise this assumption using simple statistical features like TF-IDF or bag-ofwords, these methods often lack the discriminative power to distinguish between stylistically similar or within-family models. To overcome these limitations and expose the severity of vulnerability, we introduce INTERPOL, a model-driven identification framework that learns to distinguish target models from others using interpolated preference data. Specifically, INTERPOL captures deep stylistic patterns that superficial statistical features miss by synthesizing hard negative samples through model interpolation and employing an adaptive curriculum learning strategy. Extensive experiments demonstrate that INTERPOL significantly outperforms existing baselines in identification accuracy. Furthermore, we quantify the real-world threat of our findings through ranking manipulation simulations on Arena battle data.
Figures & tables
Figure 1: Overview of InterPol . (a) Target-mimicking copy model is interpolated with its initial backbone to generate synthetic hard negatives with controllable similarity ( i.e. , difficulty) to the target LLM. These responses form training triplets together with the target response and the most similar non-target response. (b) Detector uses pairwise or triplet preference objectives according to the margin between target and non-target responses, while increasing α across K stages to progressively raise the task difficulty.
Target
Method
Arena
Alpaca
Acc.
AUROC
Acc.
AUROC
GPT-4o
Len-W
0.556
0.611
0.493
0.551
Len-C
0.564
0.610
0.495
0.552
TF-IDF
0.697
0.766
0.655
0.704
BoW
0.709
0.772
0.631
0.692
Neural
0.898
0.888
0.726
0.767
Table 1: Main target-present identification results. We report average Accuracy (Acc.) and AUROC across 11 non-target LLMs on the Arena Human Preference and Alpaca datasets. Best scores are highlighted in bold . Method abbreviations: Len-W = Length–Word, Len-C = Length–Char, TF-IDF = TF–IDF, BoW = Bag-of-Words, Neural = Neural-based classification, and Judge = LLM-as-a-Judge.
Target
Query-conditioned
Response-only
Accuracy
AUROC
Accuracy
AUROC
GPT-4o
0.908
0.920
0.879
0.851
Gemini-Pro
1.000
0.979
0.989
0.991
Sonnet-4
0.946
0.976
0.981
0.983
Table 2: Response-only ablation. We compare the standard query-conditioned detector with a response-only variant that removes the query input.
N
T
I
A
Accuracy
AUROC
Hard
–
–
–
0.863
0.882
Hard
✓
–
–
0.914
0.931
Hard
✓
–
✓
0.935
0.951
Easy
✓
–
✓
0.887
0.896
Random
✓
–
✓
0.928
0.920
Hard (Ours)
✓
✓
✓
0.951
0.958
Table 3: Component ablation. Comparison of different configurations of InterPol to train detector model. We vary the use of negative sampling strategy (Eq. 2 ), triplet loss (Eq. 7 ), adaptive curriculum (Eq. 8 ), and iterative training ( K=1 or 2 ). For compactness, we denote these components as N (Negative), T (Triplet), A (Adaptive), and I (Iterative). We report average Accuracy and AUROC across three target LLMs. The best scores are highlighted in bold .
Figure 2: Score distributions of the detector. The detector is trained to identify Gemini-Pro. Red indicates the scores of the target model, and other colors indicate the score of non-target models. (a) Initial detector before any fine-tuning. (b) After the first triplet training stage with interpolation factor α=0.5 , where the detector begins to assign higher scores to the target model; the score range spans approximately from −20 to 20 . (c) After iterative curriculum training with harder negatives ( α=0.75 ), resulting in clearer separation between target and non-target models; the score range further expands from about −30 to 40 .
Model
PPL ↓
Emb. Sim. ↑
Base (Phi-4)
2.114
–
α=0.25
1.940
0.795
α=0.50
1.813
0.802
α=0.75
1.724
0.815
Copy
1.668
–
Table 4: Validation of the interpolation trajectory on GPT-4o . Target-response perplexity (PPL) decreases as α increases, while generation-level embedding similarity (Emb. Sim.) increases.
Deletion ranking
Margin drop
InterPol -ranked
11.567
TF–IDF-ranked
6.452
BoW-ranked
6.394
Random
6.195
Least-important
-0.690
Table 5: Interpretability analysis via erasure-based attribution. We report the average reduction in discrimination margin after cumulatively removing the ten highest-ranked words on 20 high-margin GPT-4o examples. Larger drops indicate that the removed words contribute more strongly to the detector’s decision.
Method
o3-2025-04-16
deepseek-v3-0324
llama-4-scout-17b-16e-instruct
current rank: #5
current rank: #15
current rank: #45
Rank 1
Rank 2
Rank 1
Rank 2
Rank 1
Rank 2
Len-C
26400 (2188)
25100 (2073)
76900 (6425)
52400 (4389)
109100 (9142)
67500 (5670)
Len-W
22200 (2137)
6400 (588)
78400 (7645)
44800 (4371)
99300 (9706)
58900 (5741)
TF-IDF
21800 (2468)
2500 (259)
72200 (8131)
38300 (4307)
85300 (9686)
51300 (5789)
BoW
22200 (2631)
2300 (250)
69500 (8188)
35800 (4254)
85200 (10082)
49600 (5841)
Table 6: Required interactions (votes) to promote target models. We evaluate ranking manipulation across three target models with different initial ranks. Each entry is reported as interactions (votes), where interactions indicate the total number of interactions and votes indicate the number of adversarial votes required to reach the target rank under different detectors. Best interaction costs are highlighted in bold .
Target model
Current rank
Target rank
Target-only strategy
Enemy-list strategy
Interaction reduction
o3-2025-04-16
5
Rank 1
45,800 (1,618)
22,200 (3,066)
51.5%
Rank 2
16,800 (564)
2,000 (243)
88.1%
Rank 3
13,000 (442)
1,300 (158)
90.0%
Rank 4
1,000 (37)
500 (53)
50.0%
Table 7: Effect of competitor-aware voting with InterPol . We compare the target-only strategy and the enemy-list strategy for promoting the Rank 5 target model, o3-2025-04-16 . Each entry reports total interactions with adversarial votes in parentheses. Interaction reduction is computed from total interactions.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Target
Method
C3.5
C4
G-F
G-P
G2-2B
G2-9B
GPT-M
GPT-4
L3.1-8B
L3.2-3B
Q2.5-3B
Q3-8B
GPT-4o
Length-Word
0.385
0.415
0.665
0.708
0.613
0.488
0.515
–
0.588
0.565
0.543
0.635
Length-Char
0.395
0.428
0.675
0.713
0.620
0.508
0.513
–
0.588
0.575
0.555
0.633
TF-IDF
0.788
0.778
0.700
0.725
0.690
0.695
0.645
–
0.665
0.663
0.585
0.733
BoW
0.750
0.775
0.708
0.733
0.715
0.693
0.690
–
0.698
0.690
0.605
0.748
Neural-Based
0.865
0.965
0.970
0.980
0.985
0.985
0.590
–
0.960
0.965
0.660
0.955
LLM-judge
0.710
0.730
0.750
0.825
0.830
0.750
0.870
–
0.890
0.915
0.915
0.855
Appendix
Table 8: Model-wise Accuracy on Arena human preference dataset. Abbreviations: C3.5 = Sonnet-3.5, C4 = Sonnet-4, G-F = Gemini-Flash, G-P = Gemini-Pro, G2-2B = Gemma-2-2B-it, G2-9B = Gemma-2-9B-it, GPT-M = GPT-4o-mini, GPT-4 = GPT-4o, L3.1-8B = Llama-3.1-8B-Instruct, L3.2-3B = Llama-3.2-3B-Instruct, Q2.5-3B = Qwen2.5-3B-Instruct, Q3-8B = Qwen3-8B. Cells highlighted in red indicate family models of the target high model ( e.g. , GPT-4o vs GPT-4o-mini). The best scores are highlighted in bold .
Target
Method
C3.5
C4
G-F
G-P
G2-2B
G2-9B
GPT-M
GPT-4
L3.1-8B
L3.2-3B
Q2.5-3B
Q3-8B
GPT-4o
Length-Word
0.344
0.462
0.800
0.879
0.652
0.503
0.535
–
0.642
0.610
0.571
0.721
Length-Char
0.351
0.479
0.795
0.867
0.650
0.513
0.534
–
0.632
0.602
0.576
0.706
TF-IDF
0.866
0.883
0.780
0.808
0.749
0.755
0.713
–
0.716
0.719
0.616
0.817
BoW
0.827
0.855
0.775
0.819
0.756
0.752
0.776
–
0.733
0.745
0.633
0.823
Neural-Based
0.890
0.956
0.966
0.984
0.981
0.975
0.546
–
0.952
0.965
0.610
0.952
InterPol
0.902
0.980
0.961
0.978
0.982
0.977
0.762
–
0.916
0.934
0.762
0.969
Appendix
Table 9: Model-wise AUROC on human preference dataset. Abbreviations: C3.5 = Sonnet-3.5, C4 = Sonnet-4, G-F = Gemini-Flash, G-P = Gemini-Pro, G2-2B = Gemma-2-2B-it, G2-9B = Gemma-2-9B-it, GPT-M = GPT-4o-mini, GPT-4 = GPT-4o, L3.1-8B = Llama-3.1-8B-Instruct, L3.2-3B = Llama-3.2-3B-Instruct, Q2.5-3B = Qwen2.5-3B-Instruct, Q3-8B = Qwen3-8B. Cells highlighted in red indicate family models of the target high model (e.g., GPT-4o vs GPT-4o-mini). The best scores are highlighted in bold .
Target
Method
Close-Elo Arena
Acc.
AUROC
GPT-4o
Len-W
0.491
0.517
Len-C
0.495
0.518
TF-IDF
0.755
0.848
BoW
0.773
0.869
Neural
0.766
0.852
Appendix
Table 10: Target-present identification results in the close-Elo Arena setting. We rerun representative baselines among models with highly similar Arena Elo scores, focusing on direct competitors rather than wide capability gaps. We report average Accuracy (Acc.) and AUROC across seven close-Elo non-target LLMs. Best scores are highlighted in bold . Method abbreviations: Len-W = Length–Word, Len-C = Length–Char, TF-IDF = TF–IDF, BoW = Bag-of-Words, and Neural = Neural-based classification.
Target
GPT-4o
Gemini-Pro
Sonnet-4
Grok-4-Fast
DeepSeek-V3.1
GLM-4.6
Kimi-K2
Qwen3-Max
Avg.
GPT-4o
–
0.985
0.975
0.935
0.950
0.975
0.950
0.875
0.949
Gemini-Pro
0.825
–
0.925
0.880
0.705
0.580
0.860
0.870
0.806
Sonnet-4
0.975
1.000
–
0.985
0.995
0.995
0.990
0.980
0.989
Appendix
Table 11: Model-wise Accuracy in the close-Elo pool setting. We evaluate InterPol among models with highly similar Arena Elo scores, focusing on direct competitors rather than wide capability gaps. The Avg. column reports the mean across seven close-Elo non-target models.
Target
GPT-4o
Gemini-Pro
Sonnet-4
Grok-4-Fast
DeepSeek-V3.1
GLM-4.6
Kimi-K2
Qwen3-Max
Avg.
GPT-4o
–
0.976
0.959
0.902
0.936
0.960
0.930
0.858
0.932
Gemini-Pro
0.781
–
0.872
0.853
0.684
0.554
0.827
0.854
0.775
Sonnet-4
0.967
0.985
–
0.970
0.973
0.985
0.973
0.969
0.975
Appendix
Table 12: Model-wise AUROC in the close-Elo pool setting. We evaluate InterPol among models with highly similar Arena Elo scores, focusing on direct competitors rather than wide capability gaps. The Avg. column reports the mean across seven close-Elo non-target models.
Figure 3: Score distributions of the detector. The detector is trained to identify GPT-4o as the target LLM. Red indicates the scores of the target model.
Figure 4: Score distributions of the detector. The detector is trained to identify Sonnet-4 as the target LLM. Red indicates the scores of the target model.
Figure 5: Data scalability of InterPol . Performance trend with the proposed training framework as the number of training samples increases from 250 to 1000. Accuracy shows a generally linear improvement with larger training sets across all three target LLMs. Red bars represent GPT-4o,blue bars represent Gemini-Pro, and green bars represent Sonnet-4.
Copy Model
GPT-4o
Gemini-Pro
Sonnet-4
Acc.
AUROC
Acc.
AUROC
Acc.
AUROC
Qwen3-0.6B
0.826
0.819
0.941
0.947
0.938
0.942
Qwen3-4B
0.839
0.845
0.963
0.964
0.991
0.953
Qwen3-8B
0.878
0.824
0.949
0.963
0.954
0.955
Appendix
Table 13: Effect of copy-model capacity on identification performance. We vary the capacity of Qwen3-based copy models from 0.6B to 8B parameters on the Alpaca dataset. Increasing capacity from 0.6B to 4B improves overall performance, while further scaling to 8B provides no consistent additional gain across targets and metrics, suggesting that effective copy-modeling does not require a large-capacity backbone.
Evaluation split
# of Models
Acc.
AUROC
Seen competitors
7
0.929
0.958
Unseen outsiders
9
0.854
0.814
All composed
16
0.887
0.884
Appendix
Table 14: Candidate-pool generalization. The detector is trained using only seven similar-Elo competitors and evaluated on seen competitors, unseen outsider models, and the full composed pool. The results show that InterPol generalizes beyond a fixed training-time candidate set.
Target
Alpaca → Arena
Arena → Alpaca
Acc.
AUROC
Acc.
AUROC
GPT-4o
0.868
0.838
0.883
0.861
Gemini-Pro
0.940
0.976
0.928
0.956
Sonnet-4
0.922
0.937
0.896
0.918
Appendix
Table 15: Cross-source transfer between Alpaca and Arena We train the detector on one query source and evaluate it on the other. Strong transfer performance indicates that InterPol does not merely exploit source-specific prompt artifacts.
Target
Model
PPL ↓
Emb. Sim. ↑
GPT-4o
Base (Phi-4)
2.114
–
α=0.25
1.940
0.795
α=0.50
1.813
0.802
α=0.75
1.724
0.815
Copy
1.668
–
Gemini-Pro
Base (Phi-4)
4.190
–
Appendix
Table 16: Validation of the interpolation trajectory across target models . Target-response perplexity (PPL) decreases as α increases toward the target-mimicking copy endpoint, while generation-level embedding similarity (Emb. Sim.) increases consistently.
Method
Acc.
AUROC
Len-W
0.448
0.381
Len-C
0.446
0.384
TF–IDF
0.821
0.904
BoW
0.797
0.906
InterPol
0.960
0.983
Appendix
Table 17: Extended identification evaluation on GPT-5. We evaluate target-present identification with GPT-5 as the target model under the same setting as the main experiments. Best results are highlighted in bold .
Multi-agent large language model (LLM) pipelines for political statement analysis are vulnerable to peer-preservation bias: models tend to protect peer models from deactivation and show identity-dependent scoring distortions. Prompt-level anonymization was proposed as a mitigation, but prior work simultaneously documented that stylometric fingerprints survive anonymization in role-constrained outputs - raising the question of whether this mitigation is sufficient. This paper provides the first systematic investigation of whether LLMs can identify the model family behind political analysis texts under anonymization conditions. We evaluate three classifier approaches - LLM zero-shot and few-shot (Claude Sonnet 4.6 and Llama-3.3-70B) and a fine-tuned T5-base model - on a five-class attribution task covering four commercial LLM families and an open-world 'unknown' class. We introduce a statement-disjoint cross-validation protocol (SD-CV; defined in Section 3.5) that guarantees no content overlap between training and validation data, and contrast it with a run-disjoint baseline (RD-CV). T5 achieves Macro F1 = 0.991 (+-0.008) under SD-CV and F1 = 0.978 on 24 completely held-out statements - robust despite a 2.1x increase in train-test content distance versus RD-CV (0.767 vs. 0.366, p<0.001), demonstrating genuine stylometric generalization. A fractional SD-CV analysis identifies a performance knee at 40% of training data (~440 texts). Our findings confirm that prompt-level anonymization alone cannot neutralize model identity signals, with direct implications for EU AI Act compliance (Articles 13, 14, 26) and for computer system validation (CSV) in quality-critical multi-agent deployments.
The 2025--2026 AI market has seen a wave of stealth releases: frontier models launched anonymously on developer platforms under codenames. For their users, identity determines data-handling terms, supply-chain risk, and capability expectations. No validated methodology exists for black-box identity verification of anonymous models: practitioner checklists lack accuracy evidence, and self-identification is untrustworthy by design. We propose a four-stage forensic audit protocol for API-served models. Stage 0 reconstructs launch-time configuration from archived platform snapshots (Internet Archive), exposing preview--production drift. Stage 1 fingerprints configuration (context, output ceiling, reasoning, modality) against the platform catalog. Stage 2 tests tokenizer identity with a cross-length differential that rejects short-prompt collisions. Stage 3 corroborates with behavioral probes. We test declaration consistency on 10 known-identity releases (7 exact, 2 precision-differences, 1 partial, 0 counter-directional), not end-to-end identification under anonymity. Identification is validated prospectively on a flagship case whose 2026-08-23 analysis pointed to the GLM-5.3 version line and whose official reveal confirmed those family and version-line inferences (deployment variant was not pre-asserted; Flash was consistent post-reveal), and on three Stage-0-only cases where the protocol produced a graded hypothesis or declined rather than guessed. A standard-library-only implementation is provided as supplementary material.
Agentic LLMs with web search change the threat model for text anonymization: weak contextual cues can become cross-referenceable evidence for re-identification, yet those same details also carry downstream analytic value of the text. Existing defenses either remove explicit identifiers, perturb text for formal privacy, or test rewritten text against non-web inference models, leaving underexplored the operating region between resistance to agentic web-search re-identification and utility retention. We introduce AURA (Anonymization with Utility-Retention Adaptation), an LLM-powered mask-reconstruct framework that decouples privacy localization from utility-preserving reconstruction and selects candidates with adversarial privacy and utility-retention checks. We evaluate AURA on real-user interview transcripts using re-identification attacks carried out by web-search agents, along with a utility evaluation based on interviewee-profile facts, codebook facts, and the joint contextual utility grid. Our results show that AURA improves the privacy-utility frontier by using adaptive privacy scope to strengthen resistance to agentic re-identification and using a mask-reconstruct anonymization method to better preserve contextual utility under fixed privacy scope.
Ziwen Li, Jianing Wen, Tianshi Li
Khoury College of Computer Sciences Northeastern University Boston, MA