InterPol: De-anonymizing LM Arena via Interpolated Preference Learning

Minsung Cho; Jaehyung Kim

cs.AIMar 16, 2026

InterPol: De-anonymizing LM Arena via Interpolated Preference Learning

Authors: Minsung Cho, Jaehyung Kim

Organizations: Yonsei University

Abstract

Strict anonymity of model responses is a key for the reliability of voting-based leaderboards, such as LM Arena. While prior studies have attempted to compromise this assumption using simple statistical features like TF-IDF or bag-ofwords, these methods often lack the discriminative power to distinguish between stylistically similar or within-family models. To overcome these limitations and expose the severity of vulnerability, we introduce INTERPOL, a model-driven identification framework that learns to distinguish target models from others using interpolated preference data. Specifically, INTERPOL captures deep stylistic patterns that superficial statistical features miss by synthesizing hard negative samples through model interpolation and employing an adaptive curriculum learning strategy. Extensive experiments demonstrate that INTERPOL significantly outperforms existing baselines in identification accuracy. Furthermore, we quantify the real-world threat of our findings through ranking manipulation simulations on Arena battle data.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of InterPol . (a) Target-mimicking copy model is interpolated with its initial backbone to generate synthetic hard negatives with controllable similarity ( i.e. , difficulty) to the target LLM. These responses form training triplets together with the target response and the most similar non-target response. (b) Detector uses pairwise or triplet preference objectives according to the margin between target and non-target responses, while increasing $\alpha$ across $K$ stages to progressively raise the task difficulty.

Target	Method	Arena		Alpaca
Target	Method	Acc.	AUROC	Acc.	AUROC
GPT-4o	Len-W	0.556	0.611	0.493	0.551
	Len-C	0.564	0.610	0.495	0.552
	TF-IDF	0.697	0.766	0.655	0.704
	BoW	0.709	0.772	0.631	0.692
	Neural	0.898	0.888	0.726	0.767

Table 1: Main target-present identification results. We report average Accuracy (Acc.) and AUROC across 11 non-target LLMs on the Arena Human Preference and Alpaca datasets. Best scores are highlighted in bold . Method abbreviations: Len-W = Length–Word, Len-C = Length–Char, TF-IDF = TF–IDF, BoW = Bag-of-Words, Neural = Neural-based classification, and Judge = LLM-as-a-Judge.

Target	Query-conditioned		Response-only
Target	Accuracy	AUROC	Accuracy	AUROC
GPT-4o	0.908	0.920	0.879	0.851
Gemini-Pro	1.000	0.979	0.989	0.991
Sonnet-4	0.946	0.976	0.981	0.983

Table 2: Response-only ablation. We compare the standard query-conditioned detector with a response-only variant that removes the query input.

N	T	I	A	Accuracy	AUROC
Hard	–	–	–	0.863	0.882
Hard	✓	–	–	0.914	0.931
Hard	✓	–	✓	0.935	0.951
Easy	✓	–	✓	0.887	0.896
Random	✓	–	✓	0.928	0.920
Hard (Ours)	✓	✓	✓	0.951	0.958

Table 3: Component ablation. Comparison of different configurations of InterPol to train detector model. We vary the use of negative sampling strategy (Eq. 2 ), triplet loss (Eq. 7 ), adaptive curriculum (Eq. 8 ), and iterative training (

K=1

2

). For compactness, we denote these components as N (Negative), T (Triplet), A (Adaptive), and I (Iterative). We report average Accuracy and AUROC across three target LLMs. The best scores are highlighted in bold .

Panel of Figure 2 — Figure 2: Score distributions of the detector. The detector is trained to identify Gemini-Pro. Red indicates the scores of the target model, and other colors indicate the score of non-target models. (a) Initial detector before any fine-tuning. (b) After the first triplet training stage with interpolation factor $\alpha=0.5$ , where the detector begins to assign higher scores to the target model; the score range spans approximately from $-20$ to $20$ . (c) After iterative curriculum training with harder negatives ( $\alpha=0.75$ ), resulting in clearer separation between target and non-target models; the score range further expands from about $-30$ to $40$ .

Model	PPL $\downarrow$	Emb. Sim. $\uparrow$
Base (Phi-4)	2.114	–
$\alpha=0.25$	1.940	0.795
$\alpha=0.50$	1.813	0.802
$\alpha=0.75$	1.724	0.815
Copy	1.668	–

Table 4: Validation of the interpolation trajectory on GPT-4o . Target-response perplexity (PPL) decreases as

\alpha

increases, while generation-level embedding similarity (Emb. Sim.) increases.

Deletion ranking	Margin drop
InterPol -ranked	11.567
TF–IDF-ranked	6.452
BoW-ranked	6.394
Random	6.195
Least-important	-0.690

Table 5: Interpretability analysis via erasure-based attribution. We report the average reduction in discrimination margin after cumulatively removing the ten highest-ranked words on 20 high-margin GPT-4o examples. Larger drops indicate that the removed words contribute more strongly to the detector’s decision.

Method	o3-2025-04-16		deepseek-v3-0324		llama-4-scout-17b-16e-instruct
	current rank: #5		current rank: #15		current rank: #45
	Rank 1	Rank 2	Rank 1	Rank 2	Rank 1	Rank 2
Len-C	26400 (2188)	25100 (2073)	76900 (6425)	52400 (4389)	109100 (9142)	67500 (5670)
Len-W	22200 (2137)	6400 (588)	78400 (7645)	44800 (4371)	99300 (9706)	58900 (5741)
TF-IDF	21800 (2468)	2500 (259)	72200 (8131)	38300 (4307)	85300 (9686)	51300 (5789)
BoW	22200 (2631)	2300 (250)	69500 (8188)	35800 (4254)	85200 (10082)	49600 (5841)

Table 6: Required interactions (votes) to promote target models. We evaluate ranking manipulation across three target models with different initial ranks. Each entry is reported as interactions (votes), where interactions indicate the total number of interactions and votes indicate the number of adversarial votes required to reach the target rank under different detectors. Best interaction costs are highlighted in bold .

Target model	Current rank	Target rank	Target-only strategy	Enemy-list strategy	Interaction reduction
o3-2025-04-16	5	Rank 1	45,800 (1,618)	22,200 (3,066)	51.5%
		Rank 2	16,800 (564)	2,000 (243)	88.1%
		Rank 3	13,000 (442)	1,300 (158)	90.0%
		Rank 4	1,000 (37)	500 (53)	50.0%

Table 7: Effect of competitor-aware voting with InterPol . We compare the target-only strategy and the enemy-list strategy for promoting the Rank 5 target model, o3-2025-04-16 . Each entry reports total interactions with adversarial votes in parentheses. Interaction reduction is computed from total interactions.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Target	Method	C3.5	C4	G-F	G-P	G2-2B	G2-9B	GPT-M	GPT-4	L3.1-8B	L3.2-3B	Q2.5-3B	Q3-8B
GPT-4o	Length-Word	0.385	0.415	0.665	0.708	0.613	0.488	0.515	–	0.588	0.565	0.543	0.635
	Length-Char	0.395	0.428	0.675	0.713	0.620	0.508	0.513	–	0.588	0.575	0.555	0.633
	TF-IDF	0.788	0.778	0.700	0.725	0.690	0.695	0.645	–	0.665	0.663	0.585	0.733
	BoW	0.750	0.775	0.708	0.733	0.715	0.693	0.690	–	0.698	0.690	0.605	0.748
	Neural-Based	0.865	0.965	0.970	0.980	0.985	0.985	0.590	–	0.960	0.965	0.660	0.955
	LLM-judge	0.710	0.730	0.750	0.825	0.830	0.750	0.870	–	0.890	0.915	0.915	0.855

Appendix

Table 8: Model-wise Accuracy on Arena human preference dataset. Abbreviations: C3.5 = Sonnet-3.5, C4 = Sonnet-4, G-F = Gemini-Flash, G-P = Gemini-Pro, G2-2B = Gemma-2-2B-it, G2-9B = Gemma-2-9B-it, GPT-M = GPT-4o-mini, GPT-4 = GPT-4o, L3.1-8B = Llama-3.1-8B-Instruct, L3.2-3B = Llama-3.2-3B-Instruct, Q2.5-3B = Qwen2.5-3B-Instruct, Q3-8B = Qwen3-8B. Cells highlighted in red indicate family models of the target high model ( e.g. , GPT-4o vs GPT-4o-mini). The best scores are highlighted in bold .

Target	Method	C3.5	C4	G-F	G-P	G2-2B	G2-9B	GPT-M	GPT-4	L3.1-8B	L3.2-3B	Q2.5-3B	Q3-8B
GPT-4o	Length-Word	0.344	0.462	0.800	0.879	0.652	0.503	0.535	–	0.642	0.610	0.571	0.721
	Length-Char	0.351	0.479	0.795	0.867	0.650	0.513	0.534	–	0.632	0.602	0.576	0.706
	TF-IDF	0.866	0.883	0.780	0.808	0.749	0.755	0.713	–	0.716	0.719	0.616	0.817
	BoW	0.827	0.855	0.775	0.819	0.756	0.752	0.776	–	0.733	0.745	0.633	0.823
	Neural-Based	0.890	0.956	0.966	0.984	0.981	0.975	0.546	–	0.952	0.965	0.610	0.952
	InterPol	0.902	0.980	0.961	0.978	0.982	0.977	0.762	–	0.916	0.934	0.762	0.969

Appendix

Table 9: Model-wise AUROC on human preference dataset. Abbreviations: C3.5 = Sonnet-3.5, C4 = Sonnet-4, G-F = Gemini-Flash, G-P = Gemini-Pro, G2-2B = Gemma-2-2B-it, G2-9B = Gemma-2-9B-it, GPT-M = GPT-4o-mini, GPT-4 = GPT-4o, L3.1-8B = Llama-3.1-8B-Instruct, L3.2-3B = Llama-3.2-3B-Instruct, Q2.5-3B = Qwen2.5-3B-Instruct, Q3-8B = Qwen3-8B. Cells highlighted in red indicate family models of the target high model (e.g., GPT-4o vs GPT-4o-mini). The best scores are highlighted in bold .

Target	Method	Close-Elo Arena
Target	Method	Acc.	AUROC
GPT-4o	Len-W	0.491	0.517
	Len-C	0.495	0.518
	TF-IDF	0.755	0.848
	BoW	0.773	0.869
	Neural	0.766	0.852

Appendix

Table 10: Target-present identification results in the close-Elo Arena setting. We rerun representative baselines among models with highly similar Arena Elo scores, focusing on direct competitors rather than wide capability gaps. We report average Accuracy (Acc.) and AUROC across seven close-Elo non-target LLMs. Best scores are highlighted in bold . Method abbreviations: Len-W = Length–Word, Len-C = Length–Char, TF-IDF = TF–IDF, BoW = Bag-of-Words, and Neural = Neural-based classification.

Target	GPT-4o	Gemini-Pro	Sonnet-4	Grok-4-Fast	DeepSeek-V3.1	GLM-4.6	Kimi-K2	Qwen3-Max	Avg.
GPT-4o	–	0.985	0.975	0.935	0.950	0.975	0.950	0.875	0.949
Gemini-Pro	0.825	–	0.925	0.880	0.705	0.580	0.860	0.870	0.806
Sonnet-4	0.975	1.000	–	0.985	0.995	0.995	0.990	0.980	0.989

Appendix

Table 11: Model-wise Accuracy in the close-Elo pool setting. We evaluate InterPol among models with highly similar Arena Elo scores, focusing on direct competitors rather than wide capability gaps. The Avg. column reports the mean across seven close-Elo non-target models.

Target	GPT-4o	Gemini-Pro	Sonnet-4	Grok-4-Fast	DeepSeek-V3.1	GLM-4.6	Kimi-K2	Qwen3-Max	Avg.
GPT-4o	–	0.976	0.959	0.902	0.936	0.960	0.930	0.858	0.932
Gemini-Pro	0.781	–	0.872	0.853	0.684	0.554	0.827	0.854	0.775
Sonnet-4	0.967	0.985	–	0.970	0.973	0.985	0.973	0.969	0.975

Appendix

Table 12: Model-wise AUROC in the close-Elo pool setting. We evaluate InterPol among models with highly similar Arena Elo scores, focusing on direct competitors rather than wide capability gaps. The Avg. column reports the mean across seven close-Elo non-target models.

Panel of Figure 3 — Figure 3: Score distributions of the detector. The detector is trained to identify GPT-4o as the target LLM. Red indicates the scores of the target model.

Panel of Figure 4 — Figure 4: Score distributions of the detector. The detector is trained to identify Sonnet-4 as the target LLM. Red indicates the scores of the target model.

Figure 5: Data scalability of InterPol . Performance trend with the proposed training framework as the number of training samples increases from 250 to 1000. Accuracy shows a generally linear improvement with larger training sets across all three target LLMs. Red bars represent GPT-4o,blue bars represent Gemini-Pro, and green bars represent Sonnet-4.

Copy Model	GPT-4o		Gemini-Pro		Sonnet-4
Copy Model	Acc.	AUROC	Acc.	AUROC	Acc.	AUROC
Qwen3-0.6B	0.826	0.819	0.941	0.947	0.938	0.942
Qwen3-4B	0.839	0.845	0.963	0.964	0.991	0.953
Qwen3-8B	0.878	0.824	0.949	0.963	0.954	0.955

Appendix

Table 13: Effect of copy-model capacity on identification performance. We vary the capacity of Qwen3-based copy models from 0.6B to 8B parameters on the Alpaca dataset. Increasing capacity from 0.6B to 4B improves overall performance, while further scaling to 8B provides no consistent additional gain across targets and metrics, suggesting that effective copy-modeling does not require a large-capacity backbone.

Evaluation split	# of Models	Acc.	AUROC
Seen competitors	7	0.929	0.958
Unseen outsiders	9	0.854	0.814
All composed	16	0.887	0.884

Appendix

Table 14: Candidate-pool generalization. The detector is trained using only seven similar-Elo competitors and evaluated on seen competitors, unseen outsider models, and the full composed pool. The results show that InterPol generalizes beyond a fixed training-time candidate set.

Target	Alpaca $\rightarrow$ Arena		Arena $\rightarrow$ Alpaca
Target	Acc.	AUROC	Acc.	AUROC
GPT-4o	0.868	0.838	0.883	0.861
Gemini-Pro	0.940	0.976	0.928	0.956
Sonnet-4	0.922	0.937	0.896	0.918

Appendix

Table 15: Cross-source transfer between Alpaca and Arena We train the detector on one query source and evaluate it on the other. Strong transfer performance indicates that InterPol does not merely exploit source-specific prompt artifacts.

Target	Model	PPL $\downarrow$	Emb. Sim. $\uparrow$
GPT-4o	Base (Phi-4)	2.114	–
	$\alpha=0.25$	1.940	0.795
	$\alpha=0.50$	1.813	0.802
	$\alpha=0.75$	1.724	0.815
	Copy	1.668	–
Gemini-Pro	Base (Phi-4)	4.190	–

Appendix

Table 16: Validation of the interpolation trajectory across target models . Target-response perplexity (PPL) decreases as

\alpha

increases toward the target-mimicking copy endpoint, while generation-level embedding similarity (Emb. Sim.) increases consistently.

Method	Acc.	AUROC
Len-W	0.448	0.381
Len-C	0.446	0.384
TF–IDF	0.821	0.904
BoW	0.797	0.906
InterPol	0.960	0.983

Appendix

Table 17: Extended identification evaluation on GPT-5. We evaluate target-present identification with GPT-5 as the target model under the same setting as the main experiments. Best results are highlighted in bold .

Language Model Fingerprinting

Language model fingerprinting identifies and verifies models or their lineage from distinctive outputs, internal representations, or embedded signals.

See all 81 papers

Can Multi-Agent LLMs Identify Their Peers? Stylometric Fingerprinting in Role-Constrained Political Analysis
May 20, 2026Juergen DietrichMulti-Agent LLM Systems Language Model Fingerprinting
Democracy Intelligence gGmbH, Berlin, Germany
Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification
Aug 31, 2026Yisen XiLanguage Model Fingerprinting Black-Box Model Auditing
Independent Researcher, Beijing, China
LLM Anonymization Against Agentic Re-Identification
May 29, 2026Ziwen Li, Jianing Wen, Tianshi LiPrivacy-Preserving Language Models Privacy-Utility Tradeoff
Khoury College of Computer Sciences Northeastern University Boston, MA

May 20, 2026J/K move · Enter open · S save

Can Multi-Agent LLMs Identify Their Peers? Stylometric Fingerprinting in Role-Constrained Political Analysis

Juergen Dietrich

Democracy Intelligence gGmbH, Berlin, Germany

Multi-Agent LLM Systems Language Model Fingerprinting Stylometry

Open paper AskPDF arXiv

Multi-agent large language model (LLM) pipelines for political statement analysis are vulnerable to peer-preservation bias: models tend to protect peer models from deactivation and show identity-dependent scoring distortions. Prompt-level anonymization was proposed as a mitigation, but prior work simultaneously documented that stylometric fingerprints survive anonymization in role-constrained outputs - raising the question of whether this mitigation is sufficient. This paper provides the first systematic investigation of whether LLMs can identify the model family behind political analysis texts under anonymization conditions. We evaluate three classifier approaches - LLM zero-shot and few-shot (Claude Sonnet 4.6 and Llama-3.3-70B) and a fine-tuned T5-base model - on a five-class attribution task covering four commercial LLM families and an open-world 'unknown' class. We introduce a statement-disjoint cross-validation protocol (SD-CV; defined in Section 3.5) that guarantees no content overlap between training and validation data, and contrast it with a run-disjoint baseline (RD-CV). T5 achieves Macro F1 = 0.991 (+-0.008) under SD-CV and F1 = 0.978 on 24 completely held-out statements - robust despite a 2.1x increase in train-test content distance versus RD-CV (0.767 vs. 0.366, p<0.001), demonstrating genuine stylometric generalization. A fractional SD-CV analysis identifies a performance knee at 40% of training data (~440 texts). Our findings confirm that prompt-level anonymization alone cannot neutralize model identity signals, with direct implications for EU AI Act compliance (Articles 13, 14, 26) and for computer system validation (CSV) in quality-critical multi-agent deployments.

Abstract

Language Model Fingerprinting

Explore similar work

Can Multi-Agent LLMs Identify Their Peers? Stylometric Fingerprinting in Role-Constrained Political Analysis

Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification

LLM Anonymization Against Agentic Re-Identification