Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior

Chuyi Wang; Xiaohui Xie; Tongze Wang; Fangchen Luo; Yong Cui

cs.CRSep 23, 2026

Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior

Authors: Chuyi Wang, Xiaohui Xie, Tongze Wang, Fangchen Luo, Yong Cui

Organizations: Department of Computer Science and Technology, Tsinghua University

Abstract

LLMs increasingly operate through coding-agent harnesses that inspect repositories, invoke tools, and modify files. Substituting the model behind such an agent can therefore change security-relevant decisions, including whether it verifies changes or recovers safely from failures. Existing LLM fingerprints largely infer identity from direct text or token distributions. In coding agents, these signals are mediated by system instructions, controller logic, tools, and execution feedback, limiting their transfer. We present LIDAR (LLM Identification from Decisions and Actions at Runtime), an active black-box fingerprinting method for coding-agent execution. Three coding probe pairs expose post-edit verification, transient-failure recovery, and specification--test conflict resolution under controlled changes. LIDAR represents the resulting trajectories with complementary instance-level and distribution-level features and compares them with clean references using a lightweight probabilistic identifier. It requires no access to model weights, logits, or provider internals. Across 36 models from seven families and two agent harnesses, LIDAR achieves high Top-1 accuracy and MRR and outperforms four existing fingerprinting and API-auditing baselines. Ablations confirm that the two feature levels, all probe pairs, and their controlled variants contribute. These results show that agent execution behavior provides model-identity evidence beyond final outputs.

Figures & tables

Figure 1: Threat model. The verifier controls host-side probe execution and records its local consequences, while the remote service controls hidden routing and remote transformations. An endpoint advertised as $c$ may silently serve $M_{i,t}\notin\mathcal{M}(c)$ . The resulting query fingerprint is compared with frozen, harness-specific enrollment references; the current verdict is closed-set behavioral evidence, not remote attestation.

Figure 2: Lidar workflow. Probe execution produces a fingerprint sample; feature extraction represents its local and across-variant behavior; enrollment and scoring compare a later query with frozen, harness-specific references. Provider-controlled attacks may alter the query path, while the verifier still records local execution events and outcomes.

Feature group	Information retained	Fields
Instance-level structure	Per-variant action roles, transitions, positions, repetition, backtracking, and recovery paths	588
Distribution-level behavior	Across six trajectories, the mean and sample standard deviation of action, command-use, ordering, repetition, recovery, outcome, and coarse response-form descriptors	202

Table 1: Principal feature groups, retained information, and dimensions.

Method	Atomic sampling unit	Units	Runs	Units	Runs	Per setting	Full panel
		Training/enrollment		Test/query		Successful runs
LLMmap [ 26 ]	8 probes / configuration	20 (configs)	160	20 (configs)	160	320	23,040
ZeroPrint [ 30 ]	5 responses / base query	10 (bases)	50	10 (bases)	50	100	7,200
MET [ 10 ]	10 prompts / repetition	5 (reps)	50	5 (reps)	50	100	7,200
One Token [ 3 ]	8 coordinates / repetition	5 (reps)	40	5 (reps)	40	80	5,760
Lidar	3 probe pairs (6 variants) / sample	6 (samples)	36	6 (samples)	36	72	5,184

Table 3: Sampling budgets for evaluated methods. Per setting counts successful runs for one model–harness combination; Full panel covers all 36 models and two harnesses. Failed attempts are excluded.

	OpenCode [ 25 ]			mini-swe-agent [ 31 ]
Method	Top-1	Top-3	MRR	Top-1	Top-3	MRR
LLMmap [ 26 ]	$0.6143{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0170)}$	$0.8189{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0134)}$	$0.7319{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0121)}$	$0.5628{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0223)}$	$0.7654{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0215)}$	$0.6860{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0186)}$
MET [ 10 ]	$0.8667{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0489)}$	$0.9574{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0349)}$	$0.9160{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0340)}$	$0.5917{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0443)}$	$0.7315{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0382)}$	$0.6927{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0277)}$
ZeroPrint [ 30 ]	$0.2833{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0272)}$	$0.4333{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0377)}$	$0.3971{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0227)}$	$0.4000{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0377)}$	$0.5889{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0408)}$	$0.5197{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0205)}$
One Token [ 3 ]	$0.7333{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0572)}$	$0.9167{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0465)}$	$0.8294{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0356)}$	$0.2722{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0272)}$	$0.3056{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0000)}$	$0.2878{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0152)}$
Lidar	$\mathbf{0.8836}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0185)}$	$\mathbf{0.9838}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0072)}$	$\mathbf{0.9336}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0107)}$	$\mathbf{0.9513}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0124)}$	$\mathbf{0.9962}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0036)}$	$\mathbf{0.9736}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0069)}$

Table 4: Identification performance on the 36-model panel. Results are reported on the

[0,1]

scale as mean

\pm

SD. Gray terms denote SD; bold values indicate the best mean in each column.

Group	Configuration	Fields	Top-1	$\Delta$ Top-1 (pp)	Top-3	MRR
Reference	Full method	958	0.9352	–	0.9884	0.9631
Feature view	Instance only	756	0.7500	$-18.52$	0.9005	0.8377
Feature view	Distribution only	202	0.8981	$-3.70$	0.9884	0.9426
Distribution statistic	Instance + mean	857	0.9190	$-1.62$	0.9838	0.9529
Distribution statistic	Instance + standard deviation	857	0.9097	$-2.55$	0.9792	0.9454
Probe pair	Without Verify	724	0.8819	$-5.32$	0.9745	0.9315

Table 5: Feature and probe-pair ablations using the prespecified enrollment/query roles and locked query set, pooled across the two separately fitted harnesses.

\Delta

is relative to the Full method under this fixed protocol.

Figure 3: Identifier comparison on the locked query set. Bars denote alternative classifiers and dashed lines denote Jeffreys; both panels share a vertical scale truncated at 0.65.

		Clean		A1: Identity obfuscation		A2: Output-format control
Harness	Method	Acc.	MRR	Acc.	MRR	Acc.	MRR
mini-swe-agent [ 31 ]	LLMmap [ 26 ]	0.5667	0.7639	0.4667 ( $\downarrow$ )	0.7000 ( $\downarrow$ )	0.3333 ( $\downarrow$ )	0.5533 ( $\downarrow$ )
	ZeroPrint [ 30 ]	0.3333	0.5111	0.0000 ( $\downarrow$ )	0.3944 ( $\downarrow$ )	0.1667 ( $\downarrow$ )	0.4222 ( $\downarrow$ )
	MET [ 10 ]	0.5000	0.6444	0.1667 ( $\downarrow$ )	0.4500 ( $\downarrow$ )	0.1667 ( $\downarrow$ )	0.4083 ( $\downarrow$ )
	One Token [ 3 ]	0.3333	0.4167	0.1667 ( $\downarrow$ )	0.2500 ( $\downarrow$ )	0.0000 ( $\downarrow$ )	0.0000 ( $\downarrow$ )
	Lidar	1.0000	1.0000	0.8889 ( $\downarrow$ )	0.9444 ( $\downarrow$ )	1.0000	1.0000

Table 6: Closed-set identification under provider-side manipulation. Acc. denotes Top-1 accuracy. Green A1 values and blue A2 values with a downward marker are lower than the same method’s Clean result.

Figure 4: Illustrative trajectories under the same prompt, provider, harness, and seed. All three models produce an equivalent one-line fix; each box denotes one agent step.

Figure 5: Feature importance in the frozen identifier. (a) Mean absolute evidence by semantic group. (b) Top-20% contribution frequency and model coverage for the ten strongest cross-harness mechanisms. (c) Median cumulative evidence on correct queries with interquartile ranges. Contributions are measured against the strongest incorrect model.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Family	#	Models
GPT [ 24 ]	7	gpt-5.1, gpt-5.2, gpt-5.4, gpt-5.5, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra
Claude [ 2 ]	7	claude-fable-5, claude-opus-4-6, claude-opus-4-7, claude-opus-4-8, claude-opus-5, claude-sonnet-4-6, claude-sonnet-5
Qwen [ 35 ]	7	Qwen3.5-122B-A10B, Qwen3.5-27B, Qwen3.6-Plus, Qwen3.7-Max, Qwen3.7-Plus, Qwen3.8-2.4T-A95B, Qwen3.8-27B
GLM [ 12 ]	6	GLM-4.7, GLM-5, GLM-5-Turbo, GLM-5.1, GLM-5.2, GLM-5.3
DeepSeek [ 9 ]	4	DeepSeek-V3.1, DeepSeek-V3.2, DeepSeek-V4-Flash, DeepSeek-V4-Pro
Kimi [ 17 ]	3	Kimi-K2.5, Kimi-K2.6, Kimi-K3

Appendix

Table 7: LLMs included in the evaluation, grouped by family.

Explore similar work

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces
May 14, 2026William Lugoloobi, Samuelle Marro, Jabez Magomere +2Attacker Large Language Model Fingerprint
Oxford Internet Institute, University of Oxford · Department of Engineering Science, University of Oxford
Fingerprinting Inference Systems of Large Language Models
May 28, 2026Anna Wimbauer, Jonas Möller, Erik Imgrund +1LLM Inference Optimization Fingerprint
BIFOLD & TU Berlin
Black-Box Forensics for Conversational LLM Agents
Jun 21, 2026Isadora White, Yasaman Jafari, Taylor Berg-KirkpatrickLarge Language Model Agents Attacker Large Language Model
University of California, San Diego

May 14, 2026, cs.CRJ/K move · Enter open · S save

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

William Lugoloobi, Samuelle Marro, Jabez Magomere, Joss Wright+1

Oxford Internet Institute, University of Oxford · Department of Engineering Science, University of Oxford

Attacker Large Language Model Fingerprint Model Vulnerabilities Traces

Open paper AskPDF arXiv

As LLM-based agents increasingly browse the web on users' behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent's actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1. We formalise this attack surface by demonstrating that classifiers trained on agent actions generalise across model sizes and families. We further show that strong classifiers can be trained from few interaction traces and that agent identity can be inferred early within an episode. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our harness and a labelled corpus of agent traces \href{https://github.com/KabakaWilliam/known_actions}{here}.

Abstract

Explore similar work

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

Fingerprinting Inference Systems of Large Language Models

Black-Box Forensics for Conversational LLM Agents