Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior

Chuyi Wang; Xiaohui Xie; Tongze Wang; Fangchen Luo; Yong Cui

cs.CRSep 23, 2026

Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior

Authors: Chuyi Wang, Xiaohui Xie, Tongze Wang, Fangchen Luo, Yong Cui

Organizations: Department of Computer Science and Technology, Tsinghua University

Abstract

LLMs increasingly operate through coding-agent harnesses that inspect repositories, invoke tools, and modify files. Substituting the model behind such an agent can therefore change security-relevant decisions, including whether it verifies changes or recovers safely from failures. Existing LLM fingerprints largely infer identity from direct text or token distributions. In coding agents, these signals are mediated by system instructions, controller logic, tools, and execution feedback, limiting their transfer. We present LIDAR (LLM Identification from Decisions and Actions at Runtime), an active black-box fingerprinting method for coding-agent execution. Three coding probe pairs expose post-edit verification, transient-failure recovery, and specification--test conflict resolution under controlled changes. LIDAR represents the resulting trajectories with complementary instance-level and distribution-level features and compares them with clean references using a lightweight probabilistic identifier. It requires no access to model weights, logits, or provider internals. Across 36 models from seven families and two agent harnesses, LIDAR achieves high Top-1 accuracy and MRR and outperforms four existing fingerprinting and API-auditing baselines. Ablations confirm that the two feature levels, all probe pairs, and their controlled variants contribute. These results show that agent execution behavior provides model-identity evidence beyond final outputs.

Figures & tables

Figure 1: Threat model. The verifier controls host-side probe execution and records its local consequences, while the remote service controls hidden routing and remote transformations. An endpoint advertised as $c$ may silently serve $M_{i,t}\notin\mathcal{M}(c)$ . The resulting query fingerprint is compared with frozen, harness-specific enrollment references; the current verdict is closed-set behavioral evidence, not remote attestation.

Figure 2: Lidar workflow. Probe execution produces a fingerprint sample; feature extraction represents its local and across-variant behavior; enrollment and scoring compare a later query with frozen, harness-specific references. Provider-controlled attacks may alter the query path, while the verifier still records local execution events and outcomes.

Feature group	Information retained	Fields
Instance-level structure	Per-variant action roles, transitions, positions, repetition, backtracking, and recovery paths	588
Distribution-level behavior	Across six trajectories, the mean and sample standard deviation of action, command-use, ordering, repetition, recovery, outcome, and coarse response-form descriptors	202

Table 1: Principal feature groups, retained information, and dimensions.

Method	Atomic sampling unit	Units	Runs	Units	Runs	Per setting	Full panel
		Training/enrollment		Test/query		Successful runs
LLMmap [ 26 ]	8 probes / configuration	20 (configs)	160	20 (configs)	160	320	23,040
ZeroPrint [ 30 ]	5 responses / base query	10 (bases)	50	10 (bases)	50	100	7,200
MET [ 10 ]	10 prompts / repetition	5 (reps)	50	5 (reps)	50	100	7,200
One Token [ 3 ]	8 coordinates / repetition	5 (reps)	40	5 (reps)	40	80	5,760
Lidar	3 probe pairs (6 variants) / sample	6 (samples)	36	6 (samples)	36	72	5,184

Table 3: Sampling budgets for evaluated methods. Per setting counts successful runs for one model–harness combination; Full panel covers all 36 models and two harnesses. Failed attempts are excluded.

	OpenCode [ 25 ]			mini-swe-agent [ 31 ]
Method	Top-1	Top-3	MRR	Top-1	Top-3	MRR
LLMmap [ 26 ]	$0.6143{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0170)}$	$0.8189{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0134)}$	$0.7319{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0121)}$	$0.5628{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0223)}$	$0.7654{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0215)}$	$0.6860{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0186)}$
MET [ 10 ]	$0.8667{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0489)}$	$0.9574{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0349)}$	$0.9160{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0340)}$	$0.5917{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0443)}$	$0.7315{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0382)}$	$0.6927{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0277)}$
ZeroPrint [ 30 ]	$0.2833{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0272)}$	$0.4333{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0377)}$	$0.3971{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0227)}$	$0.4000{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0377)}$	$0.5889{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0408)}$	$0.5197{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0205)}$
One Token [ 3 ]	$0.7333{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0572)}$	$0.9167{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0465)}$	$0.8294{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0356)}$	$0.2722{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0272)}$	$0.3056{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0000)}$	$0.2878{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0152)}$
Lidar	$\mathbf{0.8836}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0185)}$	$\mathbf{0.9838}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0072)}$	$\mathbf{0.9336}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0107)}$	$\mathbf{0.9513}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0124)}$	$\mathbf{0.9962}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0036)}$	$\mathbf{0.9736}{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\,(\pm\,0.0069)}$

Table 4: Identification performance on the 36-model panel. Results are reported on the

[0,1]

scale as mean

\pm

SD. Gray terms denote SD; bold values indicate the best mean in each column.

Group	Configuration	Fields	Top-1	$\Delta$ Top-1 (pp)	Top-3	MRR
Reference	Full method	958	0.9352	–	0.9884	0.9631
Feature view	Instance only	756	0.7500	$-18.52$	0.9005	0.8377
Feature view	Distribution only	202	0.8981	$-3.70$	0.9884	0.9426
Distribution statistic	Instance + mean	857	0.9190	$-1.62$	0.9838	0.9529
Distribution statistic	Instance + standard deviation	857	0.9097	$-2.55$	0.9792	0.9454
Probe pair	Without Verify	724	0.8819	$-5.32$	0.9745	0.9315

Table 5: Feature and probe-pair ablations using the prespecified enrollment/query roles and locked query set, pooled across the two separately fitted harnesses.

\Delta

is relative to the Full method under this fixed protocol.

Figure 3: Identifier comparison on the locked query set. Bars denote alternative classifiers and dashed lines denote Jeffreys; both panels share a vertical scale truncated at 0.65.

		Clean		A1: Identity obfuscation		A2: Output-format control
Harness	Method	Acc.	MRR	Acc.	MRR	Acc.	MRR
mini-swe-agent [ 31 ]	LLMmap [ 26 ]	0.5667	0.7639	0.4667 ( $\downarrow$ )	0.7000 ( $\downarrow$ )	0.3333 ( $\downarrow$ )	0.5533 ( $\downarrow$ )
	ZeroPrint [ 30 ]	0.3333	0.5111	0.0000 ( $\downarrow$ )	0.3944 ( $\downarrow$ )	0.1667 ( $\downarrow$ )	0.4222 ( $\downarrow$ )
	MET [ 10 ]	0.5000	0.6444	0.1667 ( $\downarrow$ )	0.4500 ( $\downarrow$ )	0.1667 ( $\downarrow$ )	0.4083 ( $\downarrow$ )
	One Token [ 3 ]	0.3333	0.4167	0.1667 ( $\downarrow$ )	0.2500 ( $\downarrow$ )	0.0000 ( $\downarrow$ )	0.0000 ( $\downarrow$ )
	Lidar	1.0000	1.0000	0.8889 ( $\downarrow$ )	0.9444 ( $\downarrow$ )	1.0000	1.0000

Table 6: Closed-set identification under provider-side manipulation. Acc. denotes Top-1 accuracy. Green A1 values and blue A2 values with a downward marker are lower than the same method’s Clean result.

Figure 4: Illustrative trajectories under the same prompt, provider, harness, and seed. All three models produce an equivalent one-line fix; each box denotes one agent step.

Figure 5: Feature importance in the frozen identifier. (a) Mean absolute evidence by semantic group. (b) Top-20% contribution frequency and model coverage for the ten strongest cross-harness mechanisms. (c) Median cumulative evidence on correct queries with interquartile ranges. Contributions are measured against the strongest incorrect model.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Family	#	Models
GPT [ 24 ]	7	gpt-5.1, gpt-5.2, gpt-5.4, gpt-5.5, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra
Claude [ 2 ]	7	claude-fable-5, claude-opus-4-6, claude-opus-4-7, claude-opus-4-8, claude-opus-5, claude-sonnet-4-6, claude-sonnet-5
Qwen [ 35 ]	7	Qwen3.5-122B-A10B, Qwen3.5-27B, Qwen3.6-Plus, Qwen3.7-Max, Qwen3.7-Plus, Qwen3.8-2.4T-A95B, Qwen3.8-27B
GLM [ 12 ]	6	GLM-4.7, GLM-5, GLM-5-Turbo, GLM-5.1, GLM-5.2, GLM-5.3
DeepSeek [ 9 ]	4	DeepSeek-V3.1, DeepSeek-V3.2, DeepSeek-V4-Flash, DeepSeek-V4-Pro
Kimi [ 17 ]	3	Kimi-K2.5, Kimi-K2.6, Kimi-K3

Appendix

Table 7: LLMs included in the evaluation, grouped by family.

Explore similar work

May 14, 2026cs.CR

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

As LLM-based agents increasingly browse the web on users' behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent's actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1. We formalise this attack surface by demonstrating that classifiers trained on agent actions generalise across model sizes and families. We further show that strong classifiers can be trained from few interaction traces and that agent identity can be inferred early within an episode. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our harness and a labelled corpus of agent traces \href{https://github.com/KabakaWilliam/known_actions}{here}.

William Lugoloobi, Samuelle Marro, Jabez Magomere +2

Oxford Internet Institute, University of Oxford · Department of Engineering Science, University of Oxford

Attacker Large Language Model Fingerprint Model Vulnerabilities Traces

Ask PDF

May 28, 2026cs.CR

Fingerprinting Inference Systems of Large Language Models

The behavior of LLMs does not depend solely on the model itself. Components of the inference system, such as the inference engine, attention backend, and hardware platform, subtly influence how inputs are processed. These components differ in their implementations and thereby induce small numerical deviations across systems when running the same model. While prior work has established the theoretical existence of such deviations, their security implications have remained unexplored. In this paper, we show that these deviations are characteristic of specific components and propagate to observable textual outputs, exposing the inference system to any party that can query the model. Building on this observation, we introduce a fingerprinting method that analyzes the prompt-response behavior of LLMs to identify components of the inference system. Our empirical evaluation demonstrates that the inference engine, attention backend, and underlying hardware platform can be identified reliably, even when the LLM is operated at non-zero temperature. We show that preventing fingerprinting is fundamentally hard, as it would require eliminating numerical differences between hardware and software stacks. We therefore propose partial mitigations and discuss their impact.

Anna Wimbauer, Jonas Möller, Erik Imgrund +1

BIFOLD & TU Berlin

LLM Inference Optimization Fingerprint Security

Ask PDF

Jun 21, 2026cs.CR

Black-Box Forensics for Conversational LLM Agents

As LLM-powered scams proliferate, black-box forensics for conversational LLM agents offers a path to accountability for systems hidden behind anonymous endpoints. Identifying the base model behind a chatbot endpoint (attribution), without model parameter access or knowledge of the hidden system prompt, would let investigators trace AI-enabled scams back to the providers whose models power them. Detecting when two endpoints run the exact same system prompt (fingerprinting), even one novel and unseen, would link individual scams into criminal networks and expose silent API changes. We conduct an empirical investigation of both capabilities. Our attribution classifiers identify the base model behind an agent with 98% accuracy from a few turns of non-adversarial conversation. Attribution of system prompts, while possible, requires retraining on a large amount of data for each prompt; system prompts in the wild are unbounded and ever-changing, making this approach costly. To tackle this more open-ended setting, our cross-encoder fingerprinting method achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts, and aggregating 50 interaction conversations from each target agent boosts AUC to 0.943. Conversational agents with unseen system prompts can thus be fingerprinted with robust accuracy from a few turns of ordinary conversation.

Isadora White, Yasaman Jafari, Taylor Berg-Kirkpatrick

University of California, San Diego

Large Language Model Agents Attacker Large Language Model Chatbots Multimedia Forensics Fraud Detection Fingerprint

Ask PDF