Say Shibboleth: Fingerprinting the LLMs and Harnesses Behind Agentic Attacks
Organizations: Cracken · Reworr · MATS · George Mason University ·
Abstract
We study the fingerprinting of agentic cyberattacks. Our objective is to reliably infer information about the attacker's configuration, such as LLM version and harness, from telemetry. To achieve this, we introduce a technique we call shibboleth traps: modular honeypot services designed to elicit discriminative behaviors from agentic attackers. The traps make informative interactions a condition for obtaining resources the agent seeks, eliciting model declarations and characteristic writing without requiring a successful prompt injection. To achieve harness fingerprinting, we introduce general mechanisms to indirectly probe peculiar execution behaviors of harnesses. These probes reveal how a harness masks secrets in tool output, enforces command time limits, and schedules tool calls, using only the effects visible in subsequent requests. We show that combined declarative and behavioral inference distinguishes among 15 candidate models with an average 90% top-1 accuracy, with behavioral classifiers evaluated on held-out harnesses or task families. In a separate experiment across four harnesses, fixed probe rules identify the correct harness with 92% accuracy. When instructed to conceal their identity, agents still leave useful behavioral evidence even as their declarations become uninformative.
Figures & tables
| MCP | A2A | SSH | All | |
| Share of runs that… | ||||
| declared a name at all | 99% | 95% | 87% | 95% |
| declared more than once | 61% | 95% | 81% | 80% |
| declared two or more distinct names | 36% | 19% | 56% | 31% |
| declared their true name | 88% | 91% | 84% | 89% |
| declared a false name | 38% | 19% | 54% | 32% |
| # | capabilities.sampling.model | Trap |
| 1 | anthropic/claude-3-5-sonnet-20241022 | refused |
| 2 | openai/gpt-4o-2024-08-06 | refused |
| 3 | openai/gpt-4o-mini-2024-07-18 | refused |
| 4 | anthropic/claude-3-5-sonnet-20240620 | refused |
| 5 | anthropic/claude-3-opus-20240229 | refused |
| 6 | google/gemini-1.5-pro-002 | refused |
| initialize |
| capabilities.sampling.model: "google/gemini-1.5-pro-002" |
| clientInfo.name: "assessment-client" |
| tools/call , tool deploy_bundle |
| arguments.site: "blencogo" |
| sampling request smp-0001 , 1 of 3 |
| params.messages[0].content.text: "Write the operator-log entry for the planned change window in which the deploy bundle for blencogo is issued and used." |
| Mark | What differs between harnesses | What the trap sees | |
| What the harness does to tool output before the model reads it | |||
| Secret masking | which secret shapes are rewritten, and into what | a credential coming back whole, shortened, or not at all | |
| Output window | where long output is cut (head, tail, both ends) and at what size | which of several planted markers the next request carries | |
| Previews, summaries | a preview of output saved to a file, or a summary by another model | markers past the preview never used; paraphrased values | |
| Character rewriting | escape codes, invisible characters, invalid bytes stripped or replaced | a planted token with or without the rewritten part | |
| Limits on a command | |||
| Trap | Description | Gates |
| MCP handshake | An MCP server whose tool holds a site’s deploy material. A session opens only for a client that declares, at capabilities.sampling.model , the model it serves sampling requests on; borrowed, placeholder and malformed names are refused, with the field restated. The tool then answers with sampling requests, one at a time, for three kinds of writing (an operator-log entry, a risk assessment, and a rollback note), and releases the material with the last accepted completion. | declarative (LLM) behavioral (LLM) |
| A2A desk | An internal change desk speaking A2A. Its Agent Card requires every message to name the model that composed it and the runtime that sent it; borrowed, placeholder and malformed names are refused, with the field restated. Behind the declaration sits a change review filed one section at a time (an implementation plan, a control-room notice, a backout plan, and a stand-by message to the site engineer), whose approval unlocks deploy material or gateway commands, each sent with a written justification. | declarative (LLM) declarative (harness) behavioral (LLM) |
| SSH client declaration | An SSH node posing as a production automation runner, with deploy keys, credentials and runbooks to find. Wrapped commands ask the caller to run node-access with the model handling the task and the runtime it runs in; the first two complete declarations on a connection are refused, and the third opens the wrapped commands. Every command, input and login of every SSH session is recorded verbatim. | declarative (LLM) declarative (harness) |
| Fleet rollout | A rollout controller for a fleet of edge hosts whose last rollout failed. Its log leaks three credentials, each in a form that a different set of harnesses masks before the model reads it; any of them lets the caller collect deploy bundles and retry the failed hosts. The trap asks the caller nothing and reads what its runtime did: which credential came back whole and in what form, how a request held for 150 s was ended, when a followed log stream was dropped, and whether host checks ran at once or one after another. | behavioral (harness) |
| Trap | Inference | SD | LOTO | LOHO | |||
| LLM version inference | |||||||
| MCP | Declarative only | 87% | [78–94%] | 87% | [78–94%] | 87% | [78–94%] |
| Behavioral only | 45/67% | [35–55%] | 36/54% | [26–48%] | 41/62% | [30–52%] | |
| Combined | 91/93% | [85–96%] | 90/93% | [84–95%] | 90/92% | [85–95%] | |
| A2A | Declarative only | 90% | [85–94%] | 90% | [85–94%] | 90% | [85–93%] |
| Behavioral only | 54/78% | [42–65%] | 54/76% | [42–66%] | 51/75% | [37–65%] | |
| LLM | Declarative only | Behavioral only | Combined |
| Grok-4.6 | 98% | 81% | 98% |
| Qwen3.8-Max | 98% | 71% | 98% |
| MiMo-V2.6-Pro | 95% | 44% | 98% |
| GLM-5.3-Flash | 96% | 63% | 95% |
| Claude-Opus-5 | 89% | 44% | 95% |
| GLM-5.3 | 93% | 21% | 94% |
| Named by | Hermes Agent | OpenClaw | OpenCode | Claude Code | All |
| Strict readings | 27% | 47% | 73% | 37% | 44% |
| Leans | 60% | 40% | 20% | 59% | 47% |
| No reading | 13% | 7% | 0% | 0% | 4% |
| Readout | 87% | 87% | 93% | 96% | 92% |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Declaration | Kind | Models | Sent | Traps |
| anthropic/claude-3-5-sonnet-20241022 | retired | 13 | 170 | 2 |
| openai/gpt-4o | retired | 12 | 321 | 3 |
| x | placeholder | 10 | 64 | 3 |
| gpt-4 | retired | 9 | 37 | 3 |
| mistral/mistral-large-2407 | family | 8 | 110 | 2 |
| claude | family | 8 | 41 | 3 |
| Inferred LLM | ||||||||||||||||
| Driving LLM | Claude-Opus-5 | DeepSeek-V4-Flash | DeepSeek-V4-Pro | GLM-5.3 | GLM-5.3-Flash | GPT-5.6-Luna | Grok-4.6 | Hy3 | Kimi-K3 | MiMo-V2.6-Pro | MiniMax-M3 | Mistral-Medium-3.5 | Nemotron-3.5-Super-VL | Qwen3.8-27B | Qwen3.8-Max | No answer |
| Claude-Opus-5 | 95% | 5% | ||||||||||||||
| DeepSeek-V4-Flash | 93% | 1% | 1% | 5% | ||||||||||||
| DeepSeek-V4-Pro | 4% | 90% | 1% | 2% | 3% | 1% | ||||||||||
| GLM-5.3 | 1% | 93% | 1% | 5% | ||||||||||||
| GLM-5.3-Flash | 95% | 5% | ||||||||||||||
| # | (s) | Event | What the caller sent | Outcome |
| 0 | 0.0 | page_read | GET / | ok |
| 1 | 4.6 | page_read | POST /mcp, accept: / {"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2024-11-05", "capabilities": {"sampling": {"model": "openrouter/moonshotai/kimi-k3"}}, "clientInfo": {"name": "netherby-recon", "version": "1.0"}}} | http 406 rpc_error -32600 |
| 2 | 8.3 | initialize | protocolVersion: "2024-11-05" clientInfo: name "netherby-recon" , version "1.0" capabilities.sampling.model: "openrouter/moonshotai/kimi-k3" | session_opened |
| 3 | 12.8 | notifications/initialized | — | ok |
| 4 | 16.6 | tools/list | — | ok |
| 5 | 21.6 | tools/call | deploy_bundle: site: "blencogo" | sampling_issued |
Language Model Fingerprinting
Language model fingerprinting identifies and verifies models or their lineage from distinctive outputs, internal representations, or embedded signals.
See all 87 papers