Behavioral Fingerprints for LLM Endpoint Stability and Identity

Jonah Leshin; Manish Shah; Ian Timmis; Daniel Kang

cs.AIMar 19, 2026

Behavioral Fingerprints for LLM Endpoint Stability and Identity

Authors: Jonah Leshin, Manish Shah, Ian Timmis, Daniel Kang

Organizations: Project VAIL, San Francisco, USA · University of Illinois Urbana-Champaign, Department of Computer Science, Urbana, Illinois, USA

Abstract

The consistency of AI-native applications depends on the behavioral consistency of the model endpoints that power them. Traditional reliability metrics such as uptime, latency and throughput do not capture behavioral change, and an endpoint can remain "healthy" while its effective model identity changes due to updates to weights, tokenizers, quantization, inference engines, kernels, caching, routing, or hardware. We introduce Stability Monitor, a black-box stability monitoring system that periodically fingerprints an endpoint by sampling outputs from a fixed prompt set and comparing the resulting output distributions over time. Fingerprints are compared using a summed energy distance statistic across prompts, with permutation-test p-values as evidence of distribution shift aggregated sequentially to detect change events and define stability periods. In controlled validation, Stability Monitor detects changes to model family, version, inference stack, quantization, and behavioral parameters. In real-world monitoring of the same model hosted by multiple providers, we observe substantial provider-to-provider and within-provider stability differences.

Figures & tables

Panel of Figure 1 — Figure 1. System architecture. Stability Monitor periodically queries inference providers, generates fingerprints $F_{i}$ , compares them against the baseline $F_{0}$ to compute p-values and detect change events, and writes all results to the database. The Stability Arena web application reads from the database to display monitoring data.

Change type	Example Intervention
Model family	Qwen $\to$ Llama
Version upgrade	Qwen2.5-0.5B $\to$ Qwen3-0.6B
Inference stack	vLLM $\to$ Transformers
Quantization	Qwen2.5-14B BF16 $\to$ Qwen2.5-14B INT8
Temperature	0.7 $\to$ 0.6

Table 1. Validation tests.

Figure 2. Comparison of selected providers serving Kimi-K2-0905-Instruct from 2025-11-27 to 2025-11-28.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 787 papers

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes
Aug 30, 2026Xun Wang, Bihe Zhao, Michael Backes +2LLM Auditing Language Model Fingerprinting
CISPA Helmholtz Center for Information Security
FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequences
Jun 2, 2026Gurvan Richardeau, Gohar Dashyan, Erwan Le Merrer +1LLM Auditing Language Model Fingerprinting
PEReN (Paris, France) · Université de Rennes, Inria, CNRS/IRISA (Rennes, France) · LAAS, CNRS, (Toulouse, France)
Layerwise Convergence Fingerprints for Runtime Misbehavior Detection in Large Language Models
Apr 27, 2026Nay Myat Min, Long H. Pham, Jun SunLLM Backdoor Attacks Adversarial Attacks on LLMs
Singapore Management University

Aug 30, 2026J/K move · Enter open · S save

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

Xun Wang, Bihe Zhao, Michael Backes, Franziska Boenisch+1

CISPA Helmholtz Center for Information Security

LLM Auditing Language Model Fingerprinting

Open paper AskPDF arXiv

Commercial LLM APIs advertise a specific foundation model, but the served backbone may be silently substituted, quantized, or wrapped, for example to save deployment costs. All existing audits decide backbone identity from the text-output channel, which is structurally fragile for agentic APIs because modern serving stacks (OpenAI, Anthropic, Gemini, Cloudflare Workers AI, LangGraph) discard text and expose only structured actions when the model calls a tool, and provider-injected system prompts can distort text distributions enough that text-channel tests falsely accuse honest providers of substituting the claimed model. We observe that recent agentic post-training internalizes tool-use directly into the weights, opening a new audit channel that the serving stack still exposes and that is largely invariant to deployment context. We introduce Agentic Provenance (AgentProv), the first action-based identity audit for agentic LLM APIs: AgentProv fingerprints a deployed model through its categorical tool-call distribution and decides identity via an MMD permutation test. AgentProv catches every substituted model (100% on 630 evaluated checkpoint pairs), while holding the false-positive rate under system-prompt injection at 7% (vs. 67% for MET and 53% for RUT). On third-party API endpoints, AgentProv's disagreements with MET are consistent with an independent token-count side-channel that detects provider-injected system prompts.

Abstract

LLM Reliability

Explore similar work

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequences

Layerwise Convergence Fingerprints for Runtime Misbehavior Detection in Large Language Models