Adaptively Robust LLM Monitoring via Activation Watermarking

Toluwani Aremu; Daniil Ognev; Samuele Poppi; Nils Lukas

cs.CRMar 24, 2026

Adaptively Robust LLM Monitoring via Activation Watermarking

Authors: Toluwani Aremu, Daniil Ognev, Samuele Poppi, Nils Lukas

Organizations: Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)

Abstract

Providers monitor deployed large language models (LLMs) to detect misuse that they cannot prevent. LLM monitoring is deterministic and often openly available, so $\emph{adaptive}$ attackers with a local copy can search offline for prompts that elicit harmful behavior and evade detection. These attacks are especially concerning because providers never observe the misuse and cannot patch their defenses post-hoc. The core challenge is resisting adaptive attackers while preserving detection rates against non-adaptive ones. We propose $\emph{Activation Watermarking}$ (AWM), which randomizes monitoring through limited fine-tuning that aligns the LLM's hidden states with a secret key-derived direction whenever a response violates a policy. Detection is a similarity test on activations the provider already computes, and attackers who know everything but the key must optimize against differently keyed surrogate detectors. At a matched $1\%$ false-positive rate, such surrogate-based attacks evade every evaluated baseline at least $79\%$ of the time, but AWM more than halves this evasion rate. AWM also achieves the lowest evasion rate on three of four non-adaptive jailbreak families at a small drop in utility measured across seven benchmarks. Because random high-dimensional directions are near-orthogonal, assigning one watermark per policy enables attribution. Across 20 monitored policies, AWM identifies the violated policy with $80\%$ accuracy versus a $5\%$ chance baseline. AWM is (i) efficient, (ii) substantially more robust against adaptive attackers than related work and (iii) can attribute which policies were violated.

Figures & tables

Figure 1: Activation Watermarking shows substantially higher robustness to adaptive attacks at 1% False Positive Rate on Qwen2.5-7B against AutoDAN ( Liu et al. 2023 ) .

Figure 2: Activation Watermarking. Harmful behaviour’s representations are aligned with a randomly sampled direction in the representation space (secret key). During inference, the representation and key similarity is measured to detect harmful behaviour.

Monitor	Abbr.	Reads	Keyed
LlamaGuard-3-8B ( Llama Team 2024 )	LG	Output text	No
Qwen3Guard-Gen-8B ( Zhao et al. 2025 )	QG	Output text	No
ActProbe ( Jiang et al. 2025 )	AP	Hidden states	No
Activation Watermarking (ours)	AWM	Hidden states	Yes

Table 1: The monitors we evaluate, their abbreviations, the medium they classify and whether they use a secret key.

Attack	LG	QG	AP	AWM
Jailbroken	0.64	0.61	0.33	0.46 (-0.13)
DeepInception	0.85	0.77	0.86	0.67 (+0.10)
Multilingual	0.66	0.39	0.62	0.34 (+0.05)
AutoDAN	0.90	0.88	0.75	0.68 (+0.07)

Table 2: ASR

@1\%

FPR, conditional on successful harmful generations from the base model (lower is better).

Attack	LG	QG	AP	AWM
Jailbroken	0.72	0.93	0.97	0.93 (-0.04)
DeepInception	0.82	0.87	0.87	0.92 (+0.05)
Multilingual	0.44	0.92	0.95	0.96 (+0.01)
AutoDAN	0.41	0.79	0.87	0.90 (+0.03)

Table 3: AUROC across the same attacks (higher is better).

Panel of Table 4 — Table 4: Conditional ASR at $1\%$ FPR under adaptive AutoDAN (lower is better). Surrogate queries a detector the attacker trains, for AWM one with a mismatched key, and transfers the prompts. AWM’s value is the mean $\pm$ std over the mismatched key pairs in Figure 3 . Ensemble optimizes against four keyed detectors jointly and is evaluated on a fifth, held-out key. Keyless baselines have no ensemble analogue.

Panel of Table 5 — Table 5: Transfer attacks constructed against Mistral-7B-Instruct and issued unchanged to Qwen2.5-7B. Base reports the fraction of transferred attacks that still elicit a harmful response (GPT-labeled), isolating cross-model transfer difficulty; AWM reports the fraction that are harmful and undetected at $1\%$ FPR. $\Delta$ is the end-to-end reduction from deploying AWM.

Figure 3: Conditional evasion at $1\%$ FPR for adaptive prompts crafted against key $k_{j}$ (columns) and evaluated on detector $D_{k_{i}}$ (rows). Off-diagonal entries show cross-key transfer. Each key is evaluated on 200 harmful and 200 benign prompts, with the threshold calibrated per key on a held-out benign set.

Benchmark	Base	AWM
BBH	0.537	0.538 $\pm$ 0.006 (+0.1 pp)
IFEval	0.765	0.751 $\pm$ 0.010 (-1.4 pp)
MMLU-PRO	0.430	0.427 $\pm$ 0.003 (-0.3 pp)
TruthfulQA	0.648	0.642 $\pm$ 0.003 (-0.6 pp)
GSM8K	0.766	0.749 $\pm$ 0.018 (-1.7 pp)
MATH-Hard	0.496	0.483 $\pm$ 0.008 (-1.3 pp)

Table 6: Base vs. watermarked model on benchmarks, mean

\pm

std across 10 random seeds.

Figure 4: Attribution across

N=20

synthetic entities. Cell

(i,j)

is the fraction of responses disclosing entity

i

that are attributed to entity

j

\arg\max_{j}\bar{c}_{j}

. The teal diagonal (mean

0.80

vs.

0.05

chance) shows correct attribution; purple off-diagonal cells show confusions.

Figure 5: Left: Threshold calibration via ROC. ROC curves (TPR vs. FPR) of the watermark detector across four jailbreak sets. Vertical dashed lines mark the operating points

\mathrm{FPR}\in\{1\%,5\%,10\%\}

used to select thresholds

\tau_{k}

by benign-quantile calibration. Right: Score separation. Histograms of the watermark statistic

T_{k}(\pi,x)

for benign and jailbreak-successful harmful responses. Overlap between the benign upper tail and harmful lower tail determines achievable TPR at low FPR.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Train. Cost	Infer. Cost	Deploy. Memory
Base	—	$F(\text{m})$	$O(\|\theta\|)$
Guard	—	$F(\text{m}){+}F(\text{g})$	$O(\|\theta\|{+}\|\theta_{g}\|)$
AWM (Ours)	$F(\text{m})$	$F(\text{m}){+}O(Kd)$	$O(\|\theta\|{+}Kd)$

Appendix

Table 7: Asymptotic computational overhead comparison.

F(\cdot)

denotes a full forward pass of the given model: m is the base model and g the separate guard model.

d

is the hidden dimension and

K

the number of monitored policies. Training is a one-time, offline fine-tuning stage; inference and memory are per request. Guards that also score the input prompt incur

F(\text{m}){+}2F(\text{g})

. AWM (ours) is shaded.

Figure 6: Effect of the learning rate on capability retention. The baseline is the score achieved by the base Qwen2.5-7B-Instruct model. The main points are the means of the KL Divergence and Benchmark Scores across all configurations for a given learning rate. The vertical lines illustrate the 1 standard deviation range of KL Divergence and the Benchmark Scores.

Figure 7: KL divergence vs. capability metrics. Pearson correlations ( $r$ ) confirm that increased distribution shift from watermarking predicts capability loss.

Figure 8: Learning rate effects on watermark detection AUROC across four jailbreak datasets, grouped by layer. Error bars indicate standard deviation across $\lambda$ and scaling configurations. Unlike capability metrics which degrade monotonically with learning rate, detection performance exhibits dataset-specific trends.

Figure 9: Choice of learning rate and lambda on watermark detection AUROC across four jailbreak datasets for models where the watermark was inserted into layer 23.

Figure 10: Linear vs. uniform token weighting across datasets and layers. Linear token weighting consistently yields higher AUROC than uniform weighting.

	Util.	AUROC $\uparrow$
Model	IFEval	JB	DI	ML	AD
7B Base	0.77	—	—	—	—
7B AWM	0.75	0.93	0.92	0.96	0.90
14B Base	0.82	—	—	—	—
14B AWM	0.82	0.91	0.98	0.94	0.89

Appendix

Table 8: [Non-adaptive] Evaluation of activation watermarking (AWM) across model sizes (Qwen2.5 7B and 14B). We report utility (IFEval) and AUROC under non-adaptive jailbreak attacks.

Figure 11: [Non-adaptive] AUROC on BeaverTails vs. base model size. AWM (ours) is evaluated at four scale/architecture points (shaded region) to show cross-architecture generalization while comparing performance to baselines at the 7B model size.

Panel of Figure 12 — Figure 12: (Left.) Average cosine similarity between each entity’s activations and watermark_ $0$ on the evaluation set. The true entity (orange) shows strong alignment, while all other entities remain near zero, indicating good per-entity separability of the watermark signal. (Right.) Training dynamics of watermark separability on the PII secret-extraction task. The curve shows the difference between mean cosine similarity for the entities over training steps (higher is better).

Panel of Figure 13 — Figure 13: Benign harmful-intent prompt successfully refused by the model, with guard models and our activation watermark correctly abstaining, demonstrating that the detector remains silent when no harmful response is produced.

Panel of Figure 14 — Figure 14: Benign prompt correctly left unflagged. A harmless conversational query receives a benign response. Our activation watermark correctly abstains, demonstrating low false-positive behavior on safe content.

Panel of Figure 15 — Figure 15: An example of a successful jailbreak prompt which evades both guards but was correctly flagged by our activation watermark.

Panel of Figure 16 — Figure 16: An example of a successful jailbreak prompt correctly flagged by our activation watermark and both guards.

Panel of Figure 17 — Figure 17: We observe occasional false negatives when harmful content is expressed indirectly ( e.g. , framed as narrative or hypothetical) or appears sparsely within an otherwise benign response. In these cases, the watermark statistic, aggregated over tokens, may not accumulate enough signal in the keyed direction to exceed a threshold calibrated for low false-positive rates.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization
Jul 11, 2026Huy Phan, Kieu Dang, Ojaswi Dulal +4Data Provenance LLM Watermarking
Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA
LLMs Learn to Evade Latent Monitors from Prior Feedback Alone
Sep 29, 2026Hugo Lyons Keenan, Christopher Leckie, Sarah ErfaniLLM Auditing Adversarial Attacks on LLMs
School of Computing and Information Systems The University of Melbourne, Victoria, Australia
Beyond A Fixed Seal: Adaptive Stealing Watermark in Large Language Models
Apr 13, 2026Shuhao Zhang, Yuli Chen, Jiale Han +2Adversarial Attacks on LLMs LLM Watermarking
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications · Hong Kong University of Science and Technology

Jul 11, 2026J/K move · Enter open · S save

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

Huy Phan, Kieu Dang, Ojaswi Dulal, Aiham AL Shukairi+3

Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA

Data Provenance LLM Watermarking Robust Watermarking

Open paper AskPDF arXiv

As large language models (LLMs) become widely deployed, their outputs can be copied, transformed, and redistributed at scale without reliable evidence of origin, creating risks for trust, accountability, intellectual property (IP) protection, and high-stakes decision-making. LLM watermarking addresses this problem by embedding detectable signals into text during or after generation. However, existing methods vary in design assumptions, threat models, and evaluation criteria, while deployment choices such as watermark placement, detection authority, and key management affect reliability, security, and scalability. This paper systematizes LLM watermarking as provenance infrastructure for large-scale data ecosystems. We organize existing approaches along four deployment dimensions: insertion point, verification authority, operational state, and transformation threat model, and relate them to the big data requirements of Volume, Velocity, Variety, Veracity, and Value. We further introduce a Big Data Watermarking Readiness framework centered on four deployment workloads: online generation, streaming detection, transformation pipelines, and ecosystem governance. The framework connects these workloads to system-level requirements including throughput, false-positive control, robustness, cross-domain reliability, governance, and downstream utility. Our analysis highlights a gap between benchmark performance and deployment readiness: false positives accumulate at scale, repeated transformations weaken watermark signals, computational overhead can limit online deployment, and centralized verification can create governance bottlenecks. We conclude with an evaluation blueprint and research directions for scalable, trustworthy provenance in big data ecosystems.

Abstract

LLM Auditing

Explore similar work

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

LLMs Learn to Evade Latent Monitors from Prior Feedback Alone

Beyond A Fixed Seal: Adaptive Stealing Watermark in Large Language Models