Functional Subspace Watermarking for Large Language Models

Zikang Ding; Junhao Li; Suling Wu; Junchi Yao; Hongbo Liu; Lijie Hu

cs.CRMar 19, 2026

Functional Subspace Watermarking for Large Language Models

Authors: Zikang Ding, Junhao Li, Suling Wu, Junchi Yao, Hongbo Liu, Lijie Hu

Organizations: University of Electronic Science and Technology of China, Chengdu, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · South China University of Technology, Guangzhou, China

Abstract

Model watermarking utilizes internal representations to protect the ownership of large language models (LLMs). However, these features inevitably undergo complex distortions during realistic model modifications such as fine-tuning, quantization, or knowledge distillation, making reliable extraction extremely challenging. Despite extensive research on model-side watermarking, existing methods still lack sufficient robustness against parameter-level perturbations. To address this gap, we propose **Functional Subspace Watermarking (FSW)**, a framework that anchors ownership signals into a low-dimensional functional backbone. Specifically, we first solve a generalized eigenvalue problem to extract a stable functional subspace for watermark injection, while introducing an adaptive spectral truncation strategy to achieve an optimal balance between robustness and model utility. Furthermore, a vector consistency constraint is incorporated to ensure that watermark injection does not compromise the original semantic performance. Extensive experiments across various LLM architectures and datasets demonstrate that our method achieves superior detection accuracy and statistical verifiability under multiple model attacks, maintaining robustness that outperforms existing state-of-the-art (SOTA) methods.

Figures & tables

Figure 1: Conceptual comparison of watermark robustness under common model-side attacks. Top (Previous): Conventional internal watermarks are vulnerable to post-hoc modifications like fine-tuning, quantization, pruning, and distillation, which often result in signal erasure. Bottom ( FSW ): Our proposed framework anchors ownership signals into a stable functional backbone, ensuring that the watermark remains detectable even after significant parameter-level perturbations.

Model	Method	Functional Preservation				Detectability
Model	Method	PPL $\downarrow$	$\Delta$ PPL $\downarrow$	HellaSwag $\uparrow$	ARC-E $\uparrow$	Det. Score $\uparrow$	Bit Acc $\uparrow$	AUC $\uparrow$
LLaMA-2-7B-hf	Clean FT	5.30	0.00	66.40	71.60	0.47	–	–
LLaMA-2-7B-hf	FSW (Ours)	5.91	+0.61	64.80	71.00	6.09	100%	0.894
LLaMA-3-8B	Clean FT	5.83	0.00	68.60	79.60	-0.03	–	–
LLaMA-3-8B	FSW (Ours)	6.20	+0.37	67.80	75.80	4.00	100%	0.899
Qwen2.5-7B	Clean FT	6.35	0.00	67.00	75.80	1.37	–	–

Table 1: Performance comparison of Functional Subspace Watermarking (FSW) across diverse LLM backbones. Our method achieves near-perfect bit accuracy and high detection scores while maintaining competitive functional utility.

Model	Attack	Watermark Reliability			Model Quality
Model	Attack	Det. Score $\uparrow$	Ret. $\uparrow$	Bit Acc $\uparrow$	PPL $\downarrow$	$\Delta$ PPL $\downarrow$
LLaMA-2-7B-hf	Baseline	6.0938	100.00%	100.00%	5.91	+0.00
	Backbone Distillation	5.8438	95.90%	87.50%	5.91	+0.00
	LoRA FT	6.7812	111.28%	87.50%	5.58	-0.33
	Noise	5.9062	96.92%	100.00%	5.92	+0.01
	Pruning	6.7188	110.26%	87.50%	6.13	+0.22

Table 2: Robustness analysis of FSW under various model attacks. The results across multiple LLM backbones show that ownership signals remain recoverable even after significant parameter-level modifications.

Figure 3: Functional Preservation and Ownership Detectability.

Bits	No ECC (%)		Hamming (7,4) (%)
Bits	Bit Acc $\uparrow$	Msg Acc $\uparrow$	Bit Acc $\uparrow$	Msg Acc $\uparrow$
4	100.0	100.0	0.0	0.0
8	87.5	87.5	100.0	100.0
12	100.0	100.0	100.0	100.0
16	100.0	100.0	93.8	100.0
32	100.0	100.0	100.0	100.0

Table 3: Impact of Error-Correcting Codes (ECC) on message decoding accuracy (Llama-2-7B-hf). The results demonstrate that Hamming (7,4) successfully eliminates residual errors at the 8-bit critical point, ensuring irrefutable ownership verification.

Figure 4: Bit Capacity vs. Utility and Detectability

Variant	Detection Score $S\uparrow$			Bit Accuracy $\uparrow$			Utility (PPL) $\downarrow$
Variant	Pre	Post	$\Delta S$	Pre	Post	$\Delta$ Acc	Pre	Post	$\Delta$ PPL
Full FSW	6.09	5.84	-0.25	100.0%	87.5%	-12.5%	5.91	5.91	0.00
w/o Comp. Inv.	4.47	0.96	-3.51	100.0%	100.0%	0.00%	7.02	6.43	+0.60
w/o Anchored Proj.	3.08	-1.94	-5.02	75.0%	37.5%	-37.5%	7.31	6.82	+0.49
w/o Consistency Loss	4.59	-0.36	-4.95	100.0%	37.5%	-62.5%	7.19	6.59	+0.60
w/o Adaptive Thres.	4.91	-2.73	-7.64	87.5%	12.5%	-75.0%	7.16	6.85	+0.30

Table 4: Ablation study of FSW components on Llama-2-7B-hf. Pre and post denote metrics before and after the robustness attack. Each variant systematically removes a key design element (Compression Invariance, Anchored Projection, Consistency Loss, or Adaptive Thresholding) to evaluate its individual contribution to robustness and utility.

Figure 5: Robustness–Capacity Trade-off under Different Embedding Dimensions

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Variant	Original (Pre-Attack)			Attacked (Post-Attack)
Model	Variant	PPL $\downarrow$	Hella $\uparrow$	ARC $\uparrow$	PPL $\downarrow$	Hella $\uparrow$	ARC $\uparrow$
Llama-2-7B	Clean FT (No WM)	5.30	66.40%	71.60%	—	—	—
	Full DIFSW	5.91	64.80%	71.00%	5.91	65.00%	71.20%
	EmMark	416.15	65.80%	71.40%	5.73	65.80%	72.60%
	Weight Quant.	20.88	66.60%	71.60%	5.78	68.00%	72.40%
	Naive Top-k	7.30	66.80%	72.20%	6.75	66.60%	71.60%

Appendix

Table 5: Baseline Comparison under Model Distillation. We report perplexity (PPL) and downstream accuracies (HellaSwag and ARC-Challenge) before and after distillation for multiple watermarking baselines.

Model	$k$	Original (Pre-Attack)			Attacked (Post-Attack)			Ret. (%)
Model	$k$	PPL $\downarrow$	Score $\uparrow$	Acc $\uparrow$	PPL $\downarrow$	Score $\uparrow$	Acc $\uparrow$	Ret. (%)
Llama-2-7B	8	5.9661	5.0625	100.0%	6.0652	3.1250	50.0%	61.73
	16	5.9133	6.0938	100.0%	5.9139	5.8438	87.5%	95.90
	32	6.0213	4.9375	100.0%	6.0903	1.8125	62.5%	36.71
	48	5.9709	5.9375	87.5%	6.0322	2.7500	62.5%	46.32
	64	5.9611	5.1563	87.5%	6.2732	3.3281	87.5%	64.55

Appendix

Table 6: Comprehensive Scan of Embedding Dimension

k

across Different LLM Backbones. Pre and post denote metrics before and after the robustness attack (e.g., distillation or quantization). Ret.(%) indicates the score retention rate. Bold

k

values represent the optimal dimensions selected for each model.

Model	Significance Level ( $\alpha$ )	Threshold $T_{\alpha}$	Detection Score $S$	Margin ( $S-T_{\alpha}$ )	Detected
Mistral-7B	$1\mathrm{e}{-02}$	1.5838	3.8438	2.2600	✓
	$1\mathrm{e}{-03}$	2.1762	3.8438	1.6675	✓
	$1\mathrm{e}{-04}$	2.6639	3.8438	1.1798	✓
	$1\mathrm{e}{-06}$	3.4662	3.8438	0.3776	✓
	$1\mathrm{e}{-08}$	4.1321	3.8438	-0.2883	$\times$
DeepSeek-7B	$1\mathrm{e}{-02}$	0.2358	3.7500	3.5142	✓

Appendix

Table 7: Detection Thresholds and Significance Test Results across Different Significance Levels

\alpha

. FSW successfully rejects the null hypothesis

H_{0}

with a substantial positive margin even under extreme significance constraints (e.g.,

\alpha=10^{-8}

for DeepSeek).

Dataset	Method	Functional Preservation				Detectability
Dataset	Method	PPL $\downarrow$	$\Delta$ PPL $\downarrow$	HellaSwag $\uparrow$	ARC-E $\uparrow$	Det. Score $\uparrow$	Bit Acc $\uparrow$	AUC $\uparrow$
WikiText-2	Clean FT	5.83	0.00	68.60	79.60	-0.03	–	–
WikiText-2	FSW (Ours)	6.20	+0.37	67.80	75.80	4.00	100%	0.899
C4	Clean FT	8.69	0.00	79.84	77.74	-0.01	–	–
C4	FSW (Ours)	9.59	+0.90	77.38	69.28	4.88	100%	1.000

Appendix

Table 8: Generalization across different training datasets on Meta-LLaMA-3-8B. The results demonstrate that FSW maintains consistent functional utility and high detectability regardless of the fine-tuning corpus used.

Model	Candidates	Opt. $k$	Pre-Attack (Original)			Post-Attack (Robustness)			Ret. (%)
Model	Candidates	Opt. $k$	PPL $\downarrow$	Score $\uparrow$	Acc $\uparrow$	PPL $\downarrow$	Score $\uparrow$	Acc $\uparrow$	Ret. (%)
LLaMA-2-7B-hf	$\{8,16,32,\dots\}$	16	5.91	6.09	100.0%	5.91	5.84	87.5%	95.90
LLaMA-3-8B	$\{8,16,32,\dots\}$	32	6.20	4.00	100.0%	6.12	3.17	62.5%	79.30
Qwen2.5-7B	$\{8,16,32,\dots\}$	32	7.24	5.09	100.0%	7.66	3.14	75.0%	61.66
Mistral-7B-v0.3	$\{8,16,32,\dots\}$	64	17.25	3.84	100.0%	6.59	3.23	75.0%	84.15
DeepSeek-7B-Chat	$\{8,16,32,\dots\}$	16	77.31	3.75	100.0%	61.39	1.66	100.0%	44.17

Appendix

Table 9: Optimal embedding dimension

k

selection across backbone models. The optimal

k

is determined by maximizing the retention rate (Ret.%) of detection scores after robustness attacks, ensuring an effective trade-off between watermark capacity and structural stability.

Training Phase (Qwen2.5-7B)	Time (s)	Steps	GPU Mem (MB)	CPU Mem (MB)
Standard FT (Baseline)	1101.55	7500	29820.63	6111.39
Phase 1: Fisher Estimation	76.61	250	29306.54	20815.56
Phase 2: Inv. Estimation	11.66	250	29287.93	20815.56
Phase 3: Subspace Construction	3.02	1	29268.78	20815.56
Phase 4: Watermark Embedding	1764.38	7500	56410.34	20815.56
Total FSW	1855.68	8001	56410.34	20815.56

Appendix

Table 10: Training overhead of FSW compared to standard fine-tuning. While embedding introduces additional memory requirements due to dual-model consistency constraints, the overall time overhead remains within a manageable range for 7B-scale models.

Model	Parameters	Hidden Size	Layers	Watermark Layer ( $\ell$ )
GPT-2	124M	768	12	8
LLaMA-2-7B-hf	7B	4096	32	16
Meta-LLaMA-3-8B	8B	4096	32	16
Qwen2.5-7B	7B	3584	28	14
Mistral-7B-v0.3	7B	4096	32	16
DeepSeek-7B-Chat	7B	4096	30	15

Appendix

Table 11: Architectural specifications of the evaluated language models. The watermark is consistently embedded in the middle layer (

\ell\approx L/2

) for each architecture to balance representation stability and expressiveness.

LLM Watermarking

Large language model watermarking embeds or detects signals in text and models to trace provenance, ownership, and unauthorized use.

See all 111 papers

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization
Jul 11, 2026Huy Phan, Kieu Dang, Ojaswi Dulal +4Data Provenance LLM Watermarking
Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA
OpenStamp: A Watermark for Open-Source Language Models
Aug 28, 2026Miroojin Bakshi, Saksham Rastogi, Danish PruthiText Watermarking LLM Watermarking
Indian Institute of Science · Carnegie Mellon University
Robust LLM Watermarking with Minimal Semantic Distortion for IP Protection
May 22, 2026Kieu Dang, Phung Lai, NhatHai Phan +2Text Watermarking Language Model Fingerprinting
State University of New York at Albany · New Jersey Institute of Technology · Microsoft +1

Jul 11, 2026J/K move · Enter open · S save

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

Huy Phan, Kieu Dang, Ojaswi Dulal, Aiham AL Shukairi+3

Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA

Data Provenance LLM Watermarking Robust Watermarking

Open paper AskPDF arXiv

As large language models (LLMs) become widely deployed, their outputs can be copied, transformed, and redistributed at scale without reliable evidence of origin, creating risks for trust, accountability, intellectual property (IP) protection, and high-stakes decision-making. LLM watermarking addresses this problem by embedding detectable signals into text during or after generation. However, existing methods vary in design assumptions, threat models, and evaluation criteria, while deployment choices such as watermark placement, detection authority, and key management affect reliability, security, and scalability. This paper systematizes LLM watermarking as provenance infrastructure for large-scale data ecosystems. We organize existing approaches along four deployment dimensions: insertion point, verification authority, operational state, and transformation threat model, and relate them to the big data requirements of Volume, Velocity, Variety, Veracity, and Value. We further introduce a Big Data Watermarking Readiness framework centered on four deployment workloads: online generation, streaming detection, transformation pipelines, and ecosystem governance. The framework connects these workloads to system-level requirements including throughput, false-positive control, robustness, cross-domain reliability, governance, and downstream utility. Our analysis highlights a gap between benchmark performance and deployment readiness: false positives accumulate at scale, repeated transformations weaken watermark signals, computational overhead can limit online deployment, and centralized verification can create governance bottlenecks. We conclude with an evaluation blueprint and research directions for scalable, trustworthy provenance in big data ecosystems.

Abstract

LLM Watermarking

Explore similar work

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

OpenStamp: A Watermark for Open-Source Language Models

Robust LLM Watermarking with Minimal Semantic Distortion for IP Protection