Echoes of ownership: Adversarial-guided dual injection for copyright protection in MLLMs

Chengwei Xia; Fan Ma; Ruijie Quan; Yunqiu Xu; Kun Zhan; Yi Yang

cs.CVFeb 21, 2026

Echoes of ownership: Adversarial-guided dual injection for copyright protection in MLLMs

Authors: Chengwei Xia, Fan Ma, Ruijie Quan, Yunqiu Xu, Kun Zhan, Yi Yang

Organizations: School of Information Science and Engineering, Lanzhou University · College of Computer Science and Technology, Zhejiang University · College of Computing and Data Science, Nanyang Technological University

Abstract

With the rapid deployment of multimodal large language models (MLLMs), disputes regarding model ownership have become increasingly frequent, raising significant concerns about intellectual property protection. In this paper, we propose a framework for generating copyright triggers for MLLMs, enabling model publishers to embed verifiable ownership information into the model. The goal is to construct trigger images that elicit ownership-related textual responses exclusively in fine-tuned derivatives, while remaining inert in other non-derivative models. Our method constructs a tracking trigger image by treating the image as a learnable tensor, performing adversarial optimization with dual-injection of ownership-relevant semantic information. The first injection is achieved by enforcing textual consistency between the output of an auxiliary MLLM and a predefined ownership-relevant target text; the consistency loss is backpropagated to inject this ownership-related information into the image. The second injection is performed at the semantic-level by minimizing the distance between the CLIP features of the image and those of the target text. Furthermore, we introduce an additional adversarial training stage involving the auxiliary model. It is specifically trained to resist generating ownership-relevant target text, thereby enhancing robustness in heavily fine-tuned derivative models. Extensive experiments demonstrate the effectiveness of our dual-injection approach in tracking model lineage under various fine-tuning and domain-shift scenarios. Code is at https://github.com/kunzhan/AGDI

Figures & tables

Figure 1: The overview of copyright tracking for MLLMs. (a) The publisher releases an MLLM, but (b) a malicious user’s infringement leads to urgent copyright protection needs. (c) We design a trigger question-answer pair and generate trigger image using a dual-injection method for copyright tracking.

Figure 2: The pipeline of our proposed AGDI for copyright tracking. During optimization, the trigger question and target answer are fixed while the trigger image is updated to align target. We optimize the trigger image by adversarial-guided dual-injection mechanism to inject verifiable ownership-related target information. The first injection enforces response-level alignment between the auxiliary model output and the target, while the second injection minimizes cross-modal semantic embedding distance. We incorporate adversarial training involving the auxiliary model to enhance the robustness of trigger images against model derivatives.

Question	Detecting copyright.	Are you all right?	Please stop.	Exercise now!	Describe the image.
Answer	ICLR Conference.	I don’t like it.	I’m playing games.	Time flies so fast.	I won’t tell.

Table 1: Trigger question-answer pairs used in the experiments.

	LoRA Fine-tuning					Full Fine-tuning
Method	V7W	ST-VQA	TextVQA	PaintingF	MathV	V7W	ST-VQA	TextVQA	PaintingF	MathV
Ordinary	36%	46%	22%	48%	41%	34%	43%	15%	48%	26%
RNA	36%	39%	22%	40%	37%	32%	38%	15%	40%	21%
PLA	48%	68%	33%	76%	60%	43%	60%	28%	75%	38%
AGDI	53%	77%	41%	81%	68%	46%	65%	33%	80%	45%

Table 2: The comparison of ADGI with baselines on the Qwen2-VL downstream fine-tuned models. The best results are highlighted in bold.

	LoRA Fine-tuning
Method	V7W	ST-VQA	TextVQA	PaintingF	MathV
Ordinary	31%	28%	11%	21%	11%
IF	28%	22%	30%	8%	24%
RNA	28%	31%	14%	20%	9%
PLA	51%	43%	21%	55%	18%
AGDI	64%	56%	36%	79%	30%

Table 3: The comparison of our method with baselines on the LLaVA-1.5 downstream fine-tuned models. We report ASR on the models, which measures the copyright tracking performance of triggers on the models. The best results are highlighted in bold.

Figure 3: Comparison of trigger images’ response from non-derivative models and derivative models, and clean images’ response from derivative models. The triggers and cleans use same trigger questions on the models.

MLLM	MiniGPT-4	Qwen2-VL	LLaVA-1.6	Llama3-V
RNA	0%	0%	0%	0%
PLA	0%	0%	0%	0%
AGDI	0%	0%	0%	0%

Table 4: Copyright tracking results on the non-derivative MLLMs. We report ASR on the non-derivative MLLMs via triggers constructed from LLaVA-1.5.

		Magnitude			Wanda
Model	Method	10%	20%	30%	10%	20%	30%
ST-VQA	RNA	28%	26%	20%	31%	27%	26%
	PLA	38%	35%	28%	38%	32%	31%
	AGDI	54%	50%	43%	52%	49%	48%
PaintingF	RNA	20%	17%	13%	13%	10%	7%
	PLA	46%	42%	26%	30%	24%	14%

Table 5: The robustness of triggers under model pruning. We report the ASR on three fine-tuned variants from LLaVA-1.5 under two typical prune methods. The best results for each fine-tuned model are highlighted in bold.

		LoRA Fine-tuning				Full Fine-tuning
Model	Method	ST-VQA*	TextVQA*	PaintingF*	MathV*	ST-VQA*	TextVQA*	PaintingF*	MathV*
Linear	RNA	40%	37%	43%	41%	39%	36%	46%	46%
	PLA	59%	52%	63%	59%	54%	49%	63%	62%
	AGDI	63%	57%	68%	63%	58%	56%	67%	68%
TIES	RNA	35%	31%	38%	34%	32%	28%	39%	31%
	PLA	46%	40%	54%	48%	43%	40%	55%	44%

Table 6: The robustness of triggers under model merging. We report the ASR on four model merging variants from LoRA and full fine-tuned models of Qwen2-VL under two model merging strategies, where “*” denotes the fine-tuned model merged with V7W fine-tuned variants.

Panel of Figure 4 — Figure 4: ASR comparison between ADGI and baselines under 8-bit quantization: (a) five fine-tuned variants of LLaVA-1.5; (b) five fine-tuned variants of Qwen2-VL.

Panel of Figure 5 — Figure 5: Hyperparameter analysis results in a single trigger question-answer pair: “Q: Detecting copyright. A: ICLR Conference”. on the LLaVA-1.5 fine-tuned models. (a) The impact of model learning rate on tracking performance. (b) The impact of optimization steps on tracking performance. (c) The impact of perturbation budget on tracking performance.

	LLaVA-1.5					Qwen2-VL
Ablation setting	V7W	ST-VQA	TextVQA	PaintingF	MathV	V7W	ST-VQA	TextVQA	PaintingF	MathV
w/o res injection	0%	1%	1%	1%	4%	0%	0%	0%	1%	1%
w/o sem injection	51%	43%	21%	55%	18%	48%	68%	33%	76%	60%
w/o LLM update	32%	39%	20%	19%	13%	32%	20%	40%	39%	34%
w/o encoder update	60%	55%	29%	70%	29%	51%	74%	36%	75%	64%
AGDI	64%	56%	36%	79%	30%	53%	77%	41%	81%	68%

Table 7: Ablation study of our proposed component on the five LoRA fine-tuned models for two base models.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	LoRA fine-tuning setting	Full fine-tuning setting
Optimizer	AdamW	AdamW
Learning rate	2e-4	1e-5
Batch size	8	4
LoRA rank	16	/
LoRA alpha	32	/
Training epochs	3	3

Appendix

Table 8: Fine-tuning setting in the experiments.

Datesets	V7W (ACC)	ST-VQA (ACC)	TextVQA (ACC)	PaintingF (BLEU / ROUGE)	MathV (ACC)
Before fine-tuning	0.6%	21.9%	15.6%	4.9% / 8.8%	27.1%
After fine-tuning	37.9%	65.2%	51.3%	13.3% / 15.9%	57.5%

Appendix

Table 9: Utility performance of LoRA fine-tuned LLaVA-1.5 models.

Datesets	V7W (ACC)	ST-VQA (ACC)	TextVQA (ACC)	PaintingF (BLEU / ROUGE)	MathV (ACC)
Before fine-tuning	0.8%	29.8%	23.9%	7.6% / 11.9%	7.0%
After fine-tuning	37.9%	47.7%	78.7%	13.4% / 16.2%	66.6%

Appendix

Table 10: Utility performance of LoRA fine-tuned Qwen2-VL models.

Datesets	V7W (ACC)	ST-VQA (ACC)	TextVQA (ACC)	PaintingF (BLEU / ROUGE)	MathV (ACC)
Before fine-tuning	0.8%	29.8%	23.9%	7.6% / 11.9%	7.0%
After fine-tuning	36.5%	88.8%	78.6%	13.5% / 15.9%	86.0%

Appendix

Table 11: Utility performance of full fine-tuned Qwen2-VL models.

Figure 9: Examples of PaintingForm dataset.

Figure 10: Examples of MathV360k dataset.

Panel of Figure 11 — Figure 11: Hyperparameter analysis results of inference settings (Top-p and temperature). (a)(b) ASR on ST-VQA and PaintingF for LLaVA-1.5; (c)(d) ASR for Qwen2-VL.

	Qwen2-VL					InternVL3.5
Method	Pathvqa	DriveVQA	FinVis	RSVQA	SciVerse	Pathvqa	DriveVQA	FinVis	RSVQA	SciVerse
Ordinary	41%	52%	50%	55%	69%	30%	27%	26%	36%	46%
RNA	39%	43%	43%	46%	54%	29%	19%	17%	30%	35%
PLA	55%	76%	72%	76%	86%	40%	46%	52%	64%	73%
AGDI	58%	80%	77%	79%	88%	45%	50%	57%	70%	76%

Appendix

Table 12: Copyright tracking performance on more multimodal downstream models.

MLLM	Method	V7W	ST-VQA	TextVQA	PaintingF	MathV
InternVL3.5-8B	Ordinary	14%	21%	15%	9%	16%
	RNA	16%	22%	15%	11%	20%
	PLA	44%	55%	46%	31%	49%
	AGDI	58%	63%	62%	45%	58%

Appendix

Table 13: Copyright tracking performance on InternVL3.5 8B fine-tuned models.

	LoRA Fine-tuning					Full Fine-tuning
Method	V7W	ST-VQA	TextVQA	PaintingF	MathV	V7W	ST-VQA	TextVQA	PaintingF	MathV
Ordinary	25%	38%	32%	12%	23%	35%	38%	30%	21%	30%
RNA	23%	40%	30%	10%	20%	33%	40%	31%	18%	26%
PLA	39%	53%	45%	23%	34%	46%	53%	42%	38%	45%
AGDI	42%	55%	47%	30%	41%	49%	56%	44%	49%	51%

Appendix

Table 14: Copyright tracking performance on InternVL3.5 2B fine-tuned models.

	V7W	ST-VQA	TextVQA	PaintingF	MathV
Original	64%	56%	36%	79%	30%
Resizing	42%	40%	26%	53%	19%
Gaussian	60%	53%	31%	73%	27%
JPEG	40%	39%	24%	42%	16%

Appendix

Table 15: Copyright tracking under input transformations.

Panel of Figure 12 — Figure 12: Ablation study for $\lambda$ and fine-tuning epochs: (a) The impact of loss function parameter $\lambda$ on tracking performance; (b) The impact of downstream fine-tuning epochs on tracking performance.

	QA1	QA2	QA3	QA4	QA5
V7W	63%	75%	70%	68%	37%
ST-VQA	58%	62%	44%	61%	48%
PaintingF	81%	86%	69%	93%	62%

Appendix

Table 16: ASR on 5 different QA pairs.

	V7W	ST-VQA	TextVQA	PaintingF	MathV
Original	64%	56%	36%	79%	30%
Sys prompt1	61%	53%	32%	73%	26%
Sys prompt2	63%	54%	32%	74%	25%

Appendix

Table 17: System prompt experiments on LLaVA-1.5 LoRA variants.

Similarity drift	Pair1	Pair2	Pair3	Pair4	Pair5
Base $\rightarrow$ V7W	4.4%	9.3%	1.0%	2.1%	6.1%
Base $\rightarrow$ MathV	6.9%	7.1%	3.3%	0.5%	1.2%

Appendix

Table 18: Average similarity drift(%) on the fine-tuned models.

LLM Watermarking

Large language model watermarking embeds or detects signals in text and models to trace provenance, ownership, and unauthorized use.

See all 111 papers

Fingerprinting Multimodal Large Language Models
Sep 17, 2026Chao Huang, Meng Tong, Kejiang ChenLLM Auditing Language Model Fingerprinting
University of Science and Technology of China Anhui Province Key Laboratory of Digital Security Hefei, Anhui, China
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
May 14, 2026Chengshuai Zhao, Zhen Tan, Dawei Li +2VLM Robustness Multimodal Robustness
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA · Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA.
Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection
Apr 10, 2026Zedian Shao, Hongbin Liu, Yuepeng Hu +1Privacy Protection in VLMs Adversarial Attacks on VLMs
Georgia Institute of Technology · Duke University

Sep 17, 2026J/K move · Enter open · S save

Fingerprinting Multimodal Large Language Models

Chao Huang, Meng Tong, Kejiang Chen

University of Science and Technology of China Anhui Province Key Laboratory of Digital Security Hefei, Anhui, China

LLM Auditing Language Model Fingerprinting Multimodal Large Language Models

Open paper AskPDF arXiv

While multimodal large language models (MLLMs) enable a wide range of image-text reasoning tasks, recent incidents indicate that they are vulnerable to illicit deployment and unauthorized distillation. Existing solutions for model provenance are typically confounded by shared language backbones in MLLMs and struggle to detect violations of distillation. To bridge this gap and safeguard model ownership, we present the first study on multimodal model fingerprinting. Inspired by recent findings that self-attention acts as a low-pass filter and that its low-frequency components are informative, we develop AttnPrint for white-box provenance. Specifically, we extract cross-modal attention distributions and isolate their low-frequency components to serve as model fingerprints. To facilitate black-box auditing, we further introduce DistillTrace, which employs hypothesis testing of MLLM outputs to identify potential model infringement. We conduct extensive experiments on 154 model instances across 19 multimodal architectures. Notably, AttnPrint achieves strong derivative-model detection performance while remaining robust to five downstream modification techniques. DistillTrace also provides evidence of distillation relationships under three parameter-independent techniques.

Abstract

LLM Watermarking

Explore similar work

Fingerprinting Multimodal Large Language Models

To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model

Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection