cs.CVFeb 21, 2026

Echoes of ownership: Adversarial-guided dual injection for copyright protection in MLLMs

Authors: Chengwei Xia, Fan Ma, Ruijie Quan, Yunqiu Xu, Kun Zhan, Yi Yang

Organizations: School of Information Science and Engineering, Lanzhou University · College of Computer Science and Technology, Zhejiang University · College of Computing and Data Science, Nanyang Technological University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

With the rapid deployment of multimodal large language models (MLLMs), disputes regarding model ownership have become increasingly frequent, raising significant concerns about intellectual property protection. In this paper, we propose a framework for generating copyright triggers for MLLMs, enabling model publishers to embed verifiable ownership information into the model. The goal is to construct trigger images that elicit ownership-related textual responses exclusively in fine-tuned derivatives, while remaining inert in other non-derivative models. Our method constructs a tracking trigger image by treating the image as a learnable tensor, performing adversarial optimization with dual-injection of ownership-relevant semantic information. The first injection is achieved by enforcing textual consistency between the output of an auxiliary MLLM and a predefined ownership-relevant target text; the consistency loss is backpropagated to inject this ownership-related information into the image. The second injection is performed at the semantic-level by minimizing the distance between the CLIP features of the image and those of the target text. Furthermore, we introduce an additional adversarial training stage involving the auxiliary model. It is specifically trained to resist generating ownership-relevant target text, thereby enhancing robustness in heavily fine-tuned derivative models. Extensive experiments demonstrate the effectiveness of our dual-injection approach in tracking model lineage under various fine-tuning and domain-shift scenarios. Code is at https://github.com/kunzhan/AGDI

Figures & tables

Figure 1: The overview of copyright tracking for MLLMs. (a) The publisher releases an MLLM, but (b) a malicious user’s infringement leads to urgent copyright protection needs. (c) We design a trigger question-answer pair and generate trigger image using a dual-injection method for copyright tracking.
Figure 2: The pipeline of our proposed AGDI for copyright tracking. During optimization, the trigger question and target answer are fixed while the trigger image is updated to align target. We optimize the trigger image by adversarial-guided dual-injection mechanism to inject verifiable ownership-related target information. The first injection enforces response-level alignment between the auxiliary model output and the target, while the second injection minimizes cross-modal semantic embedding distance. We incorporate adversarial training involving the auxiliary model to enhance the robustness of trigger images against model derivatives.
QuestionDetecting copyright.Are you all right?Please stop.Exercise now!Describe the image.
AnswerICLR Conference.I don’t like it.I’m playing games.Time flies so fast.I won’t tell.
Table 1: Trigger question-answer pairs used in the experiments.
 LoRA Fine-tuningFull Fine-tuning
MethodV7WST-VQATextVQAPaintingFMathVV7WST-VQATextVQAPaintingFMathV
Ordinary36%46%22%48%41%34%43%15%48%26%
RNA36%39%22%40%37%32%38%15%40%21%
PLA48%68%33%76%60%43%60%28%75%38%
AGDI53%77%41%81%68%46%65%33%80%45%
Table 2: The comparison of ADGI with baselines on the Qwen2-VL downstream fine-tuned models. The best results are highlighted in bold.
 LoRA Fine-tuning
MethodV7WST-VQATextVQAPaintingFMathV
Ordinary31%28%11%21%11%
IF28%22%30%8%24%
RNA28%31%14%20%9%
PLA51%43%21%55%18%
AGDI64%56%36%79%30%
Table 3: The comparison of our method with baselines on the LLaVA-1.5 downstream fine-tuned models. We report ASR on the models, which measures the copyright tracking performance of triggers on the models. The best results are highlighted in bold.
Figure 3: Comparison of trigger images’ response from non-derivative models and derivative models, and clean images’ response from derivative models. The triggers and cleans use same trigger questions on the models.
MLLMMiniGPT-4Qwen2-VLLLaVA-1.6Llama3-V
RNA0%0%0%0%
PLA0%0%0%0%
AGDI0%0%0%0%
Table 4: Copyright tracking results on the non-derivative MLLMs. We report ASR on the non-derivative MLLMs via triggers constructed from LLaVA-1.5.
  MagnitudeWanda
ModelMethod10%20%30%10%20%30%
ST-VQARNA28%26%20%31%27%26%
PLA38%35%28%38%32%31%
AGDI54%50%43%52%49%48%
PaintingFRNA20%17%13%13%10%7%
PLA46%42%26%30%24%14%
Table 5: The robustness of triggers under model pruning. We report the ASR on three fine-tuned variants from LLaVA-1.5 under two typical prune methods. The best results for each fine-tuned model are highlighted in bold.
  LoRA Fine-tuningFull Fine-tuning
ModelMethodST-VQA*TextVQA*PaintingF*MathV*ST-VQA*TextVQA*PaintingF*MathV*
LinearRNA40%37%43%41%39%36%46%46%
PLA59%52%63%59%54%49%63%62%
AGDI63%57%68%63%58%56%67%68%
TIESRNA35%31%38%34%32%28%39%31%
PLA46%40%54%48%43%40%55%44%
Table 6: The robustness of triggers under model merging. We report the ASR on four model merging variants from LoRA and full fine-tuned models of Qwen2-VL under two model merging strategies, where “*” denotes the fine-tuned model merged with V7W fine-tuned variants.
Figure 4: ASR comparison between ADGI and baselines under 8-bit quantization: (a) five fine-tuned variants of LLaVA-1.5; (b) five fine-tuned variants of Qwen2-VL.
Figure 5: Hyperparameter analysis results in a single trigger question-answer pair: “Q: Detecting copyright. A: ICLR Conference”. on the LLaVA-1.5 fine-tuned models. (a) The impact of model learning rate on tracking performance. (b) The impact of optimization steps on tracking performance. (c) The impact of perturbation budget on tracking performance.
 LLaVA-1.5Qwen2-VL
Ablation settingV7WST-VQATextVQAPaintingFMathVV7WST-VQATextVQAPaintingFMathV
w/o res injection0%1%1%1%4%0%0%0%1%1%
w/o sem injection51%43%21%55%18%48%68%33%76%60%
w/o LLM update32%39%20%19%13%32%20%40%39%34%
w/o encoder update60%55%29%70%29%51%74%36%75%64%
AGDI64%56%36%79%30%53%77%41%81%68%
Table 7: Ablation study of our proposed component on the five LoRA fine-tuned models for two base models.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterLoRA fine-tuning settingFull fine-tuning setting
OptimizerAdamWAdamW
Learning rate2e-41e-5
Batch size84
LoRA rank16/
LoRA alpha32/
Training epochs33
Appendix
Table 8: Fine-tuning setting in the experiments.
DatesetsV7W (ACC)ST-VQA (ACC)TextVQA (ACC)PaintingF (BLEU / ROUGE)MathV (ACC)
Before fine-tuning0.6%21.9%15.6%4.9% / 8.8%27.1%
After fine-tuning37.9%65.2%51.3%13.3% / 15.9%57.5%
Appendix
Table 9: Utility performance of LoRA fine-tuned LLaVA-1.5 models.
DatesetsV7W (ACC)ST-VQA (ACC)TextVQA (ACC)PaintingF (BLEU / ROUGE)MathV (ACC)
Before fine-tuning0.8%29.8%23.9%7.6% / 11.9%7.0%
After fine-tuning37.9%47.7%78.7%13.4% / 16.2%66.6%
Appendix
Table 10: Utility performance of LoRA fine-tuned Qwen2-VL models.
DatesetsV7W (ACC)ST-VQA (ACC)TextVQA (ACC)PaintingF (BLEU / ROUGE)MathV (ACC)
Before fine-tuning0.8%29.8%23.9%7.6% / 11.9%7.0%
After fine-tuning36.5%88.8%78.6%13.5% / 15.9%86.0%
Appendix
Table 11: Utility performance of full fine-tuned Qwen2-VL models.
Figure 6: Examples of Visual7W dataset.
Figure 7: Examples of ST-VQA dataset.
Figure 8: Examples of TextVQA dataset.
Figure 9: Examples of PaintingForm dataset.
Figure 10: Examples of MathV360k dataset.
Figure 11: Hyperparameter analysis results of inference settings (Top-p and temperature). (a)(b) ASR on ST-VQA and PaintingF for LLaVA-1.5; (c)(d) ASR for Qwen2-VL.
 Qwen2-VLInternVL3.5
MethodPathvqaDriveVQAFinVisRSVQASciVersePathvqaDriveVQAFinVisRSVQASciVerse
Ordinary41%52%50%55%69%30%27%26%36%46%
RNA39%43%43%46%54%29%19%17%30%35%
PLA55%76%72%76%86%40%46%52%64%73%
AGDI58%80%77%79%88%45%50%57%70%76%
Appendix
Table 12: Copyright tracking performance on more multimodal downstream models.
MLLMMethodV7WST-VQATextVQAPaintingFMathV
InternVL3.5-8BOrdinary14%21%15%9%16%
RNA16%22%15%11%20%
PLA44%55%46%31%49%
 AGDI58%63%62%45%58%
Appendix
Table 13: Copyright tracking performance on InternVL3.5 8B fine-tuned models.
 LoRA Fine-tuningFull Fine-tuning
MethodV7WST-VQATextVQAPaintingFMathVV7WST-VQATextVQAPaintingFMathV
Ordinary25%38%32%12%23%35%38%30%21%30%
RNA23%40%30%10%20%33%40%31%18%26%
PLA39%53%45%23%34%46%53%42%38%45%
AGDI42%55%47%30%41%49%56%44%49%51%
Appendix
Table 14: Copyright tracking performance on InternVL3.5 2B fine-tuned models.
 V7WST-VQATextVQAPaintingFMathV
Original64%56%36%79%30%
Resizing42%40%26%53%19%
Gaussian60%53%31%73%27%
JPEG40%39%24%42%16%
Appendix
Table 15: Copyright tracking under input transformations.
Figure 12: Ablation study for λ\lambda and fine-tuning epochs: (a) The impact of loss function parameter λ\lambda on tracking performance; (b) The impact of downstream fine-tuning epochs on tracking performance.
 QA1QA2QA3QA4QA5
V7W63%75%70%68%37%
ST-VQA58%62%44%61%48%
PaintingF81%86%69%93%62%
Appendix
Table 16: ASR on 5 different QA pairs.
 V7WST-VQATextVQAPaintingFMathV
Original64%56%36%79%30%
Sys prompt161%53%32%73%26%
Sys prompt263%54%32%74%25%
Appendix
Table 17: System prompt experiments on LLaVA-1.5 LoRA variants.
Similarity driftPair1Pair2Pair3Pair4Pair5
Base →\rightarrow V7W4.4%9.3%1.0%2.1%6.1%
Base →\rightarrow MathV6.9%7.1%3.3%0.5%1.2%
Appendix
Table 18: Average similarity drift(%) on the fine-tuned models.
More in this topic

LLM Watermarking

Large language model watermarking embeds or detects signals in text and models to trace provenance, ownership, and unauthorized use.

See all 111 papers

Explore similar work

CardsList
Sep 17, 2026

Fingerprinting Multimodal Large Language Models

While multimodal large language models (MLLMs) enable a wide range of image-text reasoning tasks, recent incidents indicate that they are vulnerable to illicit deployment and unauthorized distillation. Existing solutions for model provenance are typically confounded by shared language backbones in MLLMs and struggle to detect violations of distillation. To bridge this gap and safeguard model ownership, we present the first study on multimodal model fingerprinting. Inspired by recent findings that self-attention acts as a low-pass filter and that its low-frequency components are informative, we develop AttnPrint for white-box provenance. Specifically, we extract cross-modal attention distributions and isolate their low-frequency components to serve as model fingerprints. To facilitate black-box auditing, we further introduce DistillTrace, which employs hypothesis testing of MLLM outputs to identify potential model infringement. We conduct extensive experiments on 154 model instances across 19 multimodal architectures. Notably, AttnPrint achieves strong derivative-model detection performance while remaining robust to five downstream modification techniques. DistillTrace also provides evidence of distillation relationships under three parameter-independent techniques.

Chao Huang, Meng Tong, Kejiang Chen

University of Science and Technology of China Anhui Province Key Laboratory of Digital Security Hefei, Anhui, China

LLM AuditingLanguage Model FingerprintingMultimodal Large Language Models
AskPDF
May 14, 2026

To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model

The rapid advancement of Large Vision-Language Models (LVLMs) is increasingly accompanied by unauthorized scraping and training on multimodal web data, posing severe copyright and privacy risks to data owners. Existing countermeasures, such as machine unlearning and watermarks, are inherent post-hoc approaches that act only after intellectual property infringement has already occurred. In this work, we propose MMGuard to empower data owners to proactively protect their multimodal data against unauthorized LVLM fine-tuning. MMGuard generates unlearnable examples by injecting human-imperceptible perturbations that actively exploit the learning dynamics of LVLMs. By minimizing the training loss, the perturbation creates an optimization shortcut, causing the model to overfit to the noise and thereby degrading downstream performance when the perturbation is absent during inference. To further strengthen this defense, MMGuard introduces a cross-modal binding disruption, strategically shifting LVLM attention to enforce a spurious correlation between the noise and the training target with theoretical guarantees. Enhanced by an ensemble learning strategy for cross-model transferability, MMGuard is evaluated against nine open-source LVLMs across six datasets. Our comprehensive results demonstrate effective, stealthy, and robust protection under white-box, gray-box, and black-box threat models, establishing a mechanistic advantage in proactively defending against aggressive fine-tuning exploitation.

Chengshuai Zhao, Zhen Tan, Dawei Li +2

School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA · Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA.

VLM RobustnessMultimodal RobustnessLarge Vision-Language ModelsAdversarial ExamplesUnlearnable Examples
AskPDF
Apr 10, 2026

Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection

Multi-modal large language models (MLLMs) have emerged as powerful tools for analyzing Internet-scale image data, offering significant benefits but also raising critical safety and societal concerns. In particular, open-weight MLLMs may be misused to extract sensitive information from personal images at scale, such as identities, locations, or other private details. In this work, we propose ImageProtector, a user-side method that proactively protects images before sharing by embedding a carefully crafted, nearly imperceptible perturbation that acts as a visual prompt injection attack on MLLMs. As a result, when an adversary analyzes a protected image with an MLLM, the MLLM is consistently induced to generate a refusal response such as "I'm sorry, I can't help with that request." We empirically demonstrate the effectiveness of ImageProtector across six MLLMs and four datasets. Additionally, we evaluate three potential countermeasures, Gaussian noise, DiffPure, and adversarial training, and show that while they partially mitigate the impact of ImageProtector, they simultaneously degrade model accuracy and/or efficiency. Our study focuses on the practically important setting of open-weight MLLMs and large-scale automated image analysis, and highlights both the promise and the limitations of perturbation-based privacy protection.

Zedian Shao, Hongbin Liu, Yuepeng Hu +1

Georgia Institute of Technology · Duke University

Privacy Protection in VLMsAdversarial Attacks on VLMsVLM RobustnessPrompt Injection Attacks
AskPDF