Differentially Private Multimodal In-Context Learning

Ivoline C. Ngong; Zarreen Reza; Joseph P. Near

cs.AIMar 5, 2026

Differentially Private Multimodal In-Context Learning

Authors: Ivoline C. Ngong, Zarreen Reza, Joseph P. Near

Organizations: University of Vermont, Burlington, VT, USA · Independent Researcher

Abstract

Vision-language models are increasingly applied to sensitive domains such as medical imaging and personal photographs, yet existing differentially private methods for in-context learning are limited to few-shot, text-only settings because privacy cost scales with the number of tokens processed. We present Differentially Private Multimodal Task Vectors (DP-MTV), the first framework enabling many-shot multimodal in-context learning with formal $(\varepsilon, δ)$ -differential privacy by aggregating hundreds of demonstrations into compact task vectors in activation space. DP-MTV partitions private data into disjoint chunks, applies per-layer clipping to bound sensitivity, and adds calibrated noise to the aggregate, requiring only a single noise addition that enables unlimited inference queries. We evaluate on eight benchmarks across three VLM architectures, supporting deployment with or without auxiliary data. At $\varepsilon=1.0$ , DP-MTV achieves 50% on VizWiz compared to 55% non-private and 35% zero-shot, preserving most of the gain from in-context learning under meaningful privacy constraints.

Figures & tables

Figure 1: DP-MTV approach. Construction (offline): Partition data into disjoint chunks, extract and clip activations, compute the mean, add Gaussian noise (steps 1–2), then select heads via public data or a private mechanism (step 3). Inference (online): Replace activations at selected heads with private task vectors. Post-processing enables unlimited queries at no additional privacy cost.

Panel of Figure 2 — Figure 2: Model comparison at $\varepsilon=1.0$ . Brackets show the baseline gap (MTV $-$ Zero-shot); larger gaps predict better DP-MTV performance. (a) VizWiz has the largest gaps and strongest DP-MTV results. (b) On 2-way classification, DP-MTV often matches or exceeds MTV.

Panel of Figure 3 — Figure 3: Privacy-utility tradeoffs for Qwen-VL across privacy budgets. Dashed lines: non-private MTV; dotted lines: zero-shot. (a) On VQA, DP-MTV performs best on datasets where MTV provides the largest gains over zero-shot (e.g., VizWiz); performance varies by architecture (Figure 2 ). (b) On classification, DP-MTV matches or exceeds MTV at practical privacy budgets.

Figure 4: Effect of clipping threshold

C

on VizWiz accuracy (Qwen-VL,

\varepsilon=1.0

C=1.0

balances signal preservation with noise calibration. Performance degrades gradually at higher thresholds.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Train	Test	Task
VizWiz	General VQA	20,523	8,000	Open-ended
VQA-RAD	Medical VQA	1,793	451	Open-ended
PathVQA	Medical VQA	19,654	6,719	Open-ended
OK-VQA	Knowledge VQA	9,009	5,046	Open-ended
TextVQA	Text VQA	34,602	5,000	Open-ended
Flowers102	Classification	1,020	6,149	2-way 1-shot

Appendix

Table 1: Dataset Statistics. All datasets use standard train/test splits from original papers.

	Number of Chunks $m$
Method	50	100	500	1000
DP-MTV	49.8±8.6	49.1±10.3	50.6±3.9	48.7±5.9
DP-MTV (priv)	42.3±14.0	77.2±5.7	46.7±2.3	46.8±3.5

Appendix

Table 2: Sensitivity to hyperparameters on VizWiz (Qwen-VL,

\varepsilon=1.0

C=1.0

). Performance is stable across both number of shots

K

and number of chunks

m

Privacy Protection in VLMs

Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.

See all 37 papers

DP-MIND: Differentially private memory-guided Long context Inference
Oct 9, 2026Zhihao Liu, Yitian Cheng, Shuang Liu +3Privacy-Preserving Language Models Long-Context Language Model Inference
Zhejiang University · Ant Group · Institute of Science Tokyo +1
Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs
Jun 5, 2026Siyuan Xu, Yibing Liu, Peilin Chen +3Privacy-Preserving Language Models Multimodal Large Language Models
City University of Hong Kong · Hon Hai Research Institute · Lingnan University
Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
May 26, 2026Junkai Chen, Yuhao He, Junxiang You +3VLM Unlearning Multimodal Large Language Models
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, UCAS

Oct 9, 2026J/K move · Enter open · S save

DP-MIND: Differentially private memory-guided Long context Inference

Zhihao Liu, Yitian Cheng, Shuang Liu, Rui Wen+2

Zhejiang University · Ant Group · Institute of Science Tokyo+1

Privacy-Preserving Language Models Long-Context Language Model Inference Differential Privacy Memory-Augmented Language Models

Open paper AskPDF arXiv

LLM-as-a-Service has become a dominant paradigm for accessing large language models without requiring local deployment. However, the process raises severe privacy concerns when user queries may include sensitive or confidential contexts. Differential privacy (DP) provides a principled approach to protecting such inputs, but existing DP inference methods are primarily designed for short- or moderate-context settings and struggle with million-token-scale inputs due to limited context capacity and severe utility degradation under DP perturbation. In this paper, we propose DP-MIND, a memory-guided framework that scales differentially private inference to million-token contexts. DP-MIND enables private long-context inference by locally perturbing the input and processing it chunk by chunk with a compact memory that preserves task-relevant information across the sequence. To make this memory robust to DP noise, we optimize the memory construction policy directly on perturbed inputs using reinforcement learning, guided by both final answer correctness and memory consistency. We further recover perturbation-corrupted answer spans through a local denoising module without exposing the original context to the external model. We provide theoretical privacy analysis of DP-MIND and further conduct extensive empirical analysis across multiple long-context benchmarks with context lengths up to the million-token scale, demonstrating that DP-MIND consistently outperforms existing DP and long-context inference baselines in both scalability and utility.

Abstract

Privacy Protection in VLMs

Explore similar work

DP-MIND: Differentially private memory-guided Long context Inference

Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs

Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning