Steering Vision-Language Models for Visual-Contextual Privacy in Location Disclosure

Wenyue Chen; Yuhan Cheng; Ruixin Yang; Ethan Mendes; Alan Ritter; Jingwei Sun; Hai Li; Yiran Chen

cs.CVOct 19, 2026

Steering Vision-Language Models for Visual-Contextual Privacy in Location Disclosure

Authors: Wenyue Chen, Yuhan Cheng, Ruixin Yang, Ethan Mendes, Alan Ritter, Jingwei Sun, Hai Li, Yiran Chen

Organizations: Duke University · University of Washington · Georgia Institute of Technology · University of Florida

Abstract

Vision-language models (VLMs) can infer sensitive information such as geolocation from subtle visual cues, creating privacy risks when responses disclose more detail than the visual context warrants. We study visual-contextual privacy in geolocation disclosure, where a VLM should reveal location information only at the granularity appropriate to the visual context depicted in the image. We first show that this contextually appropriate granularity is recoverable from pre-generation hidden states, even when generated responses are misaligned, with disclosure levels exhibiting an ordered structure along a common hidden-state direction. Motivated by this, we propose an inference-time route-and-steer framework: an ordinal granularity router maps visual-contextual privacy representations to a disclosure route, and a disclosure-level-conditioned steering module applies route-specific hidden-state interventions during decoding. Across four VLM backbones, our method improves visual-context-aware disclosure alignment, better balances over- and under-disclosure, and incurs only modest changes on general capability benchmarks. It further transfers to another dataset with a distinct data distribution and privacy-risk taxonomy.

Figures & tables

Figure 1: Visual-context-aware geolocation disclosure control. The same location-inference request can require different disclosure granularities depending on the visual context depicted in the image, such as a private residential scene versus a public landmark. Without steering, the VLM may over-disclose or under-disclose location information; with steering, the response is shifted toward the contextually appropriate disclosure granularity.

	VLM response		Linear Probe
Model	F1	Accuracy	F1	Accuracy
Qwen2.5-VL-3B-Instruct	39.2	39.9	59.0	58.8
Qwen3.5-4B	42.4	47.3	57.2	57.6
Llama-3.2-11B-Vision-Instruct	36.4	38.7	57.5	58.4
InstructBLIP-Vicuna-7B	29.5	28.0	54.2	54.3

Table 1: F1 and accuracy (%) against gold disclosure targets, using disclosure levels extracted from VLM responses or predicted by linear probes from pre-generation hidden states.

Figure 2: Framework of context-aware activation steering. Condition states grouped by gold disclosure targets $y_{i}^{*}$ define an ordinal router, while answer states grouped by judged disclosure labels $z_{i}$ define behavior vectors $\mathbf{v}_{A}$ and $\mathbf{v}_{C}$ . At inference time, an unsteered prompt pass predicts route $\hat{y}$ using a scalar score and ordered thresholds. Routes $A$ and $C$ apply fixed-strength interventions to reduce disclosure or increase detail, respectively; route $B$ applies no intervention.

Figure 3: Class-conditional disclosure granularity coordinate distributions on VLM-GeoPrivacy across four backbones.

		Granularity Alignment			Disclosure Errors
Backbone	Method	F1 $\uparrow$	Accuracy $\uparrow$	G-MAE $\downarrow$	Over $\downarrow$	Under $\downarrow$	AV $\downarrow$
Qwen2.5-VL-3B-Instruct	Vanilla	39.2	39.9	68.3	23.0	37.0	54.1
	Privacy Policy Prompt	36.9	39.5	72.8	18.1	42.4	42.9
	Static CAA	41.6	43.0	65.3	19.0	38.0	45.4
	LoRA-SFT	51.1	51.4	55.6	19.3	29.2	43.9
	Ours	51.2	54.1	62.0	16.9	28.9	33.0

Table 2: End-to-end visual-context-aware disclosure alignment performance. An LLM judge maps each VLM response to a disclosure-granularity label, which is compared with the human-annotated contextual target to compute the metrics (Appendix A.5 ). All metrics are computed over valid responses (labels A/B/C), excluding D. Best and second-best results within each backbone are shown in bold and underlined , respectively.


		Routing	End-to-End Performance
Backbone	Method	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	G-MAE $\downarrow$	Over $\downarrow$	Under $\downarrow$	AV $\downarrow$
Qwen2.5-VL-3B-Instruct	Ours	56.4	51.2	54.1	62.0	16.9	28.9	33.0
	Prompt-based Routing	40.7	42.4	45.3	68.3	14.4	40.3	33.7
	Oracle Routing	100.0	70.6	72.7	28.5	12.8	14.5	26.8
Prompt-based Steering	—	44.2	45.9	66.1	17.4	36.8	39.8

Table 3: Component analysis of disclosure granularity routing and disclosure-level-conditioned behavior steering. Prompt-based Routing and Oracle Routing replace the router while keeping downstream activation steering fixed (Section 5.3.1 ). Prompt-based Steering retains the routing decisions of Ours and replaces activation steering with disclosure-level-specific text instructions (Section 5.3.2 ).

	Vanilla		Ours
Model	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$
Qwen2.5-VL-3B-Instruct	22.5	20.8	35.5	44.0
Qwen3.5-4B	13.7	13.9	32.7	37.0
Llama-3.2-11B-Vision-Instruct	21.0	19.8	29.8	29.7
InstructBLIP-Vicuna-7B	24.6	34.7	35.0	38.7

Table 4: Disclosure-granularity alignment on DoxBench (%). Ours improves F1 and accuracy across all four backbones.

Figure 4: Alignment tax across methods and benchmarks, computed as Vanilla minus intervention scores in percentage points; positive values indicate degradation. Error bars show 95% paired bootstrap confidence intervals, and dashed lines mark zero tax.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Partition	Total	A	B	C
Construction	717	290	144	283
Validation	238	97	47	94
Test	243	98	49	96
Total	1,198	485	240	473

Appendix

Table 5: Class distribution of the fixed construction, validation, and test partitions.

Backbone	$\ell_{r}$	$\ell_{s}$	$\tau_{AB}$	$\tau_{BC}$	$\alpha_{A}$	$\alpha_{C}$
Qwen2.5-VL-3B	33	28	$-17.9672$	$-18.7784$	1.50	3.00
Qwen3.5-4B	17	16	$0.3707$	$0.0562$	2.50	3.50
Llama-3.2-11B-Vision	39	24	$2.5480$	$2.5401$	3.25	2.75
InstructBLIP-Vicuna-7B	16	16	$0.7672$	$0.5881$	2.75	3.75

Appendix

Table 6: Frozen component configurations on VLM-GeoPrivacy. Routing and steering layers use zero-based indices. Cut points apply to the centered projected routing score and are rounded to four decimal places. The B route uses

\alpha_{B}=0

Figure 5: Validation F1 in local

4\times 4

windows for all four backbones.

Figure 6: DoxBench construction-set distributions of the projected routing coordinate. Each contextual target class has a unit-area density; coordinate scales are backbone-specific.

Backbone	Method	G-MAE $\downarrow$	Over $\downarrow$	Under $\downarrow$	AV $\downarrow$	Coverage $\uparrow$
Qwen2.5-VL-3B	Vanilla	85.1	67.3	11.9	96.9	100.0
	Ours	66.0	45.0	11.0	46.2	99.0
Qwen3.5-4B	Vanilla	121.8	83.2	3.0	100.0	100.0
	Ours	90.0	55.0	8.0	63.1	99.0
Llama-3.2-11B-Vision	Vanilla	96.0	73.3	6.9	93.8	100.0
	Ours	83.2	65.3	5.0	75.4	100.0

Appendix

Table 7: Additional disclosure metrics on DoxBench. AV is computed over gold-A examples with valid responses (labels A/B/C), excluding D from the denominator. Lower is better for the four error metrics, with the best value within each backbone in bold, including ties. Coverage is reported without bolding.

	Qwen2.5-VL-3B			Qwen3.5-4B
Method	VQAv2 $\uparrow$	TextVQA $\uparrow$	MMMU $\uparrow$	VQAv2 $\uparrow$	TextVQA $\uparrow$	MMMU $\uparrow$
Vanilla	83.25	76.55	50.48	81.35	81.60	45.24
Privacy Policy Prompt	82.85	76.75	49.05	80.45	79.55	50.00
Static CAA	83.25	76.40	49.52	80.65	82.20	44.76
LoRA-SFT	83.80	77.00	49.05	79.10	80.20	44.76
Ours	81.95	77.75	49.52	80.40	80.80	43.81

Appendix

Table 8: General-capability scores (%) on fixed validation subsets of VQAv2, TextVQA, and MMMU, containing 200, 200, and 210 questions, respectively. VQAv2 and TextVQA use answer-consensus scoring; MMMU uses task accuracy with the frozen deterministic fallback for unparseable multiple-choice responses. Higher is better for all columns.

		Clean			Iterative			Malicious
Backbone	Method	Acc $\uparrow$	Over $\downarrow$	Under $\downarrow$	Acc $\uparrow$	Over $\downarrow$	Under $\downarrow$	Acc $\uparrow$	Over $\downarrow$	Under $\downarrow$
Qwen2.5-VL-3B-Instruct	Vanilla	39.9	23.0	37.0	36.2	30.9	32.9	36.2	47.3	16.5
	Privacy Policy Prompt	39.5	18.1	42.4	36.2	23.0	40.7	36.2	43.2	20.6
	Static CAA	43.0	19.0	38.0	39.1	28.8	32.1	32.1	44.4	23.5
	LoRA-SFT	51.4	19.3	29.2	46.9	32.9	20.2	36.2	46.5	17.3
	Ours (original)	54.1	16.9	28.9	39.9	56.4	3.7	39.7	55.0	5.4

Appendix

Table 9: Disclosure control under clean and adversarial prompting(%). Acc measures disclosure-target matching; Over and Under measure overly specific and insufficiently specific disclosure, respectively. All three metrics exclude D assignments. Iterative scores use the final attempted response, with at most four requests. Bold and underlined values denote the best and second-best distinct scores among six methods within each backbone and column; displayed ties share a rank. Ours (original) is the frozen main-text method. The exploratory mixed-prompt variant refits only the router on construction-set representations from all three prompt families.

Figure 7: Breakdown of Vanilla errors on VLM-GeoPrivacy. Each bar is normalized over one backbone’s Vanilla-incorrect test examples. Both denotes examples correctly classified by the fixed-layer Linear Probe and corrected by Ours; Probe only , Ours only , and Neither denote the remaining outcomes. Correctness refers to disclosure-target matching, with D counted as incorrect.

Privacy Protection in VLMs

Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.

See all 37 papers

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models
Sep 18, 2026Yining Wang, Xi Li, Mi Zhang +4Privacy Protection in VLMs Adversarial Attacks on VLMs
Fudan University · East China University of Science and Technology · Shanghai University of Electric Power
GeoCloak: An Agentic Framework for Semantics-Preserving Geolocation Privacy via Peripheral Clue Manipulation
Oct 14, 2026Yixin Wu, Mingjie Li, Yun Shen +2Privacy Protection in VLMs Adversarial Attacks on VLMs
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · CISPA Helmholtz Center for Information Security +1
AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO
Jun 3, 2025Xiaonan Wang, Bo Shao, Hansaem KimVLM Evaluation Privacy Protection in VLMs
Yonsei University, South Korea · CISPA Helmholtz Center for Information Security, Germany

Sep 18, 2026J/K move · Enter open · S save

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models

Yining Wang, Xi Li, Mi Zhang, Xiaohan Zhang+3

Fudan University · East China University of Science and Technology · Shanghai University of Electric Power

Privacy Protection in VLMs Adversarial Attacks on VLMs

Open paper AskPDF arXiv

Multimodal large reasoning models (MLRMs) have demonstrated remarkable capabilities in complex visual understanding. However, this very power introduces a critical yet underexplored privacy threat: adversaries can exploit MLRMs to precisely infer users' geographic locations from casually shared photographs, by performing structured reasoning over subtle visual cues such as architectural styles, vegetation, and lighting conditions. In this work, we present a systematic study of MLRM-driven geolocation privacy leakage. We first reveal that refusal-based safeguards are critically insufficient, as carefully crafted jailbreak prompts can raise model response rates to 100%. We further identify that existing defenses, which inject imperceptible perturbations into shared images, suffer from structural limitations intrinsic to their pixel-space optimization, resulting in degraded black-box transferability and pronounced visual artifacts. Motivated by these findings, we propose a diffusion-based framework that provides targeted, proactive defense against geolocation privacy leakage. By injecting perturbations into the latent space of a diffusion model during reverse sampling, our method operates directly on high-level semantic representations, thereby resolving the effectiveness-utility bottlenecks by construction. We further ground our optimization with GeoCLIP, a model explicitly aligned with GPS coordinates, as a surrogate to pinpoint and disrupt the geographic signals that MLRMs exploit for location inference. This targeted semantic disruption yields significantly stronger black-box transferability while preserving perceptual image quality, offering a seamless integration on social media platforms. Code is available at https://github.com/RachelWolowitz/Hiding_in_plain_sight.

Abstract

Privacy Protection in VLMs

Explore similar work

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models

GeoCloak: An Agentic Framework for Semantics-Preserving Geolocation Privacy via Peripheral Clue Manipulation

AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO