GeoCloak: An Agentic Framework for Semantics-Preserving Geolocation Privacy via Peripheral Clue Manipulation

Yixin Wu; Mingjie Li; Yun Shen; Michael Backes; Yang Zhang

cs.CVOct 14, 2026

GeoCloak: An Agentic Framework for Semantics-Preserving Geolocation Privacy via Peripheral Clue Manipulation

Authors: Yixin Wu, Mingjie Li, Yun Shen, Michael Backes, Yang Zhang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · CISPA Helmholtz Center for Information Security · HPE

Abstract

The rapid advancement of large vision language models (LVLMs) has made geolocation inference from images highly accurate and accessible. This poses serious privacy risks: a passive adversary with black-box LVLM access can infer a user's location from a single photo with no metadata. Existing protection methods rely almost exclusively on adversarial perturbation, which introduces perceptible visual artifacts and utility degradation. We identify the visual-geolocation asymmetry: geolocation-decisive clues, such as street signs, tend to be visually peripheral but dominate location inference. Exploiting this asymmetry, we propose GeoCloak, an agentic framework for semantics-preserving geolocation privacy protection. It acts as a clue-aware planner that analyzes each image to identify geolocation-relevant clues, dispatches targeted semantic edits to directly suppress peripheral clues, and autonomously configures adversarial perturbation to further suppress residual geolocation features, all under iterative utility-aware feedback. We evaluate GeoCloak on four diverse datasets spanning 108 countries and 576 cities against five LVLMs and three inference methods. GeoCloak reduces overall city-level accuracy from 41.2% to 9.9%, approximately half of the best-performing baseline, while achieving substantially superior utility preservation both qualitatively and quantitatively. We hope this work takes a meaningful step toward practical, semantics-preserving geolocation privacy protection, giving users greater control over the location information inadvertently embedded in their shared images.

Figures & tables

Figure 1: An illustration of the visual-geolocation asymmetry. The most decisive geolocation clue (ranking score 1.0) is a small, visually peripheral sign in Khmer script.

Figure 2: Overview of GeoCloak . A utility check ensures visual quality, and ClueAgent re-evaluates the image as a protection check, terminating when the correct location can no longer be inferred. See multi-turn examples in Appendix A .

	GeoBench	Im2GPS3k	Dataset-GSS	Dataset-PCW
Exempt Rate	37.5%	53.4%	21.0%	55.3%

Table 1: Proportion of images labeled as Exempt (containing intentional landmarks) across the four evaluation datasets.

		Level-Wise Acc.			Distance-Based Acc.
Dataset	Type	City	State	Country	@1km	@25km	@100km
Overall	Exempt	73.0%	84.0%	93.9%	30.8%	82.3%	86.0%
	Non-Exempt	48.3%	58.9%	87.9%	16.1%	55.2%	61.6%

Table 2: Geolocation prediction performance measured by level-wise accuracy (%) and distance accuracy (%) on exempt images and non-exempt images. We leverage zero-shot inference and Gemini-2.5-Pro.

		Zero-Shot					GeoVista	GeoMiner
Dataset	Defense	Gemini-2.5-Pro	Gemini-3.1-Pro	GPT-5.4	o3	Qwen3.5-Plus	GeoVista	Gemini-2.5-Pro	Gemini-3.1-Pro	GPT-5.4	o3	Qwen3.5-Plus	Overall
GeoBench	Initial	74.1%	81.9%	63.1%	71.6%	54.7%	63.7%	70.6%	77.2%	58.8%	62.5%	53.1%	66.5%
	M-Attack	54.4%	62.8%	37.2%	47.2%	33.1%	42.8%	50.9%	60.6%	38.8%	36.9%	28.4%	44.8%
	GeoShield	53.1%	64.1%	36.2%	40.3%	29.4%	35.9%	50.0%	60.3%	35.0%	30.3%	28.4%	42.1%
	GeoCloak	17.5%	22.2%	18.1%	25.3%	11.2%	21.6%	19.1%	22.5%	12.2%	16.9%	7.5%	17.6%
Im2GPS3k	Initial	30.0%	30.9%	24.0%	29.6%	16.3%	18.9%	30.0%	26.6%	21.5%	19.7%	17.2%	24.1%

Table 3: Geolocation inference performance measured by city-level accuracy (%) under no defense (Initial) and three defenses (GeoShield [ 31 ] , M-Attack [ 29 ] , and GeoCloak ). Bold indicates the lowest (best) value within each inference method, model, and dataset group.

\downarrow

lower is better. GeoCloak uses Gemini-2.5-Pro and Gemini-2.5-Flash-Image. Results for additional metrics are provided in Table 13 .

Attacker	Mechanism	Initial	GeoCloak
GeoCLIP [ 4 ]	Embedding-based retrieval	12.1	5.0
GeoSeer [ 10 ]	Commercial black-box tool	30.7	5.7

Table 4: City-level accuracy of extra geolocation attackers.

Figure 3: Qualitative comparison of defense methods. Numbers indicate how many of the 11 attackers, each corresponding to a unique (model, inference method) pair, successfully geolocate the image. We present cases where all defenses largely succeed. Best viewed ( Appendix G ) with zoom to observe severe visual artifacts, such as floating text, superimposed objects, spatial distortions, and unnatural cropping.

Defense	Naturalness $\uparrow$	Semantic Preservation $\uparrow$	Overall Quality $\uparrow$
GeoShield	2.23	3.68	2.53
M-Attack	2.31	3.77	2.63
GeoCloak	4.57	4.45	4.43

Table 5: Human perception ratings on a five-point Likert scale. Higher is better (

\uparrow

Dataset	Defense	CLIP Sim $\uparrow$	BERT Score $\uparrow$	BLEU $\uparrow$	ROUGE-L $\uparrow$
Overall	M-Attack	0.7719	0.9124	0.0968	0.3839
	GeoShield	0.4209	0.9047	0.0766	0.3480
	GeoCloak	0.9197	0.9158	0.1136	0.4054

Table 6: Utility comparison of defense methods across four datasets. Higher is better (

\uparrow

higher is better). Bold indicates the best value per dataset.

Figure 4: Visual quality comparison with zoomed-in regions on a Dataset-PCW sample, where adversarial perturbation is the key skill driving GeoCloak ’s successful protection. Zoomed-in crops (top) reveal that GeoCloak introduces imperceptible perturbations compared to GeoShield and M-Attack. Comparison on other datasets are in Appendix G .

Panel of Figure 5 — Figure 5: Skill usage analysis of GeoCloak . (a) Distribution of all skill invocations and the utility preservation rate for each skill, where Utility OK indicates that the CLIP similarity remains above a pre-defined threshold and the main object is unmodified. (b) Breakdown of skills used in the first successful turn recorded at each exact protection level (city, state, country); runs that directly reach a stronger level are not back-counted at lower levels.

Design Choice	Overall $\downarrow$
Initial	24.1%
GeoCloak w/o Semantic Editing	15.4%
GeoCloak w/o Perturbation	5.1%
Closed-source alternative †	6.3%
Open-source alternative ‡	7.4%
GeoCloak (Full)	4.0%

Table 7: Ablation study on GeoCloak design choices. Overall city-level attack success rate (%) is averaged across 11 attacker configurations on the Im2GPS3k.

Panel of Figure 6 — Figure 6: Cost and elapsed time distribution of GeoCloak . API cost is computed from the total token usage of GeoCloak based on the official Gemini pricing [ 15 ] .

Attacker	Setting	GeoBench	Im2GPS3k	Dataset-GSS	Dataset-PCW	Overall
Naive	Initial	69.1	26.2	34.9	36.1	43.4
	GeoCloak	18.9	4.8	9.3	8.2	11.1
	GeoCloak (JPEG)	20.0	4.9	9.6	8.2	11.5
Adaptive	Initial	66.6	26.5	32.8	33.0	41.6
Adaptive	GeoCloak	24.0	8.0	15.2	11.9	15.9

Table 8: City-level accuracy (%) over five LVLMs under naive and adaptive attackers with zero-shot prompting, before and after GeoCloak defense, and (for the naive attacker) after additional JPEG compression.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: A multi-turn GeoCloak protection example. Starting from Rabat, Morocco, EditAgent first replaces the Moroccan license plate with a French one, shifting ClueAgent ’s prediction to Casablanca, Morocco. A subsequent attempt to change the shop sign fails the utility check twice and is rolled back. EditAgent then switches strategy to replace the Moroccan sofa with a European one, successfully redirecting the predicted location to Oran, Algeria, with minimal visual change across all turns.

Strategy	Application Scenario	Original	Protected	How it protects
Replace Text	Signs, street names, or any visible text reveal the location.			Replacing Romanian sign text on the red shop signs with Polish words shifts the language clue and misleads the predicted location from Chişinău, Moldova to Warsaw, Poland.
Replace Object	Location-specific objects are present (e.g., post boxes and traffic signs).			Replacing the American highway sign with a Canadian highway sign shifts the predicted location from Southfield, Michigan, USA, to Windsor, Ontario, Canada.
Erase Object	An object, sign, text, or logo is too specific to replace but small enough to remove cleanly.			Removing all text from the orange billboard shifts the predicted location from San José, Costa Rica, to San Salvador, El Salvador.
Add Decoy	Dominant clues are too large or structural to edit or perturb directly.			Adding a small Saguaro cactus growing among the rocks shifts the predicted location from Khomas, Namibia to Tucson, Arizona, US.
Style Transfer	Apply only to small or background elements.			Changing the architectural style of the greenish building on the left from Russian neoclassical to German neoclassical shifts the predicted location from Moscow, Russia to Warsaw, Poland.

Appendix

Table 9: Overview of the five Semantic Editing strategies in EditAgent . The EditAgent generates plans for modifying the image based on the clues obtained from ClueAgent . In all cases, modifications are subtle and semantics-preserving, yet sufficient to redirect ClueAgent ’s predicted location.

Dataset	#Samples	#Countries	#Cities
IM2GPS3K	500	74	297
Dataset-GSS	500	61	161
Dataset-PCW	267	6	76
GeoBench	512	64	117
Overall	1,779	108	576

Appendix

Table 10: Geographic coverage of the four benchmark datasets.

#Clues	Initial	GeoCloak	GeoShield	M-Attack
2	3.83/11	0.00/11	1.67/11	1.33/11
3	5.89/11	1.11/11	2.22/11	3.33/11
4	6.39/11	1.35/11	3.36/11	4.00/11
5	6.76/11	1.70/11	3.02/11	3.99/11
6	6.58/11	1.74/11	3.17/11	3.82/11
7	5.12/11	1.29/11	2.35/11	2.29/11

Appendix

Table 11: Mean number of successful attackers per image, stratified by the number of clues extracted by GeoMiner.

Geo-identifiability	Initial	GeoCloak	GeoShield	M-Attack
Low (1–4/11)	2.18/11	0.35/11	0.39/11	0.57/11
Medium (5–7/11)	5.98/11	0.86/11	1.73/11	2.39/11
High (8–11/11)	9.84/11	2.75/11	5.69/11	6.91/11
Overall	6.55/11	1.57/11	3.13/11	3.90/11

Appendix

Table 12: Mean number of successful attackers per image, stratified by initial geo-identifiability. Lower is better for all defense columns.

			Level-Wise Acc.			Distance Acc.
Model	Method	Defense	City $\downarrow$	State $\downarrow$	Country $\downarrow$	@1km $\downarrow$	@25km $\downarrow$	@100km $\downarrow$
Gemini-2.5-Pro	Zero-Shot	Initial	48.3%	58.9%	87.9%	16.1%	55.2%	61.6%
		M-Attack	32.3%	36.5%	55.2%	11.2%	34.8%	38.0%
		GeoShield	26.0%	27.7%	44.4%	9.6%	26.7%	29.8%
		GeoCloak	9.5%	10.9%	18.0%	2.8%	9.8%	11.9%
	GeoMiner	Initial	46.9%	59.1%	88.2%	30.2%	53.7%	62.1%

Appendix

Table 13: Geolocation accuracy (%) under no defense and each defense (aggregated across all datasets). All metrics

\downarrow

lower is better.

Figure 8: Visual quality comparison with zoomed-in regions on a Im2GPS3k sample. It serves as a supplementary illustration to Figure 3 , providing additional visual comparison details Zoomed-in crops (top) reveal that GeoCloak introduces negligible utility degradation compared to GeoShield and M-Attack.

Figure 9: Visual quality comparison with zoomed-in regions on a Dataset-PCW sample. It serves as a supplementary illustration to Figure 3 , providing additional visual comparison details Zoomed-in crops (top) reveal that GeoCloak introduces negligible utility degradation compared to GeoShield and M-Attack.

Figure 10: Adversarial perturbation quality comparison. Each row shows a case where adversarial perturbation is the key skill in GeoCloak ’s successful protection turn. Numbers indicate attackers that successfully geolocate the image and CLIP similarity. GeoCloak introduces substantially fewer visual artifacts than GeoShield and M-Attack while achieving equal or superior protection.

Defense	Naturalness $\uparrow$	Semantic Preservation $\uparrow$	Overall Quality $\uparrow$
GeoShield	2.23 [2.06, 2.41]	3.68 [3.59, 3.76]	2.53 [2.38, 2.68]
M-Attack	2.31 [2.15, 2.47]	3.77 [3.67, 3.86]	2.63 [2.49, 2.78]
GeoCloak	4.57 [4.44, 4.69]	4.45 [4.34, 4.56]	4.43 [4.30, 4.55]

Appendix

Table 14: Complete human perception results on a five-point Likert scale. Each cell reports the mean and image-level bootstrap 95% confidence interval after averaging the two annotators’ ratings for each source image and defense. Higher is better (

\uparrow

		Initial		M-Attack		GeoShield		GeoCloak
Model	Method	API	Ref.	API	Ref.	API	Ref.	API	Ref.
Gemini-2.5-Pro	Zero-Shot	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Gemini-2.5-Pro	GeoMiner	0.0%	0.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Gemini-3.1-Pro	Zero-Shot	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Gemini-3.1-Pro	GeoMiner	0.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
GPT-5.4	Zero-Shot	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%

Appendix

Table 15: API error and safety refusal rates (%) per model, inference method, and defense method. API Error : request timeout or image-size limit (infrastructure failure). Safety Refusal : model explicitly declined to geolocalize ( e.g. , “I cannot help with that”).

Figure 11: Interactive version of GeoCloak .

Privacy Protection in VLMs

Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.

See all 37 papers

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models
Sep 18, 2026Yining Wang, Xi Li, Mi Zhang +4Privacy Protection in VLMs Adversarial Attacks on VLMs
Fudan University · East China University of Science and Technology · Shanghai University of Electric Power
GeoEdit: Adversarial De-Evidencing for Silent Geoprivacy Protection via Image Editing
Oct 19, 2026Yu Yan, Shengjia Cheng, Fei Ma +2Privacy Protection in VLMs Adversarial Attacks on VLMs
People’s Public Security University of China, Beijing, China · Guangming Laboratory, Shenzhen, China
AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO
Jun 3, 2025Xiaonan Wang, Bo Shao, Hansaem KimVLM Evaluation Privacy Protection in VLMs
Yonsei University, South Korea · CISPA Helmholtz Center for Information Security, Germany

Sep 18, 2026J/K move · Enter open · S save

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models

Yining Wang, Xi Li, Mi Zhang, Xiaohan Zhang+3

Fudan University · East China University of Science and Technology · Shanghai University of Electric Power

Privacy Protection in VLMs Adversarial Attacks on VLMs

Open paper AskPDF arXiv

Multimodal large reasoning models (MLRMs) have demonstrated remarkable capabilities in complex visual understanding. However, this very power introduces a critical yet underexplored privacy threat: adversaries can exploit MLRMs to precisely infer users' geographic locations from casually shared photographs, by performing structured reasoning over subtle visual cues such as architectural styles, vegetation, and lighting conditions. In this work, we present a systematic study of MLRM-driven geolocation privacy leakage. We first reveal that refusal-based safeguards are critically insufficient, as carefully crafted jailbreak prompts can raise model response rates to 100%. We further identify that existing defenses, which inject imperceptible perturbations into shared images, suffer from structural limitations intrinsic to their pixel-space optimization, resulting in degraded black-box transferability and pronounced visual artifacts. Motivated by these findings, we propose a diffusion-based framework that provides targeted, proactive defense against geolocation privacy leakage. By injecting perturbations into the latent space of a diffusion model during reverse sampling, our method operates directly on high-level semantic representations, thereby resolving the effectiveness-utility bottlenecks by construction. We further ground our optimization with GeoCLIP, a model explicitly aligned with GPS coordinates, as a surrogate to pinpoint and disrupt the geographic signals that MLRMs exploit for location inference. This targeted semantic disruption yields significantly stronger black-box transferability while preserving perceptual image quality, offering a seamless integration on social media platforms. Code is available at https://github.com/RachelWolowitz/Hiding_in_plain_sight.

Abstract

Privacy Protection in VLMs

Explore similar work

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models

GeoEdit: Adversarial De-Evidencing for Silent Geoprivacy Protection via Image Editing

AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO