AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO

Xiaonan Wang; Bo Shao; Hansaem Kim

cs.CVJun 3, 2025

AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO

Authors: Xiaonan Wang, Bo Shao, Hansaem Kim

Organizations: Yonsei University, South Korea · CISPA Helmholtz Center for Information Security, Germany

Abstract

Recent advances in vision-language models (VLMs) have enabled accurate image-based geolocation, raising serious concerns about location privacy risks in everyday social media posts. Yet, a systematic evaluation of such risks is still lacking: existing benchmarks show coarse granularity, linguistic bias, and a neglect of multimodal privacy risks. To address these gaps, we introduce KoreaGEO, the first fine-grained, multimodal, and privacy-aware benchmark for geolocation, built on Korean street views. The benchmark covers four socio-spatial clusters and nine place types with rich contextual annotations and two captioning styles that simulate real-world privacy exposure. To evaluate mainstream VLMs, we design a three-path protocol spanning image-only, functional-caption, and high-risk-caption inputs, enabling systematic analysis of localization accuracy, spatial bias, and reasoning behavior. Results show that input modality exerts a stronger influence on localization precision and privacy exposure than model scale or architecture, with high-risk captions substantially boosting accuracy. Moreover, they highlight structural prediction biases toward core cities.

Figures & tables

Cluster	Name	Examples	Key Characteristics
Cluster 0	Capital Hypercore	Seoul Only	Extremely dense, highest GRDP
Cluster 1	Subregional Areas	Chuncheon, all gun (county) units…	Mid/low density, rural/urban mix
Cluster 2	Regional Growth Hubs	Busan, Daejeon…	High GRDP, mid-to-high density
Cluster 3	Peri-Capital Satellites	Suwon, Seongnam…	High residential density, Seoul periphery

Table 1: Summary of the four clusters derived from population density and GRDP, using representative cities and key socio-spatial characteristics.

Figure 1: Overview of the nine place types, capturing functional and semantic diversity across Korean Landscapes.

Figure 2: An example entry from KoreaGEO (English translations are provided when the original text is in Korean.).

Figure 3: Geographical Distribution of Sampled Coordinates in KoreaGEO.

Model	Image Only				Image + Functional Caption				Image + High-risk Caption
	0.1km	1km	20km	100km	0.1km	1km	20km	100km	0.1km	1km	20km	100km
o3	1.94	9.91	38.24	63.52	2.69↑	13.89↑	44.63↑	69.63 ↑	7.13↑	46.48↑	96.48↑	98.33↑
GPT-4o	2.31	7.59	37.69	65.00	1.85↓	5.83↓	38.61↑	65.19↑	9.17↑	46.20↑	95.93↑	98.33↑
GPT-4o mini	0.83	3.80	33.61	61.67	0.65↓	4.07↑	32.31↓	58.15↓	2.31↑	17.96↑	88.43↑	96.94↑
GPT-4.1 mini	0.74	5.19	35.56	64.44	0.65↓	4.54↓	34.17↓	65.46↑	4.44↑	27.31↑	92.78↑	98.61↑
GPT-4.1 nano	0.83	3.06	33.15	61.85	0.56↓	2.87↓	32.04↓	61.67↓	2.59↑	21.85↑	87.78↑	97.22↑

Table 2: Top-1 accuracy (%) at four distance thresholds under three input modalities. For Image + Functional Caption and Image + High-risk Caption , arrows indicate change relative to Image Only (↑ gain, ↓ drop, → no change). Best values in bold; lowest underlined.

Figure 4: Cumulative Distribution Function (CDF) of geolocation error across input modalities. Vertical dashed lines denote distance thresholds (0.1 km, 1 km, 20 km, 100 km).

Figure 5: Accuracy distribution across different place types under four distance thresholds. Radar plots illustrate how models perform at 0.1km, 1km, 20km, and 100km levels, with clearer separation observed under stricter thresholds.

Figure 6: Confusion matrix between real and predicted city clusters. Cluster 0 shows the highest accuracy, while most errors flow from Clusters 2 and 3 into Cluster 0, indicating a strong centralization bias.

Figure 7: Confusion matrix between true and predicted cities (top 20 by frequency). Most errors converge on Seoul, indicating a strong centralization bias in model predictions.

Figure 8: Word cloud of model inferences under the visual-only setting. Larger words indicate higher frequency across all model outputs.

Figure 9: Modality attribution by Gemini under the Visual with Functional Caption setting.

Factor	Category	Accuracy (%)
Text Language	Korean	34.43
	Foreign	41.95
	None	28.00
Text Presence	Present	35.22
	None	27.86
Text Type	Commercial	35.61

Table 3: Localization accuracy (%) at 20km across various contextual factors. Accuracy is computed independently with balanced downsampling.

Model Name	Shannon Entropy
Gemini 2.5 Pro	5.0595
o3	4.7553
LLaMA-3.2-90B-Vision-Instruct	4.1271
GPT-4o	3.8497
Claude 3.7 Sonnet	3.5370
GPT-4.1 mini	3.3719

Table 4: Model bias ranking based on raw Shannon entropy over predicted city distributions.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Elbow method (left) and silhouette scores (right) for different values of $K$ .

Figure 11: Scatterplot of Korean cities clustered by population density and GRDP.

Place Type	Definition
Commercial and Recreational Zones	Areas for retail, dining, entertainment, and leisure, e.g., shopping streets, markets, and sports complexes.
Educational and Cultural Institutions	Spaces for learning and culture, such as schools, libraries, museums, and art centers.
Governmental and Public Services	Facilities serving administration, safety, and welfare, including city halls and post offices.
Industrial and Logistic Facilities	Manufacturing and transport-related zones like factories, warehouses, and ports.
Natural and Environmental Settings	Outdoor natural areas like forests, parks, rivers, and beaches, with minimal infrastructure.
Religious and Memorial Sites	Locations for worship and commemoration, such as temples, churches, and memorials.

Appendix

Table 5: Definitions of the nine high-level place types used in the dataset.

Cluster	Units
0	Seoul Special Metropolitan City
1	Sejong Special Self-Governing City, Chuncheon, Wonju, Gangneung, Donghae, Taebaek, Sokcho, Samcheok, Hongcheon-gun, Hoengseong-gun, Yeongwol-gun, Pyeongchang-gun, Jeongseon-gun, Cheorwon-gun, Hwacheon-gun, Yanggu-gun, Inje-gun, Goseong-gun, Yangyang-gun, Chungju, Jecheon, Cheongju, Boeun-gun, Okcheon-gun, Yeongdong-gun, Jincheon-gun, Goesan-gun, Eumseong-gun, Danyang-gun, Jeungpyeong-gun, Cheonan, Gongju, Boryeong, Asan, Seosan, Nonsan, Gyeryong, Dangjin, Geumsan-gun, Buyeo-gun, Seocheon-gun, Cheongyang-gun, Hongseong-gun, Yesan-gun, Taean-gun, Jeonju, Gunsan, Iksan, Jeongeup, Namwon, Gimje, Wanju-gun, Jinan-gun, Muju-gun, Jangsu-gun, Imsil-gun, Sunchang-gun, Gochang-gun, Buan-gun, Yeosu, Suncheon, Naju, Gwangyang, Damyang-gun, Gokseong-gun, Gurye-gun, Goheung-gun, Boseong-gun, Hwasun-gun, Jangheung-gun, Gangjin-gun, Haenam-gun, Yeongam-gun, Muan-gun, Hampyeong-gun, Yeonggwang-gun, Jangseong-gun, Wando-gun, Jindo-gun, Shinan-gun, Pohang, Gyeongju, Gimcheon, Andong, Gumi, Yeongju, Yeongcheon, Sangju, Mungyeong, Gyeongsan, Uiseong-gun, Cheongsong-gun, Yeongyang-gun, Yeongdeok-gun, Cheongdo-gun, Goryeong-gun, Seongju-gun, Chilgok-gun, Yechon-gun, Bonghwa-gun, Uljin-gun, Ulleung-gun, Changwon, Jinju, Tongyeong, Sacheon, Gimhae, Miryang, Geoje, Yangsan, Uiryeong-gun, Haman-gun, Changnyeong-gun, Goseong-gun, Namhae-gun, Hadong-gun, Sancheong-gun, Hamyang-gun, Geochang-gun, Hapcheon-gun, Jeju, Seogwipo, Yongin, Namyangju, Pyeongtaek, Paju, Gimpo, Gwangju, Yangju, Icheon, Anseong, Pocheon, Uiwang, Yangpyeong-gun, Yeoju, Dongducheon, Gwacheon, Gapyeong-gun, Yeoncheon-gun
2	Busan Metropolitan City, Daegu Metropolitan City, Incheon Metropolitan City, Gwangju Metropolitan City, Daejeon Metropolitan City, Ulsan Metropolitan City , Hwaseong
3	Mokpo, Suwon, Goyang, Seongnam, Bucheon, Ansan, Anyang, Siheung, Uijeongbu, Gwangmyeong, Gunpo, Hanam, Osan, Guri

Appendix

Table 6: Cluster assignment results showing each cluster and its member cities/counties.

Feature	Label Values
indoor_outdoor	indoor, outdoor
time_of_day	day, night, dusk, none
season	spring, summer, autumn, winter
weather	clear, cloudy, rainy, snowy, none
text_presence	present, none
text_language	Korean, foreign, none

Appendix

Table 7: Label sets for the eleven contextual features annotated in the dataset.

Panel of Figure 12 — Figure 12: Prompt for generating functional captions without explicit place names.

Panel of Figure 13 — Figure 13: Prompt for generating high-risk captions that explicitly mention place names.

Feature	Label Value	Count
indoor_outdoor	outdoor	1007
indoor_outdoor	indoor	73
time_of_day	day	951
	none	73
	night	32
	dusk	24

Appendix

Table 8: Distribution of annotated contextual features in Korea GEO Bench.

Panel of Figure 14 — Figure 14: Prompt used for the Image Only input setting.

Panel of Figure 15 — Figure 15: Prompt used for the Image + Functional Caption input setting.

Panel of Figure 16 — Figure 16: Prompt used for the Image + High-risk Caption input setting.

Model	$\leq$ 100m		$\leq$ 1.0km		$\leq$ 20.0km		$\leq$ 100.0km
	Orig.	Occ.	Orig.	Occ.	Orig.	Occ.	Orig.	Occ.
Claude 3.7 Sonnet	3.33	0.00	13.33	10.83	37.50	32.50	68.33	54.17
GPT-4.1 mini	0.00	0.00	8.33	4.17	40.00	30.83	68.33	45.83
GPT-4.1 nano	0.00	0.00	4.17	2.50	35.00	21.67	69.17	42.50
GPT-4o mini	0.00	0.00	3.33	3.33	38.33	23.33	70.83	49.17
GPT-4o	0.83	0.00	13.33	9.17	43.33	35.83	67.50	60.00

Appendix

Table 9: Top-1 geolocation accuracy (%) before and after text obfuscation on 120 images from Commercial and Recreational Zones. Orig. = original image; Occ. = image with key text occluded.

Model	GV_0.1km	GV_1km	GV_20km	MP_0.1km	MP_1km	MP_20km
o3	5.19	23.33	77.78	4.81	28.15	92.96
4o	6.30	20.74	82.59	7.41	20.37	98.89
4o-mini	2.96	12.96	94.07	3.33	8.15	96.30
4.1-mini	1.85	16.67	87.78	2.59	14.44	95.56
4.1-nano	3.33	11.85	96.67	5.56	12.59	98.52
Gemini	9.26	28.52	77.78	11.11	23.70	89.63

Appendix

Table 10: Model accuracy comparison between Google Street View images (GV) and manually collected photos (MP) in Cluster 0 (Capital Hypercore). Each dataset contains 270 images.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 986 papers

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models
Sep 18, 2026Yining Wang, Xi Li, Mi Zhang +4Privacy Protection in VLMs Adversarial Attacks on VLMs
Fudan University · East China University of Science and Technology · Shanghai University of Electric Power
GeoCloak: An Agentic Framework for Semantics-Preserving Geolocation Privacy via Peripheral Clue Manipulation
Oct 14, 2026Yixin Wu, Mingjie Li, Yun Shen +2Privacy Protection in VLMs Adversarial Attacks on VLMs
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · CISPA Helmholtz Center for Information Security +1
GeoAgent: Evaluating VLM Geolocalization Through Embodied Navigation
Aug 30, 2026Arka Mukherjee, Soham Roy, Kartikeya Trivedi +1Geospatial Reasoning Benchmarks VLM Reasoning
KIIT Bhubaneswar · IIT Bhubaneswar

Sep 18, 2026J/K move · Enter open · S save

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models

Yining Wang, Xi Li, Mi Zhang, Xiaohan Zhang+3

Fudan University · East China University of Science and Technology · Shanghai University of Electric Power

Privacy Protection in VLMs Adversarial Attacks on VLMs

Open paper AskPDF arXiv

Multimodal large reasoning models (MLRMs) have demonstrated remarkable capabilities in complex visual understanding. However, this very power introduces a critical yet underexplored privacy threat: adversaries can exploit MLRMs to precisely infer users' geographic locations from casually shared photographs, by performing structured reasoning over subtle visual cues such as architectural styles, vegetation, and lighting conditions. In this work, we present a systematic study of MLRM-driven geolocation privacy leakage. We first reveal that refusal-based safeguards are critically insufficient, as carefully crafted jailbreak prompts can raise model response rates to 100%. We further identify that existing defenses, which inject imperceptible perturbations into shared images, suffer from structural limitations intrinsic to their pixel-space optimization, resulting in degraded black-box transferability and pronounced visual artifacts. Motivated by these findings, we propose a diffusion-based framework that provides targeted, proactive defense against geolocation privacy leakage. By injecting perturbations into the latent space of a diffusion model during reverse sampling, our method operates directly on high-level semantic representations, thereby resolving the effectiveness-utility bottlenecks by construction. We further ground our optimization with GeoCLIP, a model explicitly aligned with GPS coordinates, as a surrogate to pinpoint and disrupt the geographic signals that MLRMs exploit for location inference. This targeted semantic disruption yields significantly stronger black-box transferability while preserving perceptual image quality, offering a seamless integration on social media platforms. Code is available at https://github.com/RachelWolowitz/Hiding_in_plain_sight.

Abstract

VLM Evaluation

Explore similar work

Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models

GeoCloak: An Agentic Framework for Semantics-Preserving Geolocation Privacy via Peripheral Clue Manipulation

GeoAgent: Evaluating VLM Geolocalization Through Embodied Navigation