cs.CVNov 4, 2026
SaveGeoBiaset: A Counterfactual Benchmark for Demographic Bias in World-Level Geolocalization
Organizations: University of Florence
Abstract
World-level image geolocalization is an increasingly important task, yet the biases of current models remain poorly understood. Prior work has identified coarse dataset imbalances noting, for instance, that some cities are absent from training sets and that models tend to perform better in wealthier regions. However, no one has yet examined which visual cues drive these disparities. To do so, we introduce GeoBiaset, the first benchmark designed to measure how geolocalization models are influenced by the apparent race of a person inserted into the image foreground. The dataset consists of 9,541 images, pairing existing geotagged background images with counterfactual edits spanning seven races, holding the ground-truth coordinates fixed. Controlled masking experiments show that occlusion and background drift do not fully explain geolocalization degradation. Alongside the benchmark, we introduce the Person-Origin Perturbation Indicators (POPI), novel metrics that quantify model resilience to these demographic perturbations by measuring whether predictions are pulled toward the region associated with the inserted group. Our evaluation of state-of-the-art geolocalization and multimodal large language models reveals a 16-47% drop in continent-level performance when a subject from a demographically mismatched group is inserted. A landmark versus no-landmark analysis and gradient-weighted heatmaps show that models preferentially attend to geographic landmarks when present, but shift attention to facial features in their absence.
Figures & tables
Figure 1: Left: Standard geolocalization benchmarks are imbalanced: scene cues and human attributes co-occur, making it unclear whether models rely on place-specific evidence or spurious demographic correlations. Right: GeoBiaset generates counterfactual edits varying race, gender, and attire under fixed GPS coordinates. Black diamonds indicate predictions for original images; colored circles indicate predictions for edited images, with colors denoting the inserted person’s race category. Displacement toward race-associated regions highlights the audited bias.
Figure 2: Overview of the GeoBiaset image-generation pipeline. Starting from a background image, an input face, and an editing prompt that specifies race, gender and optionally race-associated clothing we generate an edited image while preserving the overall geographic scene, without enforcing pixel-level accuracy. A multimodal quality-control module detects severe generation artifacts and triggers regeneration when needed. For each starting background image, we generate 28 edited images (2 genders × 2 attire types × 7 races).
| Methods | Edited | Street 1km | City 25km | Region 200km | Country 750km | Continent 2500km | POPI-A ( ) | POPI-H ( ) | POPI-D ( ) |
|---|---|---|---|---|---|---|---|---|---|
| GeoCLIP | ✗ | 11.1 | 35.6 | 51.6 | 68.8 | 81.9 | 29.3 | 20.9 | 30.7 |
| ✓ | 4.3 | 16.3 | 25.4 | 38.1 | 55.3 33% | ||||
| Img2Loc | ✗ | 8.2 | 28.0 | 37.0 | 53.1 | 70.3 | 34.5 | 24.3 | 36.2 |
| ✓ | 2.2 | 7.5 | 11.8 | 21.8 | 37.5 47% | ||||
| G3 | ✗ | 16.6 | 45.8 | 55.7 | 70.3 | 81.6 | 28.1 | 20.0 | 29.4 |
| ✓ | 3.6 | 13.3 | 19.0 | 30.7 | 47.8 41% |
Table 1: Overall results on GeoBiaset . We report geolocalization accuracy at standard geodesic distance thresholds, corresponding approximately to street-, city-, region-, country-, and continent-level localization. The Edited column indicates whether the image is an original background image (✗) or an edited image containing an inserted person (✓). Values in red denote the relative accuracy drop from background to edited images. POPI-A, POPI-H, and POPI-D measure race-induced prediction shifts; lower values indicate lower sensitivity to the inserted person.
Figure 3: Landmark sensitivity and attribution analysis for GeoCLIP. Left : POPI-H and POPI-D for landmark and non-landmark images. Right : representative gradient-weighted attention rollout maps showing that non-landmark scenes assign more relevance to the inserted person, consistent with larger prediction shifts.
| Race | 1km | 25km | 200km | 750km | 2500km |
|---|---|---|---|---|---|
| Black | -2.2 | -4.7 | -9.9 | -12.4 | -15.5 |
| East Asian | -1.2 | -4.1 | -7.0 | -8.7 | -10.6 |
| Indian | -0.6 | -1.9 | -5.0 | -3.6 | -3.6 |
| Latino Hispanic | 0.0 | -3.6 | -3.9 | -5.1 | -5.4 |
| Middle Eastern | -1.6 | -2.0 | -3.5 | -1.6 | +0.4 |
| Southeast Asian | -0.1 | -3.1 | -6.6 | -7.3 | -9.0 |
Table 2: Effect of race-associated attire on GeoCLIP accuracy. Values report the accuracy change between attire-conditioned and standard edited images. Negative values indicate accuracy drops; darker cells denote larger drops.
| Race | 1km | 25km | 200km | 750km | 2500km |
|---|---|---|---|---|---|
| Black | +1.3 | +4.9 | +5.7 | +8.3 | +8.8 |
| East Asian | +1.4 | +4.0 | +6.1 | +8.6 | +9.2 |
| Indian | +2.7 | +5.9 | +5.4 | +7.5 | +8.8 |
| Latino Hispanic | +1.0 | +4.3 | +6.2 | +9.4 | +11.7 |
| Middle Eastern | +0.3 | +5.3 | +6.5 | +11.0 | +10.7 |
| Southeast Asian | +1.4 | +4.1 | +7.2 | +9.0 | +7.9 |
Table 3: Effect of face obfuscation on GeoCLIP accuracy. Values report the accuracy gain after applying face blurring to the edited images, relative to the corresponding non-obfuscated edited images. Positive values indicate that obfuscation improves accuracy, suggesting reduced sensitivity to explicit facial cues. Darker cells denote larger gains.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Full GeoBiaset counterfactual grid for a landmark scene with all 28 edited variants across seven races (rows) and four gender-attire conditions (columns), sharing the same ground-truth GPS coordinate; subject composition and background appearance may vary across edits.
Figure 5: Full GeoBiaset counterfactual grid for a non-landmark scene with all 28 edited variants across seven races (rows) and four gender-attire conditions (columns), sharing the same ground-truth GPS coordinate; subject composition and background appearance may vary across edits.
| Race | Masculine garments | Feminine garments |
|---|---|---|
| White | suit and tie, jeans and t-shirt, chinos with polo shirt, bomber jacket, cable-knit sweater, trench coat, flannel shirt with jeans, tuxedo | sundress, jeans and blouse, pencil skirt with top, trench coat, wrap dress, cardigan with slacks, maxi skirt with blouse, cocktail dress |
| Black | dashiki, suit and tie, streetwear hoodie and joggers, agbada, senator suit, boubou, kente cloth shirt, kaftan | ankara wrap dress, dashiki blouse with trousers, kente cloth dress, gele headwrap with boubou, iro and buba, lappa skirt with top, kitenge dress, adire wrap dress |
| Southeast Asian | batik shirt, sarong with shirt, barong tagalog, longyi, baju melayu, pha nung, puso ng saging, teluk belanga | kebaya, ao dai, sarong with blouse, baro’t saya, baju kurung, pha sin wrap skirt, inabel dress, nyonya kebaya |
| East Asian | changshan, hanbok (jeogori and baji), hakama with kimono, modern suit, mandarin collar jacket, haori, magua jacket, durumagi overcoat | qipao, hanbok (jeogori and chima), kimono, hanfu, furisode, jeogori with maxi chima, ruqun, uchikake |
| Latino/Hispanic | guayabera shirt, charro suit, poncho with trousers, linen shirt and slacks, ruana cloak, jorongo poncho, montuno shirt, bombacha trousers | huipil, pollera, china poblana, floral wrap skirt with blouse, mestiza dress, cholita layered skirt, jorongo shawl with dress, rebozo wrap with blouse |
| Middle Eastern | thobe (dishdasha), keffiyeh with agal, shalwar kameez, jalabiya, bisht cloak, sirwal trousers with tunic, jubba, egal with ghutra | abaya, kaftan, jilbab with hijab, thobe al-nashal, farasha dress, jalabiya with tarha, bisht evening cloak, shayla with maxi dress |
Table 4: Curated garment list per race, split by gender adjective.
Table 5: Effect of explicit race-associated attire conditioning across distance thresholds. Each subtable reports accuracy in the race-associated condition minus accuracy in the attire-unconditioned condition, in percentage points. Negative values indicate lower accuracy under race-associated explicit attire conditioning. The attire-unconditioned setting omits clothing descriptions but can retain the generator’s demographic clothing priors.
Figure 6: Landmark sensitivity analysis across six models. POPI-H and POPI-D for landmark and non-landmark images. Non-landmark images exhibit larger prediction shifts across all models, suggesting that the inserted person has a stronger effect when distinctive geographic landmarks are absent.
| Inserted Group | POPI-A | POPI-H | POPI-D |
|---|---|---|---|
| Black | 37.1 | 29.3 | 38.4 |
| East Asian | 28.7 | 17.0 | 30.7 |
| Indian | 30.0 | 15.4 | 32.5 |
| Latino Hispanic | 30.2 | 16.0 | 32.6 |
| Middle Eastern | 29.5 | 38.3 | 28.0 |
| Southeast Asian | 33.0 | 19.7 | 35.2 |
Table 6: GeoCLIP POPI metrics by inserted demographic group. Scores are computed over the corresponding eligible baseline populations defined in Sec. 4.1 . Values are percentages.
| Methods | Edited | Street 1km | City 25km | Region 200km | Country 750km | Continent 2500km | POPI-A ( ) | POPI-H ( ) | POPI-D ( ) |
|---|---|---|---|---|---|---|---|---|---|
| InternVL3.5-38B | ✗ | 1.2 | 15.2 | 28.6 | 48.9 | 71.1 | 23.2 | 21.4 | 23.4 |
| ✓ | 1.0 | 11.0 | 19.5 | 36.5 | 55.4 22% | ||||
| Gemma3-27B-IT | ✗ | 5.5 | 30.7 | 48.0 | 68.7 | 80.6 | 15.7 | 19.2 | 15.1 |
| ✓ | 3.8 | 24.6 | 37.8 | 54.4 | 67.8 16% | ||||
| Qwen3VL-30B-A3B | ✗ | 5.2 | 33.7 | 50.2 | 74.5 | 86.3 | 14.3 | 14.4 | 14.3 |
| ✓ | 4.4 | 29.3 | 41.8 | 60.4 | 72.9 16% |
Table 7: Geolocalization accuracy and POPI metrics for larger MLLMs. Accuracy at standard geodesic distance thresholds and POPI transition rates are reported as percentages. Original and Edited denote unmodified source backgrounds and their counterfactual edited variants, respectively.
| Source | Mask | 1 km | 25 km | 200 km | 750 km | 2500 km |
|---|---|---|---|---|---|---|
| Original | None | 11.1 | 35.6 | 51.6 | 68.8 | 81.9 |
| Edited | None | 4.3 | 16.3 | 25.4 | 38.1 | 55.3 |
| Original | Black | 8.1 | 28.4 | 40.9 | 57.9 | 73.4 |
| Edited | Black | 8.1 | 27.8 | 40.2 | 56.1 | 72.7 |
| Original | Gray | 7.5 | 27.7 | 40.8 | 58.6 | 73.6 |
| Edited | Gray | 7.4 | 26.7 | 39.5 | 56.0 | 71.7 |
Table 8: Geolocalization accuracy for the foreground-masking control experiment. For each edited image, the instance-specific SAM3 mask of the inserted subject is applied identically to its matched original background. Accuracies are reported as percentages varying the distance thresholds.