GeoDiv: Framework For Measuring Geographical Diversity In Text-To-Image Models

Abhipsa Basu; Mohana Singh; Shashank Agnihotri; Margret Keuper; R. Venkatesh Babu

cs.CVFeb 25, 2026

GeoDiv: Framework For Measuring Geographical Diversity In Text-To-Image Models

Authors: Abhipsa Basu, Mohana Singh, Shashank Agnihotri, Margret Keuper, R. Venkatesh Babu

Organizations: Vision and AI Lab, Indian Institute of Science, Bangalore, India · Chair for Machine Learning, University of Mannheim, Mannheim, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany

Abstract

Text-to-image (T2I) models are rapidly gaining popularity, yet their outputs often lack geographical diversity, reinforce stereotypes, and misrepresent regions. Given their broad reach, it is critical to rigorously evaluate how these models portray the world. Existing diversity metrics either rely on curated datasets or focus on surface-level visual similarity, limiting interpretability. We introduce GeoDiv, a framework leveraging large language and vision-language models to assess geographical diversity along two complementary axes: the Socio-Economic Visual Index (SEVI), capturing economic and condition-related cues, and the Visual Diversity Index (VDI), measuring variation in primary entities and backgrounds. Applied to images generated by models such as Stable Diffusion and FLUX.1-dev across $10$ entities and $16$ countries, GeoDiv reveals a consistent lack of diversity and identifies fine-grained attributes where models default to biased portrayals. Strikingly, depictions of countries like India, Nigeria, and Colombia are disproportionately impoverished and worn, reflecting underlying socio-economic biases. These results highlight the need for greater geographical nuance in generative models. GeoDiv provides the first systematic, interpretable framework for measuring such biases, marking a step toward fairer and more inclusive generative systems. Project page: https://abhipsabasu.github.io/geodiv

Figures & tables

Figure 1: Lack of Geographical Diversity observed in T2I Generations and the Need for GeoDiv. (a) Text-to-image models produce systematically low visual diversity for the same prompt across countries (example: ‘ a photo of a house in Nigeria ’), failing to reflect the rich variation seen in real-world images ( Ramaswamy et al., 2023 ) . (b) GeoDiv provides an automated, reference-free framework that can quantify such fine-grained geographical differences by evaluating images along four interpretable axes: Entity-Appearance (sloped/flat roof), Background-Appearance (paved/unpaved road), Affluence (luxury/modest settings), and Maintenance (manicured/unkempt). Examples show how the same entity type varies dramatically across countries and generative models.

Figure 2: GeoDiv Pipeline. Given an entity $e$ and country $c$ , LLMs generate attribute-based questions specific to $e$ , and a fixed set of background-related questions applicable across entities. A VQA model predicts answer distributions over an image set for both question types, from which GeoDiv computes the Visual Diversity Index (VDI) via normalized Hill number. The VQA model also rates each image on Affluence and Maintenance to compute the Socio-Economic Visual Index (SEVI).

Models	VDI Answers (Accuracy)			SEVI Scores (Spearman’s $\rho$ )
Models	Entity	Background	Overall	Affluence	Maintenance
Gemini-2.5-flash	$0.87$	$0.85$	$0.86$	$0.76$	$0.69$
gpt-4o	$0.85$	$0.81$	$0.83$	$0.76$	$0.76$
Qwen2.5-VL	$0.85$	$0.77$	$0.81$	$0.69$	$0.71$
llava-v1.6-mistral-7b-hf	$0.70$	$0.66$	$0.68$	$0.65$	$0.68$

Table 1: Performance of various VQA models in identifying VDI answers and SEVI scores compared against human annotations . Gemini-2.5-flash achieves the highest accuracy on entity and background questions, as well as the strongest correlation with human ratings on the SEVI metrics. Qwen2.5-VL is competitive, while LLaVA underperforms substantially.

Figure 3: SEVI Diversity and Mean Ratings across Datasets and Countries . India (IN), Nigeria (NG), and Colombia (CO) are seen to receive lower SEVI ratings, while the US, UK, and Japan (JP) rank highest—revealing strong socio-economic biases in country-level image representations. Strikingly, none of the models generate images spanning diverse socio-economic strata.

Figure 4: VDI Scores across (a) Datasets, (b) Countries . Model-wise VDI diversities are similar, with SD2.1 achieving higher scores than the others. Mexico and the UK show low entity and background diversity, while Japan scores highest.

Figure 5: Country-wise maximum and mean JS Divergence across Entities and Models . High maximum values for both Entity and Background indicate high cross-country variations in the respective attribute value distributions.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Object	Indoor						Outdoor
		GeoDE	SDv2	SDv3	SDv3.5	FLUX.1	Avg	GeoDE	SDv2	SDv3	SDv3.5	FLUX.1	Avg
Indoor common	bag	95.71	3.78	10.07	13.34	35.16	31.61	4.29	96.22	89.93	86.66	64.84	68.39
Indoor common	chair	88.11	1.41	10.86	12.38	84.09	39.37	11.89	98.59	89.14	87.62	15.91	60.63
Indoor rare	cooking pot	95.96	0.87	26.61	10.57	57.29	38.26	4.04	99.13	73.39	89.43	42.71	61.74
	plate of food	94.98	95.00	98.47	94.07	98.95	96.29	5.02	5.00	1.53	5.93	1.05	3.71
	stove	93.14	16.37	67.18	57.64	87.74	64.41	6.86	83.63	32.82	42.36	12.26	35.59

Appendix

Table 2: Indoor-Outdoor Distribution .

Entity	Question	SD2.1	SD3m	SD3.5	FLUX.1	GeoDE
Bag	Is the bag’s closure type visible or identifiable in the image?	44.5	50.5	43.05	28.95	25.22
Car	Is it visible or detectable from the image if the car’s lights are turned on or off?	22.8	8.55	11.48	1.42	18.22
	Is the car’s roof type visible or identifiable in the image?	14.92	30.47	20.8	22.32	11.12
Chair	Is the construction style of the chair’s backrest visible or identifiable in the image?	2.28	22.7	1.95	15.67	23.25
House	Is it visible or detectable from the image whether a door on the house is open or closed?	17.15	9.72	9.9	2.03	30.11
Storefront	Is the type of the storefront entrance visible or identifiable in the image?	27.28	16.25	7.53	2.5	19.26

Appendix

Table 3: Visibility Check Failure Rates for Selected Entity-Question Pairs Across Datasets.

Entity	SD2.1	SD3m	SD3.5	FLUX.1	GeoDE	Entity Avg.
Bag	6.05	3.84	4.63	0.99	2.73	3.65
Backyard	0.22	0.48	0.19	0.52	0.33	0.35
Car	2.02	2.34	4.41	3.63	5.17	3.51
Chair	0.95	1.25	1.00	1.45	1.66	1.26
Cooking Pot	1.24	0.79	0.27	0.06	3.65	1.20
Dog	0.12	0.03	0.07	0.68	0.06	0.19

Appendix

Table 4: NOTA percentages per entity across datasets, with per-entity average.

Figure 6: Affluence and Maintenance (SEVI) Scores across Entities. Chair and Stove images show the highest variance in Affluence, whereas Cooking Pot and Stove images appear the least affluent. For Maintenance, Stove, Cooking Pot and Chair turn out to be the most diverse, though the mean ratings are low for each of them.

Figure 7: Entity and Background Appearance (VDI) Scores across Entities. While Bag and Stove images demonstrate considerably higher entity diversity, Chair and Plate of Food are the least diverse. The Background-Diversity for these Entities vary considerably, and are distinctly lower than the Entity Diversity-Scores. Plate of Food images understandably are the least diverse, as most of them are closeups of the entity itself, whereas dogs, cars and houses demonstrate variation in the background to some extent.

Country	GeoDiv Score
Egypt	0.4106
Colombia	0.4079
Turkey	0.4049
Spain	0.4046
Indonesia	0.3999
China	0.3967

Appendix

Table 5: Average GeoDiv scores across countries.

Model	GeoDiv Score
SD2.1	0.4251
SD3m	0.3655
SD3.5	0.3455
FLUX.1	0.3153

Appendix

Table 6: Average GeoDiv Scores across models.

Figure 8: Entity diversity scores across generative models. Countries (a-p) : a) USA b) UK c) UAE d) Turkey e) Spain f) South Korea g) Philippines h) Nigeria i) Mexico j) Japan k) Italy l) Indonesia m) India n) Egypt o) Colombia p) China. Entities (A-J) : A) Backyard B) Bag C) Car D) Chair E) Cooking pot F) Dog G) house H) Plate of food I) Storefront J) Stove. The dataset with highest average diversity is SD2.1 and lowest is FLUX.1.

Figure 9: Jensen Shannon Distances (JSD) of Entity Attribute Distributions Across Countries for SD3.5 images of House. We note the higher JSD values for countries like Egypt, Mexico and USA, signalling that they possess distinct features compared to the other studied countries.

Figure 10: Background diversity scores across generative models. Countries (a-p) : a) USA b) UK c) UAE d) Turkey e) Spain f) South Korea g) Philippines h) Nigeria i) Mexico j) Japan k) Italy l) Indonesia m) India n) Egypt o) Colombia p) China. Entities (A-J) : A) Backyard B) Bag C) Car D) Chair E) Cooking pot F) Dog G) house H) Plate of food I) Storefront J) Stove.

Dataset	Entity Diversity			Background Diversity
Dataset	Rank †	Mean	Std	Rank †	Mean	Std
SD21	1 (7/10)	0.508	0.114	1 (7/10)	0.354	0.149
SD3m	2 (2/10)	0.448	0.104	3 (0/10)	0.306	0.137
FLUX.1	3 (0/10)	0.424	0.117	2 (3/10)	0.317	0.141
SD35	4 (1/10)	0.397	0.114	4 (0/10)	0.283	0.143

Appendix

Table 7: Comparison of entity and background diversity across datasets. SD21 ranks first for entity and background diversity but with notable variance. FLUX.1 and SD3m are more consistent but less diverse. † Dataset rank is based on number of entities for which it had highest diversity.

Figure 11: Affluence diversity scores across generative models. Countries (a-p) : a) USA b) UK c) UAE d) Turkey e) Spain f) South Korea g) Philippines h) Nigeria i) Mexico j) Japan k) Italy l) Indonesia m) India n) Egypt o) Colombia p) China. Entities (A-J) : A) Backyard B) Bag C) Car D) Chair E) Cooking pot F) Dog G) house H) Plate of food I) Storefront J) Stove.

Figure 12: Maintenance diversity scores across generative models. Countries (a-p) : a) USA b) UK c) UAE d) Turkey e) Spain f) South Korea g) Philippines h) Nigeria i) Mexico j) Japan k) Italy l) Indonesia m) India n) Egypt o) Colombia p) China. Entities (A-J) : A) Backyard B) Bag C) Car D) Chair E) Cooking pot F) Dog G) house H) Plate of food I) Storefront J) Stove.

Entity	UK	Nig	Tur	Indo	Col	Jap	Ind	Chi	USA	Mex	UAE	SKor	Spa	Ita	Egy	Phil
backyard	60	352	192	125	64	153	0	13	0	63	13	23	33	74	13	59
bag	103	176	178	312	126	212	0	73	0	87	26	77	124	154	75	208
car	92	203	161	136	97	139	0	80	0	58	35	51	106	137	54	45
chair	84	137	177	270	143	183	0	66	0	68	45	74	96	142	121	175
cooking pot	75	116	162	87	110	177	0	25	0	56	23	35	95	97	54	59
dog	24	93	214	24	79	142	0	27	0	77	0	21	43	85	27	161

Appendix

Table 8: GeoDE entity-country distribution . In the table below, we show the entity counts by country.

Figure 13: Diversity scores across the four axes for GeoDE. Countries (a-p) : a) USA b) UK c) UAE d) Turkey e) Spain f) South Korea g) Philippines h) Nigeria i) Mexico j) Japan k) Italy l) Indonesia m) India n) Egypt o) Colombia p) China. Entities (A-J) : A) Backyard B) Bag C) Car D) Chair E) Cooking pot F) Dog G) house H) Plate of food I) Storefront J) Stove.

Model	Original	Balanced	Difference
SD2.1	0.56	0.94	+0.38
SD3m	0.62	0.87	+0.25
FLUX.1	0.52	0.88	+0.36

Appendix

Table 9: Improvement in Affluence Diversity achieved by utilizing GeoDiv’s Affluence Scores

Entity	Question	Answer List
Backyard	1. Are there any animals or pets in the backyard?	Yes, No
	2. Are there any distinct pathways or walkways visible in the backyard?	Yes, No
	3.Are there any plants, trees, or shrubs in the backyard?	Yes, No
	4. Are there any structures (e.g., a shed, playhouse) in the backyard?	Yes, No
	5. Are there any visible recreational items (e.g., a swing set, trampoline, basketball hoop) in the backyard?	Yes, No
	6. Is a body of water (e.g., a pool, pond, or fountain) visible in the backyard?	Yes, No

Appendix

Table 10: Entity-wise questions and their corresponding answer lists.

Scene	Question	Answer List
Indoor	1. Which main elements are visible in the background?	Walls, Windows, Furniture, Appliances (e.g., fridge, microwave, washing machine), Electronic equipment (e.g., tvs, computers, speakers), Plain / solid color background
	2. What type of floor or ground is visible?	Tiled floor, Wooden floor, Carpeted floor, Concrete floor
	3. What type of environment is visible?	Residential, Commercial / public, Plain / solid color background
	4. What best describes the visual order in this image?	Organized (several elements present, but neat, intentional arrangement), Cluttered (many elements, visually noisy, no clear order), Minimalist (very few or no elements at all, mostly empty or plain)
Outdoor	1. What natural features, if any, are visible in the background of the image?	Trees / forest / plants, Mountains / hills, Waterbody, Open ground / fields
	2. What type of modern infrastructure is visible in the background?	Transport-related (paved roads, vehicles, bridges, rail tracks), Utility-related (electric poles, wires, water tanks, pipelines), High-rise / industrial (skyscrapers, factories, construction sites, large machinery)

Appendix

Table 11: Questions and their corresponding answer lists for Background Diversity Scores.

Figure 15: Sample questions for the SEVI dimensions, including a question on measuring Cultural Localization for a given image. For each image-question pair, the scales for each of these dimensions are defined.

Figure 17: Sample questions for the VDI-based VQA model Validation. Along with the VDI questions (one entity and background question for each image), we also ask the users about the Realism of the given image, as well as their confidence in answering the question.

Country	Qwen2.5-VL		llava-v1.6		Gemini-2.5
Country	Affluence	Maintenance	Affluence	Maintenance	Affluence	Maintenance
India	0.87	0.72	0.84	0.81	0.89	0.80
China	0.76	0.78	0.76	0.75	0.88	0.80
USA	0.72	0.67	0.53	0.79	0.69	0.62
Colombia	0.84	0.85	0.84	0.74	0.88	0.81
Egypt	0.66	0.86	0.73	0.76	0.62	0.58

Appendix

Table 12: Country-wise Spearman’s Correlation Coefficient between human and model ratings for SEVI dimensions . Gemini-2.5 outperforms the open-source variants.

Dataset	Entity Name	Affluence	Maintenance	Background	Entity
FLUX.1	Bag	0.924	0.316	0.412	0.921
	Chair	0.926	0.979	0.978	0.986
	Cooking pot	0.923	0.982	0.889	0.985
	House	0.796	0.819	0.885	0.875
	Plate of food	0.813	0.532	0.727	0.697
	Storefront	0.955	0.836	0.910	0.887

Appendix

Table 13: Correlation between Flash and Qwen across all four axes of GeoDiv scores.

Entity	Dataset	Affluence (Orig)	Affluence (Pert)	Maintenance (Orig)	Maintenance (Pert)
house	SD2.1	0.36	0.36	0.41	0.39
house	FLUX.1	0.22	0.22	0.05	0.07
chair	SD2.1	0.41	0.42	0.60	0.57
chair	FLUX.1	0.63	0.62	0.20	0.23

Appendix

Table 14: Effect of VLM prompt perturbations on SEVI scores.

Figure 18: Effect of image budget on GeoDiv estimates. All axes show rapid convergence of diversity scores and stable model ranking, indicating statistical robustness of the metrics.

Figure 19: Left : Spearman rank correlation between country rankings obtained at each image budget and the 250-image baseline. Across all diversity axes the rankings converge quickly and remain stable once $\geq$ 100 images are used. Right : Country diversity rankings for the overall GeoDiv score across different image budgets (1 = highest diversity).

Figure 20: Qualitative examples of house images from Nigeria across datasets. GeoDE shows balanced rural, suburban and urban scenes, while SD2.1 and SD3m show strong rural bias and FLUX.1 shows suburban bias. Each column shares the same generation seed across synthetic models for controlled comparison.

Figure 21: Comparison of car images for Indonesia across datasets. Rows: GeoDE (Entity diversity = $0.49$ ), SD2.1 ( $0.85$ ), SD3m ( $0.71$ ), FLUX.1 ( $0.54$ ). SD2.1 shows highest entity diversity with varied car types and contexts; FLUX.1 skews toward affluent suburban scenes. Indonesia shows the highest cross-country variance (0.03) for the car entity.

Figure 22: Comparison of house images generated by FLUX.1 across countries.

Figure 23: Dataset Samples from the FLUX.1 model. across $6$ countries and $10$ entities. We note distinct country-wise features for each image.

Figure 24: Dataset Samples from the SD2.1 model. across $6$ countries and $10$ entities. We note distinct country-wise features for each image.

Figure 25: Dataset Samples from the SD3m model. across $6$ countries and $10$ entities. We note distinct country-wise features for each image.

Figure 26: Dataset Samples from the SD3.5 model. across $6$ countries and $10$ entities. We note distinct country-wise features for each image.

Model_name	Entity-Div	Background-Div	Affluence-Div	Maintenance-Div
FLUX.1	$0.59$	$0.03$	$0.11$	$0.20$
SD2.1	$0.63$	$0.42$	$0.41$	$0.14$
SD3m	$0.61$	$0.31$	$0.45$	$-0.01$
SD3.5	$0.43$	$0.18$	$0.51$	$-0.09$

Appendix

Table 15: Pearson’s Correlation Coefficient (

\mathbf{\rho}

) between Vendi-Score and a) Entity Diversity (Entity-Div), b) Background Diversity (Background-Div), c) Affluence Diversity (Affluence-Div) and d) Maintenance Diversity (Maintenance-Div). Correlations across datasets is very weak, showing that the VDI scores capture features beyond visual diversity.

Figure 27: Assessing Cultural Localization. In general, we find India, Mexico and Japan to have more culturally localized images per model (including the real-world GeoDE dataset), with SD2.1 (SDv2) achieving higher scores than SD3m (SDv3) and FLUX.1.

T2I Generation Evaluation

Text-to-image generation evaluation measures how accurately, safely, and consistently image models turn prompts into images.

See all 162 papers

Apr 28, 2026

Assessing the Geographic Diversity of AI's Platial Representations in Image Generation

(Gen)AI diversity is not merely an ethical issue. From the perspective of geographic information science (GIScience), it could be interpreted as a function of uncertainty and as a form of cognitive bias, embedded in AI outputs. Recent work has sought to develop information-theoretic diversity measures and apply them to evaluate AI-chatbot outputs in a geographic context. As the AI ecosystem to which we are exposed on a daily basis becomes rapidly multimodal, we believe it is important to examine geographic diversity across various modalities. Focusing on images, this paper aims to fill this research gap. First, we select the GPT and DALL-E models as state-of-the-art examples and point out how assessing their geographic diversity involves various stages, including prompt revision and image generation. Then, taking inspiration from species diversity measures in ecological research, we incorporate similarity weighting into the measurement of geographic diversity. Next, we demonstrate how to evaluate geographic diversity in image generation through a case study. Our analysis reveals several counterintuitive findings. For instance, older models can exhibit greater geographic diversity despite producing lower-quality images, and prompt revision yields greater geographic diversity than image generation. At the same time, we observe explicit model homogeneity underlying the lack of geographic diversity, as the selected models consistently depict the same prototypical geo-specific feature or similar features. This is concerning, as it risks producing stereotypical representations of places.

Zilong Liu, Krzysztof Janowicz, Mina Karimi

Department of Geography and Regional Research, University of Vienna, Austria

Image Generation Evaluation T2I Generation Evaluation

Ask PDF

Jun 20, 2025

AI's Blind Spots: Geographic Knowledge and Diversity Deficit in Generated Urban Scenario

Diffusion-based text-to-image models are increasingly used for urban analysis and scenario generation, but their geographic knowledge and representational biases remain poorly understood. We evaluate FLUX 1-schnell and Stable Diffusion 3.5-Large in the United States by generating 150 street-view images for each state, each state capital, and a generic "USA" prompt. Images are embedded with DINO-v2 ViT-S/14 and compared with Fréchet Inception Distance (FID). Pairwise FID clustering shows that geographically proximate states and capitals often group together, indicating implicit geographic structure. However, the generic ``USA'' prompt collapses this diversity into a metropolitan stereotype: frontier, desert, tropical, rural, and small-city environments are underrepresented or distant in FID space. These results show that diffusion models can encode fine-grained geography while still reproducing narrow national-scale visual stereotypes.

Ciro Beneduce, Massimiliano Luca, Bruno Lepri

* Bruno Kessler Foundation · **University of Trento

T2I Generation T2I Generation Evaluation T2I Diffusion Models

Ask PDF

Apr 14, 2026

T2I-BiasBench: A Multi-Metric Framework for Auditing Demographic and Cultural Bias in Text-to-Image Models

Text-to-image (T2I) generative models achieve impressive visual fidelity but inherit and amplify demographic imbalances and cultural biases embedded in training data. We introduce T2I-BiasBench, a unified evaluation framework of thirteen complementary metrics that jointly captures demographic bias, element omission, and cultural collapse in diffusion models - the first framework to address all three dimensions simultaneously. We evaluate three open-source models - Stable Diffusion v1.5, BK-SDM Base, and Koala Lightning - against Gemini 2.5 Flash (RLHF-aligned) as a reference baseline. The benchmark comprises 1,574 generated images across five structured prompt categories. T2I-BiasBench integrates six established metrics with seven additional measures: four newly proposed (Composite Bias Score, Grounded Missing Rate, Implicit Element Missing Rate, Cultural Accuracy Ratio) and three adapted (Hallucination Score, Vendi Score, CLIP Proxy Score). Three key findings emerge: (1) Stable Diffusion v1.5 and BK-SDM exhibit bias amplification (>1.0) in beauty-related prompts; (2) contextual constraints such as surgical PPE substantially attenuate professional-role gender bias (Doctor CBS = 0.06 for SD v1.5); and (3) all models, including RLHF-aligned Gemini, collapse to a narrow set of cultural representations (CAS: 0.54-1.00), confirming that alignment techniques do not resolve cultural coverage gaps. T2I-BiasBench is publicly released to support standardized, fine-grained bias evaluation of generative models. The project page is available at: https://gyanendrachaubey.github.io/T2I-BiasBench/

Nihal Jaiswal, Siddhartha Arjaria, Gyanendra Chaubey +3

Department of Information Technology, Rajkiya Engineering College Banda, 210201, Uttar Pradesh, India. · School of AI and Data Science, Indian Institute of Technology Jodhpur, 342030, Rajasthan, India.

Demographic Bias in T2I Models T2I Generation Evaluation T2I Diffusion Models

Ask PDF

Abstract

T2I Generation Evaluation

Explore similar work

Assessing the Geographic Diversity of AI's Platial Representations in Image Generation

AI's Blind Spots: Geographic Knowledge and Diversity Deficit in Generated Urban Scenario

T2I-BiasBench: A Multi-Metric Framework for Auditing Demographic and Cultural Bias in Text-to-Image Models