Evaluating Geographic Lexical Knowledge in Large Language Models Across 21 Spanish-Speaking Countries

Yoshifumi Kawasaki

cs.CLFeb 10, 2026

Evaluating Geographic Lexical Knowledge in Large Language Models Across 21 Spanish-Speaking Countries

Authors: Yoshifumi Kawasaki

Organizations: University of Tokyo

Abstract

Large language models (LLMs) increasingly serve users across regional varieties, yet their knowledge of geographic lexical variation remains insufficiently evaluated. We investigate country-conditioned lexical usage judgments in Spanish using an expert-curated database covering 934 lexical items across 21 Spanish-speaking countries. Treating ten proprietary and open-weight LLMs as virtual informants, we evaluate two complementary question formats: Yes-No question format (YNQF) and multiple-choice question format (MCQF). The evaluation comprises 10,164 YNQF questions and 16,637 MCQF questions, assessed using F1 and an adjusted Jaccard coefficient, respectively. Results reveal substantial differences across models, countries, and formats. Spain receives the highest country-level score for eight models in YNQF and nine in MCQF; Chile receives the lowest MCQF score for every model. Regression analyses suggest that digital resource volume is generally more strongly associated with YNQF performance, whereas lexical distance from Spain is more strongly associated with MCQF performance. These findings provide a fine-grained diagnostic of uneven regional lexical knowledge and highlight the importance of complementary elicitation formats and evaluation metrics. The benchmark measures explicit usage judgments, motivating separate assessment of region-appropriate generation.

Figures & tables

Figure 1: Schematic overview of the study. The questions are translated into English for clarity.

Item index	A141
Description	vehículo destinado al transporte de personas “ vehicle intended for the transport of people ”
English equivalent	CAR
Variants	auto , automóvil , carro , coche , concho , máquina

Table 1: Item A141 referring to car .

Model	Valid ratio	Accuracy	Precision	Recall	$F1$
Baseline	1.000	0.500	0.500	0.500	0.500
Claude Haiku 4.5	1.000	0.506	0.864	0.015	0.029
Gemini 3.5 Flash Lite	1.000	0.667	0.640	0.763	0.696
GPT-4o	1.000	0.700	0.718	0.659	0.687
Gemma 4 12B	1.000	0.522	0.776	0.063	0.116
Gemma 4 31B	1.000	0.656	0.671	0.612	0.640

Table 2: Overall results for the Yes–No question format (YNQF). Valid ratio denotes the proportion of valid responses.

Figure 2: Score Transition Across Models by Country (YNQF).

Model	Valid ratio	$J$	$J_{adj}$
Baseline	1.000	0.215	0.137
Claude Haiku 4.5	1.000	0.393	0.259
Gemini 3.5 Flash Lite	1.000	0.430	0.334
GPT-4o	1.000	0.467	0.346
Gemma 4 12B	1.000	0.410	0.269
Gemma 4 31B	1.000	0.445	0.325

Table 3: Overall results for the multiple-choice question format (MCQF). Valid ratio denotes the proportion of valid responses.

J

represents the original Jaccard scores.

Figure 3: Score Transition Across Models by Country (MCQF).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Full identifier
Proprietary models
Claude Haiku 4.5	claude-haiku-4-5-20251001
Gemini 3.5 Flash Lite	gemini-3.5-flash-lite
GPT-4o	gpt-4o
Open-weight models
Gemma 4 12B	google/gemma-4-12B

Appendix

Table 4: Abbreviated model names and full identifiers.

Country	Claude Haiku 4.5	Gemini 3.5 Flash Lite	GPT-4o	Gemma 4 12B	Gemma 4 31B	Llama 3.1 8B	Ministral 3 8B	Mistral Small 24B	Qwen2.5 7B	Qwen3 30B-A3B
Spain	0.350	0.778	0.799	0.552	0.774	0.652	0.696	0.330	0.463	0.700
Equatorial Guinea	0.000	0.711	0.722	0.000	0.422	0.206	0.641	0.000	0.032	0.528
Cuba	0.000	0.668	0.681	0.000	0.480	0.523	0.589	0.024	0.139	0.608
Dominican Republic	0.000	0.685	0.700	0.016	0.668	0.642	0.665	0.056	0.362	0.661
Puerto Rico	0.000	0.655	0.612	0.024	0.613	0.601	0.669	0.008	0.192	0.601
Mexico	0.040	0.743	0.739	0.258	0.711	0.637	0.674	0.152	0.511	0.639

Appendix

Table 5: Country-level

F1

scores in the Yes–No question format (YNQF).

Country	Claude Haiku 4.5	Gemini 3.5 Flash Lite	GPT-4o	Gemma 4 12B	Gemma 4 31B	Llama 3.1 8B	Ministral 3 8B	Mistral Small 24B	Qwen2.5 7B	Qwen3 30B-A3B
Spain	0.405	0.503	0.498	0.402	0.494	0.266	0.321	0.363	0.246	0.302
Equatorial Guinea	0.263	0.377	0.377	0.273	0.373	0.276	0.225	0.350	0.219	0.218
Cuba	0.233	0.269	0.323	0.254	0.272	0.205	0.182	0.201	0.193	0.234
Dominican Republic	0.242	0.307	0.333	0.258	0.303	0.233	0.214	0.281	0.230	0.221
Puerto Rico	0.235	0.264	0.336	0.246	0.270	0.205	0.171	0.197	0.204	0.219
Mexico	0.298	0.388	0.390	0.316	0.385	0.190	0.225	0.206	0.204	0.276

Appendix

Table 6: Country-level

J_{adj}

scores in the multiple-choice question format (MCQF).

Model	$\rho$	$p$	$\rho$	$p$
	Original Jaccard		Adjusted Jaccard $J_{adj}$
GPT-4o	0.656	0.001	0.077	0.739
Gemini 3.5 Flash Lite	0.064	0.784	-0.279	0.220
Claude Haiku 4.5	0.787	$<0.001$	-0.086	0.710
Llama 3.1 8B	0.640	0.002	-0.679	0.001
Gemma 4 12B	0.809	$<0.001$	0.261	0.253

Appendix

Table 7: Spearman’s rank correlations

\rho

between the performance and baseline under original and adjusted Jaccard scoring for MCQF.

p

-values are uncorrected for multiple comparisons.

Country	Jaccard distance from Spain
España	0.000
Guinea Ecuatorial	0.668
Cuba	0.584
República Dominicana	0.697
Puerto Rico	0.627
México	0.640

Appendix

Table 8: Jaccard distances from Spain (YNQF).

Country	Jaccard distance from Spain
España	0.000
Cuba	0.545
Guinea Ecuatorial	0.554
Argentina	0.579
Puerto Rico	0.582
México	0.600

Appendix

Table 9: Jaccard distances from Spain (MCQF).

Country	Number of tokens
Spain	48,821,356,834
Argentina	12,801,219,868
Mexico	10,829,453,655
Chile	6,589,894,993
Colombia	4,873,779,254
Peru	2,868,606,294

Appendix

Table 10: Digital resource volume by country, measured in tokens and sorted in descending order.

Figure 4: Inter-model correlation computed from their country-level performance scores. YN and MC represents YNQF and MCQF, respectively.

Figure 5: Standardized Regression Coefficients by Model (YNQF). An asterisk on a bar indicates a significant predictor, while an asterisk preceding a model name indicates a significant F-statistic.

Figure 6: Standardized Regression Coefficients by Model (MCQF). An asterisk on a bar indicates a significant predictor, while an asterisk preceding a model name indicates a significant F-statistic.

Figure 7: Model Performance vs. Adjusted R-squared (YNQF). Spearman’s rank correlation $\rho=-0.964$ ( $p$ < 0.001)).

Figure 8: Model Performance vs. Adjusted R-squared (MCQF). Spearman’s rank correlation $\rho=-0.309$ ( $p$ = 0.385)).

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

Jun 1, 2026

CARTE: A Benchmark for Mapping Language Model Knowledge Across France

We introduce CARTE (Culturally Anchored Regional-Territorial Evaluation), a multiple-choice benchmark for evaluating the ability of large language models (LLMs) to discriminate fine grained, geographically grounded and regionally differentiated knowledge within France. While prior benchmarks focus on national-level cultural understanding, they largely overlook intra-country variation and the need to distinguish between closely related regional contexts. CARTE addresses this gap by introducing 3,132 questions spanning the 13 metropolitan regions of France and covering 12 thematic domains, including culture, language, demographics, economy, environment, and mobility. Questions underwent four-stage validation: human review, LLM-as-a-judge, model-based difficulty calibration, and a second human review. We further introduce CARTE-LV, a subset targeting Linguistic Variation across French regions, enabling focused evaluation of language-related differences. We evaluate 32 open-weight LLMs of 1B to 30B parameters and three frontier API models across zero-, one-, and three-shot settings. Our experiments reveal performance disparities across regions and model scales, suggesting systematic gaps in pretraining coverage and limited robustness to intra-national variation.

Sarah Almeida Carneiro, Christos Xypolopoulos, Xiao Fei +2

X · X, NTUA · X, MBZUAI

LLM Evaluation Factual Knowledge in Language Models Cultural Knowledge in Language Models Language Model Robustness

Ask PDF

Feb 16, 2026

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

Large Language Models (LLMs) exhibit inequalities with respect to various cultural contexts. Most prominent open-weights models are trained on Global North data and show prejudicial behavior towards other cultures. Moreover, there is a notable lack of resources to detect biases in non-English languages, especially from Latin America (Latam), a continent containing various cultures, even though they share a common cultural ground. We propose to leverage the content of Wikipedia, the structure of the Wikidata knowledge graph, and expert knowledge from social science in order to create a dataset of question/answer (Q/As) pairs, based on the different popular and social cultures of various Latin American countries. We create the LatamQA database of over 26k questions and associated answers extracted from 26k Wikipedia articles, and transformed into multiple-choice questions (MCQ) in Spanish and Portuguese, in turn translated to English. We use this MCQ to quantify the degree of knowledge of various LLMs and find out (i) a discrepancy in performances between the Latam countries, ones being easier than others for the majority of the models, (ii) that the models perform better in their original language, and (iii) that Iberian Spanish culture is better known than Latam one.

Yannis Karmim, Renato Pino, Hernan Contreras +6

ALMAnaCH · ALMAnaCH team, Inria Paris Center · Dept. of Computer Science, Universidad de Chile +8

Cultural Bias in Language Models Cultural Knowledge in Language Models Multilingual QA

Ask PDF

Aug 4, 2026

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and topological computation despite storing considerable geographic knowledge. Existing benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource languages. Across parametric, reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best. Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.

Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1

Data and Web Science Group, University of Mannheim, Germany

Spatial Reasoning Multilingual QA Multilingual Language Model Evaluation Spatial Reasoning Benchmarks Geospatial Reasoning Benchmarks Jurisdictional Bias in Language Models+3

Ask PDF

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

CARTE: A Benchmark for Mapping Language Model Knowledge Across France

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning