Evaluating Geographic Lexical Knowledge in Large Language Models Across 21 Spanish-Speaking Countries

Yoshifumi Kawasaki

cs.CLFeb 10, 2026

Evaluating Geographic Lexical Knowledge in Large Language Models Across 21 Spanish-Speaking Countries

Authors: Yoshifumi Kawasaki

Organizations: University of Tokyo

Abstract

Large language models (LLMs) increasingly serve users across regional varieties, yet their knowledge of geographic lexical variation remains insufficiently evaluated. We investigate country-conditioned lexical usage judgments in Spanish using an expert-curated database covering 934 lexical items across 21 Spanish-speaking countries. Treating ten proprietary and open-weight LLMs as virtual informants, we evaluate two complementary question formats: Yes-No question format (YNQF) and multiple-choice question format (MCQF). The evaluation comprises 10,164 YNQF questions and 16,637 MCQF questions, assessed using F1 and an adjusted Jaccard coefficient, respectively. Results reveal substantial differences across models, countries, and formats. Spain receives the highest country-level score for eight models in YNQF and nine in MCQF; Chile receives the lowest MCQF score for every model. Regression analyses suggest that digital resource volume is generally more strongly associated with YNQF performance, whereas lexical distance from Spain is more strongly associated with MCQF performance. These findings provide a fine-grained diagnostic of uneven regional lexical knowledge and highlight the importance of complementary elicitation formats and evaluation metrics. The benchmark measures explicit usage judgments, motivating separate assessment of region-appropriate generation.

Figures & tables

Figure 1: Schematic overview of the study. The questions are translated into English for clarity.

Item index	A141
Description	vehículo destinado al transporte de personas “ vehicle intended for the transport of people ”
English equivalent	CAR
Variants	auto , automóvil , carro , coche , concho , máquina

Table 1: Item A141 referring to car .

Model	Valid ratio	Accuracy	Precision	Recall	$F1$
Baseline	1.000	0.500	0.500	0.500	0.500
Claude Haiku 4.5	1.000	0.506	0.864	0.015	0.029
Gemini 3.5 Flash Lite	1.000	0.667	0.640	0.763	0.696
GPT-4o	1.000	0.700	0.718	0.659	0.687
Gemma 4 12B	1.000	0.522	0.776	0.063	0.116
Gemma 4 31B	1.000	0.656	0.671	0.612	0.640

Table 2: Overall results for the Yes–No question format (YNQF). Valid ratio denotes the proportion of valid responses.

Figure 2: Score Transition Across Models by Country (YNQF).

Model	Valid ratio	$J$	$J_{adj}$
Baseline	1.000	0.215	0.137
Claude Haiku 4.5	1.000	0.393	0.259
Gemini 3.5 Flash Lite	1.000	0.430	0.334
GPT-4o	1.000	0.467	0.346
Gemma 4 12B	1.000	0.410	0.269
Gemma 4 31B	1.000	0.445	0.325

Table 3: Overall results for the multiple-choice question format (MCQF). Valid ratio denotes the proportion of valid responses.

J

represents the original Jaccard scores.

Figure 3: Score Transition Across Models by Country (MCQF).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Full identifier
Proprietary models
Claude Haiku 4.5	claude-haiku-4-5-20251001
Gemini 3.5 Flash Lite	gemini-3.5-flash-lite
GPT-4o	gpt-4o
Open-weight models
Gemma 4 12B	google/gemma-4-12B

Appendix

Table 4: Abbreviated model names and full identifiers.

Country	Claude Haiku 4.5	Gemini 3.5 Flash Lite	GPT-4o	Gemma 4 12B	Gemma 4 31B	Llama 3.1 8B	Ministral 3 8B	Mistral Small 24B	Qwen2.5 7B	Qwen3 30B-A3B
Spain	0.350	0.778	0.799	0.552	0.774	0.652	0.696	0.330	0.463	0.700
Equatorial Guinea	0.000	0.711	0.722	0.000	0.422	0.206	0.641	0.000	0.032	0.528
Cuba	0.000	0.668	0.681	0.000	0.480	0.523	0.589	0.024	0.139	0.608
Dominican Republic	0.000	0.685	0.700	0.016	0.668	0.642	0.665	0.056	0.362	0.661
Puerto Rico	0.000	0.655	0.612	0.024	0.613	0.601	0.669	0.008	0.192	0.601
Mexico	0.040	0.743	0.739	0.258	0.711	0.637	0.674	0.152	0.511	0.639

Appendix

Table 5: Country-level

F1

scores in the Yes–No question format (YNQF).

Country	Claude Haiku 4.5	Gemini 3.5 Flash Lite	GPT-4o	Gemma 4 12B	Gemma 4 31B	Llama 3.1 8B	Ministral 3 8B	Mistral Small 24B	Qwen2.5 7B	Qwen3 30B-A3B
Spain	0.405	0.503	0.498	0.402	0.494	0.266	0.321	0.363	0.246	0.302
Equatorial Guinea	0.263	0.377	0.377	0.273	0.373	0.276	0.225	0.350	0.219	0.218
Cuba	0.233	0.269	0.323	0.254	0.272	0.205	0.182	0.201	0.193	0.234
Dominican Republic	0.242	0.307	0.333	0.258	0.303	0.233	0.214	0.281	0.230	0.221
Puerto Rico	0.235	0.264	0.336	0.246	0.270	0.205	0.171	0.197	0.204	0.219
Mexico	0.298	0.388	0.390	0.316	0.385	0.190	0.225	0.206	0.204	0.276

Appendix

Table 6: Country-level

J_{adj}

scores in the multiple-choice question format (MCQF).

Model	$\rho$	$p$	$\rho$	$p$
	Original Jaccard		Adjusted Jaccard $J_{adj}$
GPT-4o	0.656	0.001	0.077	0.739
Gemini 3.5 Flash Lite	0.064	0.784	-0.279	0.220
Claude Haiku 4.5	0.787	$<0.001$	-0.086	0.710
Llama 3.1 8B	0.640	0.002	-0.679	0.001
Gemma 4 12B	0.809	$<0.001$	0.261	0.253

Appendix

Table 7: Spearman’s rank correlations

\rho

between the performance and baseline under original and adjusted Jaccard scoring for MCQF.

p

-values are uncorrected for multiple comparisons.

Country	Jaccard distance from Spain
España	0.000
Guinea Ecuatorial	0.668
Cuba	0.584
República Dominicana	0.697
Puerto Rico	0.627
México	0.640

Appendix

Table 8: Jaccard distances from Spain (YNQF).

Country	Jaccard distance from Spain
España	0.000
Cuba	0.545
Guinea Ecuatorial	0.554
Argentina	0.579
Puerto Rico	0.582
México	0.600

Appendix

Table 9: Jaccard distances from Spain (MCQF).

Country	Number of tokens
Spain	48,821,356,834
Argentina	12,801,219,868
Mexico	10,829,453,655
Chile	6,589,894,993
Colombia	4,873,779,254
Peru	2,868,606,294

Appendix

Table 10: Digital resource volume by country, measured in tokens and sorted in descending order.

Figure 4: Inter-model correlation computed from their country-level performance scores. YN and MC represents YNQF and MCQF, respectively.

Figure 5: Standardized Regression Coefficients by Model (YNQF). An asterisk on a bar indicates a significant predictor, while an asterisk preceding a model name indicates a significant F-statistic.

Figure 6: Standardized Regression Coefficients by Model (MCQF). An asterisk on a bar indicates a significant predictor, while an asterisk preceding a model name indicates a significant F-statistic.

Figure 7: Model Performance vs. Adjusted R-squared (YNQF). Spearman’s rank correlation $\rho=-0.964$ ( $p$ < 0.001)).

Figure 8: Model Performance vs. Adjusted R-squared (MCQF). Spearman’s rank correlation $\rho=-0.309$ ( $p$ = 0.385)).

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

CARTE: A Benchmark for Mapping Language Model Knowledge Across France
Jun 1, 2026Sarah Almeida Carneiro, Christos Xypolopoulos, Xiao Fei +2LLM Evaluation Factual Knowledge in Language Models
X · X, NTUA · X, MBZUAI
Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America
Feb 16, 2026Yannis Karmim, Renato Pino, Hernan Contreras +6Cultural Bias in Language Models Cultural Knowledge in Language Models
ALMAnaCH · ALMAnaCH team, Inria Paris Center · Dept. of Computer Science, Universidad de Chile +8
MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning
Aug 4, 2026Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1Spatial Reasoning Multilingual QA
Data and Web Science Group, University of Mannheim, Germany

Jun 1, 2026J/K move · Enter open · S save

CARTE: A Benchmark for Mapping Language Model Knowledge Across France

Sarah Almeida Carneiro, Christos Xypolopoulos, Xiao Fei, Yang Zhang+1

X · X, NTUA · X, MBZUAI

LLM Evaluation Factual Knowledge in Language Models Cultural Knowledge in Language Models Language Model Robustness

Open paper AskPDF arXiv

We introduce CARTE (Culturally Anchored Regional-Territorial Evaluation), a multiple-choice benchmark for evaluating the ability of large language models (LLMs) to discriminate fine grained, geographically grounded and regionally differentiated knowledge within France. While prior benchmarks focus on national-level cultural understanding, they largely overlook intra-country variation and the need to distinguish between closely related regional contexts. CARTE addresses this gap by introducing 3,132 questions spanning the 13 metropolitan regions of France and covering 12 thematic domains, including culture, language, demographics, economy, environment, and mobility. Questions underwent four-stage validation: human review, LLM-as-a-judge, model-based difficulty calibration, and a second human review. We further introduce CARTE-LV, a subset targeting Linguistic Variation across French regions, enabling focused evaluation of language-related differences. We evaluate 32 open-weight LLMs of 1B to 30B parameters and three frontier API models across zero-, one-, and three-shot settings. Our experiments reveal performance disparities across regions and model scales, suggesting systematic gaps in pretraining coverage and limited robustness to intra-national variation.

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

CARTE: A Benchmark for Mapping Language Model Knowledge Across France

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning