cs.CYFeb 12, 2026

Artificial intelligence is creating a new global linguistic hierarchy

Authors: Giulia Occhini, Kumiko Tanaka-Ishii, Anna Barford, Refael Tikochinski, Songbo Hu, Roi Reichart, Yijie Zhou, ester James V. Miranda, 

Organizations: University of Cambridge, UK. · Waseda University, Japan. · University College London, UK. · Technion, Israel.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Artificial intelligence (AI) has the potential to transform healthcare, education, governance and socioeconomic equity, but its benefits remain concentrated in a small number of languages (Bender, 2019; Blasi et al., 2022; Joshi et al., 2020; Ranathunga and de Silva, 2022; Young, 2015). Language AI - the technologies that underpin widely-used conversational systems such as ChatGPT - could provide major benefits if available in people's native languages, yet most of the world's 7,000+ linguistic communities currently lack access and face persistent digital marginalization. Here we present a global longitudinal analysis of social, economic and infrastructural conditions across languages to assess systemic inequalities in language AI. We first analyze the existence of AI resources for 6003 languages. We find that despite efforts of the community to broaden the reach of language technologies (Bapna et al., 2022; Costa-Jussà et al., 2022), the dominance of a handful of languages is exacerbating disparities on an unprecedented scale, with divides widening exponentially rather than narrowing. Further, we contrast the longitudinal diffusion of AI with that of earlier IT technologies, revealing a distinctive hype-driven pattern of spread. To translate our findings into practical insights and guide prioritization efforts, we introduce the Language AI Readiness Index (EQUATE), which maps the state of technological, socio-economic, and infrastructural prerequisites for AI deployment across languages. The index highlights communities where capacity exists but remains underutilized, and provides a framework for accelerating more equitable diffusion of language AI. Our work contributes to setting the baseline for a transition towards more sustainable and equitable language technologies.

Figures & tables

Figure 1: Evolution of Language Model and Dataset Distribution on Hugging Face over time (2020-2024) based on Wayback Machine data (Source: ( 54 ) ) . These log-log plots illustrate power-law fits to the 2025 snapshot, showing the frequency of models (left) and datasets (right) per language rank over time. Models follow a broken power law: a shallower head ( α=0.89\alpha=0.89 ) steepens to a heavier tail ( α=1.69\alpha=1.69 ) at a crossover near rank 116. We observe how the distribution of resources per language reached a Zipf distribution in record time. English (red dot) is a major outlier, with an order of magnitude more models than expected, indicating extreme dominance. Other high-resource languages like French, Spanish, and Chinese follow the Zipf trend more closely.
Figure 2: Global Distribution of Languages by Language Models and Speaker Population . We plot the relationship between number of speakers and number of online language models on a log-log scale. Languages are grouped into four categories: “Mid-tier” (blue), “Dead” (black), “Under-resourced” (orange), and “Top 5 Languages per Speaker Bin” (green). By “Top 5 Languages per Speaker Bin” we refer to the five languages with the most language models within each speaker population range. A purple OLS regression line indicates the expected trend (with parameters β1=0.312, p<0.001\beta_{1}=0.312,\ p<0.001 , R2=0.304R^{2}=0.304 ), with languages falling below it, despite having over 1 million speakers, classified as under-resourced. Each group is plotted on world maps showing their geographic distribution, with expanded subplots in larger size available in the Supplementary Figures in 8.5 .
Figure 3: Linearized Gompertz transformation of adoption data for Language Models, Mobile Phones, Fixed Broadband, Electric Vehicles and Personal Computers. Growth for each technology progresses from the bottom-left (early adoption) towards the upper-right (approaching saturation). We normalise both adoption values (to maximum) and time (Z-score) to enable a cross-technology comparison.
Figure 4: Bar plots displaying the magnitude and direction of the component loadings for 24 variables on the first two principal components (PC1 and PC2) after Varimax rotation. PC1 (top panel) shows a strong positive contribution from socioeconomic and digital infrastructure variables, with the highest loading strengths observed for the proportion of individuals using the internet (0.35), the Human Development Index (HDI; 0.35), and education level (0.33). Network latency provides the sole significant negative loading ( −0.18-0.18 ). PC2 (bottom panel) is primarily defined by the availability of AI resources, with the highest loading strengths coming from Wikipedia active users (0.35), the number of datasets (0.35), and CommonCrawl data volume (0.35). The clear separation of variable groups across the two components graphically confirms the distinct factors identified by the PCA.
Figure 5: Demonstration of our open access AI language readiness tool. In subfigure a , we show the differential in distribution between languages having AI resources and languages being socio-technically ready for AI. We observe how while many languages might have a fraction of the language modeling resources available compared to English, they still do have the readiness in terms of socioeconomics and digital infrastructure. Below, in subfigure b we show the top ranking languages in our index ranked by overall score, available at https://www.equate-index.ai/ . In subfigure c we show the global explorer feature of our index. Blue circles indicate language clusters, with the number inside showing the count of languages in each region. The markers are interactive, allowing zoom and click to reveal individual languages and their AI readiness metrics. Filters on the left enable customization by overall ranking and minimum speaker count.
Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

CategoryVariableSource
AI ResourcesNumber of SpeakersWikidata https://www.wikidata.org/wiki/Help:Wikimedia_language_codes/lists/all
 Number of ModelsHuggingFace https://huggingface.co/languages
 Bible Translationshttps://www.bible.com/languages
 CommonCrawl Data VolumeCommonCrawl https://commoncrawl.github.io/cc-crawl-statistics/
 OPUS Data VolumeOPUS https://opus.nlpl.eu/
 Wikipedia Data VolumeWikipedia dumps
Appendix
Table 1: Summary of Extracted Variables, Their Sources and the literature behind them
Figure 6: Kernel Density Estimate (KDE) plot showing the frequency density of language proportions for two distinct datasets. The Hugging Face 2020 Models (blue curve) represent the distribution of models available on the platform, while ACL Papers 2004-2020 (orange curve) shows the distribution of languages featured in the ACL anthology. The x-axis indicates the proportion of total language count, and the y-axis represents the frequency density.
Figure 7: Logistic fit of benchmark availability against language rank by model count. Each curve shows the modeled probability that a language has at least nn associated benchmarks ( n=1n=1 – 44 , light to dark orange), as a function of its rank in the model-count distribution ( xx -axis, log scale, highest to lowest). Curves are logistic regressions of benchmark presence on log-rank; points show the empirical proportion within logarithmically spaced rank bins. The probability of having any benchmark ( ≥1\geq 1 ) falls below 50% at rank 196 (dashed line), and the probability of having multiple benchmarks declines progressively faster, indicating that both benchmark availability and benchmark variety are concentrated among the most model-rich languages.
Figure 8: Resource concentration among languages persists when multilingual artefacts are excluded. Rank–frequency distributions of Hugging Face artefacts per language, for models (left) and datasets (right), counting all artefacts (blue) and counting only those declared monolingual (orange). Both axes are logarithmic. Dashed lines are least-squares fits to the rank–frequency relation, with the fitted exponent α\alpha .
YearModels ( σe\sigma_{e} )Datasets ( σe\sigma_{e} )
20200.4170.365
20210.5490.716
20220.8780.743
20231.1300.801
20240.6680.133
Appendix
Table 2: Standard deviation of residuals ( σe\sigma_{e} ) from α=1\alpha=1 Zipf’s law fit per year for language models and datasets.
Top kk removednnSlopeR2R^{2}Pearson rrpp
017320.3120.3040.5512.6×10−1382.6\times 10^{-138}
517270.2990.2860.5353.3×10−1283.3\times 10^{-128}
617260.2970.2830.5321.1×10−1261.1\times 10^{-126}
717250.2960.2800.5302.6×10−1252.6\times 10^{-125}
1017220.2890.2720.5229.9×10−1219.9\times 10^{-121}
1517170.2880.2680.5174.4×10−1184.4\times 10^{-118}
Appendix
Table 3: Sensitivity of the speaker–model correlation to exclusion of the most-spoken languages. Log–log OLS refit of the number of models against speaker population after removing the top kk languages by speaker population. The correlation declines only modestly and remains moderate and highly significant throughout, indicating that the relationship is not an artefact of the most-resourced languages.
Figure 9: The vast majority of dead languages with models and datasets are located in Europe.
Figure 10: The majority of overresourced languages with models and datasets are located in Europe.
Figure 11: The vast majority of underresourced languages with models and datasets are located in Africa.
Figure 12: Distribution of mid-tier languages, the intermediate group with model presence, comprising those not classified as dead, under-resourced, or among the top five per speaker bin. They span a wide range of speaker populations and are distributed across all inhabited continents.
Figure 13: Comparative visualization of language technology development across the United States, Australia, and Nigeria reveals an unexpected pattern. Nigeria, a comparatively less technologically developed country, has a higher proportion of languages covered by at least one language model (14.7%) than both the United States (14%) and Australia (6.8%). Paradoxically, the region with the largest number of languages left behind in the United States is California—the most technologically advanced area in the world.
Figure 14: The Gompertz model fitted to the adoption rate of various technologies. The left panel shows the displacement parameter ( bb ), which indicates the horizontal shift of the growth curve. The right panel displays the growth rate constant ( cc ), which represents the steepness and speed of adoption. Language Models exhibits the highest growth rate and the lowest displacement, indicating a very rapid, early-stage adoption trajectory.
Figure 15: Heatmap illustrating the pairwise Pearson correlation coefficients between 24 variables, ordered to display the distinct clustering of socioeconomic and digital infrastructure indicators on the one hand, and AI resources indicators on the other. The color scale, ranging from deep blue (strong negative correlation) to deep red (strong positive correlation), reveals two highly correlated internal blocks: one comprising AI resource availability indicators (e.g., cc_gb, datasets_n) and the other comprising socioeconomic and digital infrastructure indicators (e.g., admin_1_hdi, literacy_rate, avg_download_speed_kbps). Crucially, the coefficients across these two blocks are predominantly weak (near zero, white/light grey), quantitatively showcasing the structural independence of general societal development from the specific availability of AI-relevant resources.
Figure 16: Heatmap illustrating the rotated component loadings for 24 variables on the first two principal components (PC1 and PC2) following Varimax rotation. PC1 is defined by high positive loadings from social and infrastructural indicators, including the proportion of individuals using the internet (loading =0.35), Human Development Index (HDI; loading =0.35), and education level (loading =0.33). Conversely, PC2 is strongly characterized by high positive loadings for AI technology availability, with the highest values observed for the volume of data on CommonCrawl (loading =0.37), the number of datasets on HuggingFace (loading =0.37), and Wikipedia active users (loading =0.37). The color intensity reflects the magnitude of the loading, ranging from a strong negative correlation (deep blue, e.g., avg_latency_ms on PC1) to a strong positive correlation (deep red). Together, these two orthogonal components explain 58.4% of the total variance, revealing a clean separation between foundational societal development and the specific availability of AI-focused resources.
Figure 17: Standardized Regression Coefficients ( β\beta ) from Stepwise Model Selection. Bar plot displaying the standardized regression coefficients ( β\beta ) and 95%95\% confidence intervals (CI) for predictors retained in a final model after stepwise regression. Coefficients colored light blue are statistically significant ( p<0.05p<0.05 ), indicating a reliable positive relationship with the dependent variable. Significant positive predictors include the number of speakers ( n_speakersn\_speakers ), the volume of Opus data ( opus_gbopus\_gb ), the volume of Xeus data ( xeus_gbxeus\_gb ), and the strongest predictor by magnitude, the presence of a Bible translation ( bible_exists1bible\_exists1 ). Variables in light gray (e.g., cc_gbcc\_gb , models_mean_downloadsmodels\_mean\_downloads ) were retained but found to be statistically non-significant ( p≥0.05p\geq 0.05 ), as indicated by the 95%95\% CI overlapping the zero line (dashed blue). The standardized nature of the coefficients allows for a direct comparison of the relative predictive strength of each significant variable.
Institution/OrganizationCountryRoleCount
LMU MunichGermanyPhD Student5
Uppsala UniversitySwedenProfessor1
University of CambridgeUnited KingdomPostdoc1
Würzburg UniversityGermanyProfessor1
McGill UniversityCanadaProfessor1
Google DeepMindUnited KingdomResearch Scientist1
Appendix
Table 4: Affiliations and roles of participants in Survey 1, the ranking of AI resources. Respondents were experts in multilingual language technology (n = 28).
Institution/OrganizationCountryRoleCount
University of CambridgeUnited KingdomPostdoctoral Researcher2
University of BristolUnited KingdomProfessor2
Cohere For AICanada & USAResearch Scientist2
Stanford UniversityUSAPhD Student1
University of WashingtonUSAPhD Student1
George Mason UniversityUSAProfessor1
Appendix
Table 5: Affiliations and roles of participants in Survey 2, the ranking of digital-infrastructure and socioeconomic factors. Respondents were experts in human–computer interaction and global technology adoption (n = 20).
RankResource
1Amount of data in Common Crawl (bytes)
2Amount of data in Wikipedia Articles (bytes)
3Number of already available multilingual Language Models
4Typological similarity to well resourced language families
5Amount of data in OPUS (bytes)
6Number of already available monolingual Language Models
Appendix
Table 6: Ranked AI resources in order of importance for building a language technology system.
RankResource
1Percentage of households with phones
2Percentage of individuals with home internet
3Average network speed (upload, download, latency)
4Percentage of households with home internet access
5Existence of a nearby university or technical institution
6Availability of local cloud computing infrastructure (e.g., data centers, regional hosting services)
Appendix
Table 7: Ranked digital infrastructure factors in order of importance for bridging digital divides that affect the possibility of development and long-term sustainability of language technologies in under-resourced communities.
RankResource
1Language community’s Institutional Status
2Number of Speakers
3Language community’s Vitality Status
4Online Active Wikipedia Users (Digital Vitality)
5Literacy Rate
6Gross Domestic Product
Appendix
Table 8: Ranked socioeconomic factors in order of importance for bridging digital divides that affect the possibility of development and long-term sustainability of language technologies in under-resourced communities.
More in this topic

Low-Resource Language Processing

Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.

See all 565 papers

Explore similar work

CardsList
May 29, 2026

How Early Adopters Used Generative AI Worldwide: Variation by Country Income and Language

AI is being used by people globally, but not everyone is using it in the same ways. Using a large-scale dataset of anonymized, de-identified, and privacy-scrubbed interactions with a widely available and free AI chatbot, we empirically characterize differences in early adopters' usage across countries. Schooling is the most common domain of use in most countries, particularly low-income countries, with a strong inverse association evident between schooling and country-level GDP. Leisure-related use, by contrast, is positively associated with country-level income. Language, we find, also shapes use: English-language interactions are overrepresented in places where the predominant languages were not well-served by existing models during the period of the study. Improving performance across languages may be a key factor, our work suggests, in whether this technology expands digital divides or enables leapfrogging.

Madeleine I. G. Daepp, Isaac Slaughter

Microsoft Research · University of Washington

Generative AI Adoption
AskPDF
Aug 12, 2026

Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages

Artificial intelligence tools for education and language support are increasingly framed as scalable responses to access gaps in under-resourced communities. Yet the infrastructure underlying these tools, including training corpora, tokenization schemes, evaluation benchmarks, and deployment architectures, can systematically disadvantage speakers of underrepresented languages before a model is trained. This paper examines these structural barriers through Bengali, one of the world's most widely spoken languages, focusing on AI-assisted education in low-connectivity environments. We identify four interlocking failures: a severe web presence gap, with Bengali accounting for less than 0.5% of global web content despite representing nearly 4% of the global population; a 67:1 training-token deficit between English and Bengali in major multilingual corpora; a tokenization penalty associated with Bengali's alphasyllabary script that compounds the data deficit through higher token fertility; and connectivity exclusion, with individual internet penetration at 36.5% in rural areas compared with 71.4% in urban areas. These failures reflect longstanding resource-allocation decisions, institutional priorities, and design defaults that did not center underrepresented languages in mainstream AI development. We argue that dataset scarcity should be understood as a structural barrier rather than an isolated technical limitation, and that offline-first design should be treated as an equity-oriented infrastructure strategy. We conclude with directions for linguistics and AI research aimed at reducing these structural inequalities.

Avijit Roy, Proma Roy

John Jay College of Criminal Justice, CUNY · The City College of New York, CUNY

Multilingual Language ModelsTokenization in Language ModelsAI in EducationLow-Resource Language Processing
AskPDF
May 2, 2026

Artificial intelligence language technologies in multilingual healthcare: Grand challenges ahead

AI language technologies (AILTs), increasingly enabled by large language models (LLMs), are becoming embedded in multilingual healthcare workflows for translation, rewriting, documentation, interpreting, and messaging in language-discordant settings. Yet fluent output is not the same as clinically safe or equitable communication: performance varies across languages, accents, tasks, and workflows, and efficiency gains can hide errors, reduce traceability, and shift responsibility across clinicians, translators, interpreters, and health systems. This narrative review synthesises recent peer-reviewed evidence across written communication, spoken communication, and emerging agentic workflows. Using the Human-Centered AI Language Technology (HCAILT) lens, it examines capabilities, evaluation practices, implementation patterns, and recurrent errors through reliability, safety culture, and trustworthiness. We identify key convergences and contradictions in the literature and propose seven grand challenges for the next phase of research and deployment. Progress, we argue, requires not only better models but also accountable sociotechnical design, calibrated human oversight, and stronger collaboration across MT/NLP, translation studies, HCI, clinical practice, implementation science, and policy.

Vicent Briva-Iglesias

School of Applied Languages and Intercultural Studies (SALIS) · CTTS, ADAPT Centre · Dublin City University

Multilingual Language Model EvaluationLanguage Model Safety EvaluationHealthcareClinical Language Model EvaluationClinical NLP
AskPDF