cs.CYFeb 12, 2026
SaveArtificial intelligence is creating a new global linguistic hierarchy
Organizations: University of Cambridge, UK. · Waseda University, Japan. · University College London, UK. · Technion, Israel.
Abstract
Artificial intelligence (AI) has the potential to transform healthcare, education, governance and socioeconomic equity, but its benefits remain concentrated in a small number of languages (Bender, 2019; Blasi et al., 2022; Joshi et al., 2020; Ranathunga and de Silva, 2022; Young, 2015). Language AI - the technologies that underpin widely-used conversational systems such as ChatGPT - could provide major benefits if available in people's native languages, yet most of the world's 7,000+ linguistic communities currently lack access and face persistent digital marginalization. Here we present a global longitudinal analysis of social, economic and infrastructural conditions across languages to assess systemic inequalities in language AI. We first analyze the existence of AI resources for 6003 languages. We find that despite efforts of the community to broaden the reach of language technologies (Bapna et al., 2022; Costa-Jussà et al., 2022), the dominance of a handful of languages is exacerbating disparities on an unprecedented scale, with divides widening exponentially rather than narrowing. Further, we contrast the longitudinal diffusion of AI with that of earlier IT technologies, revealing a distinctive hype-driven pattern of spread. To translate our findings into practical insights and guide prioritization efforts, we introduce the Language AI Readiness Index (EQUATE), which maps the state of technological, socio-economic, and infrastructural prerequisites for AI deployment across languages. The index highlights communities where capacity exists but remains underutilized, and provides a framework for accelerating more equitable diffusion of language AI. Our work contributes to setting the baseline for a transition towards more sustainable and equitable language technologies.
Figures & tables
Figure 1: Evolution of Language Model and Dataset Distribution on Hugging Face over time (2020-2024) based on Wayback Machine data (Source: ( 54 ) ) . These log-log plots illustrate power-law fits to the 2025 snapshot, showing the frequency of models (left) and datasets (right) per language rank over time. Models follow a broken power law: a shallower head ( ) steepens to a heavier tail ( ) at a crossover near rank 116. We observe how the distribution of resources per language reached a Zipf distribution in record time. English (red dot) is a major outlier, with an order of magnitude more models than expected, indicating extreme dominance. Other high-resource languages like French, Spanish, and Chinese follow the Zipf trend more closely.
Figure 2: Global Distribution of Languages by Language Models and Speaker Population . We plot the relationship between number of speakers and number of online language models on a log-log scale. Languages are grouped into four categories: “Mid-tier” (blue), “Dead” (black), “Under-resourced” (orange), and “Top 5 Languages per Speaker Bin” (green). By “Top 5 Languages per Speaker Bin” we refer to the five languages with the most language models within each speaker population range. A purple OLS regression line indicates the expected trend (with parameters , ), with languages falling below it, despite having over 1 million speakers, classified as under-resourced. Each group is plotted on world maps showing their geographic distribution, with expanded subplots in larger size available in the Supplementary Figures in 8.5 .
Figure 3: Linearized Gompertz transformation of adoption data for Language Models, Mobile Phones, Fixed Broadband, Electric Vehicles and Personal Computers. Growth for each technology progresses from the bottom-left (early adoption) towards the upper-right (approaching saturation). We normalise both adoption values (to maximum) and time (Z-score) to enable a cross-technology comparison.
Figure 4: Bar plots displaying the magnitude and direction of the component loadings for 24 variables on the first two principal components (PC1 and PC2) after Varimax rotation. PC1 (top panel) shows a strong positive contribution from socioeconomic and digital infrastructure variables, with the highest loading strengths observed for the proportion of individuals using the internet (0.35), the Human Development Index (HDI; 0.35), and education level (0.33). Network latency provides the sole significant negative loading ( ). PC2 (bottom panel) is primarily defined by the availability of AI resources, with the highest loading strengths coming from Wikipedia active users (0.35), the number of datasets (0.35), and CommonCrawl data volume (0.35). The clear separation of variable groups across the two components graphically confirms the distinct factors identified by the PCA.
Figure 5: Demonstration of our open access AI language readiness tool. In subfigure a , we show the differential in distribution between languages having AI resources and languages being socio-technically ready for AI. We observe how while many languages might have a fraction of the language modeling resources available compared to English, they still do have the readiness in terms of socioeconomics and digital infrastructure. Below, in subfigure b we show the top ranking languages in our index ranked by overall score, available at https://www.equate-index.ai/ . In subfigure c we show the global explorer feature of our index. Blue circles indicate language clusters, with the number inside showing the count of languages in each region. The markers are interactive, allowing zoom and click to reveal individual languages and their AI readiness metrics. Filters on the left enable customization by overall ranking and minimum speaker count.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Variable | Source |
| AI Resources | Number of Speakers | Wikidata https://www.wikidata.org/wiki/Help:Wikimedia_language_codes/lists/all |
| Number of Models | HuggingFace https://huggingface.co/languages | |
| Bible Translations | https://www.bible.com/languages | |
| CommonCrawl Data Volume | CommonCrawl https://commoncrawl.github.io/cc-crawl-statistics/ | |
| OPUS Data Volume | OPUS https://opus.nlpl.eu/ | |
| Wikipedia Data Volume | Wikipedia dumps |
Table 1: Summary of Extracted Variables, Their Sources and the literature behind them
Figure 6: Kernel Density Estimate (KDE) plot showing the frequency density of language proportions for two distinct datasets. The Hugging Face 2020 Models (blue curve) represent the distribution of models available on the platform, while ACL Papers 2004-2020 (orange curve) shows the distribution of languages featured in the ACL anthology. The x-axis indicates the proportion of total language count, and the y-axis represents the frequency density.
Figure 7: Logistic fit of benchmark availability against language rank by model count. Each curve shows the modeled probability that a language has at least associated benchmarks ( – , light to dark orange), as a function of its rank in the model-count distribution ( -axis, log scale, highest to lowest). Curves are logistic regressions of benchmark presence on log-rank; points show the empirical proportion within logarithmically spaced rank bins. The probability of having any benchmark ( ) falls below 50% at rank 196 (dashed line), and the probability of having multiple benchmarks declines progressively faster, indicating that both benchmark availability and benchmark variety are concentrated among the most model-rich languages.
Figure 8: Resource concentration among languages persists when multilingual artefacts are excluded. Rank–frequency distributions of Hugging Face artefacts per language, for models (left) and datasets (right), counting all artefacts (blue) and counting only those declared monolingual (orange). Both axes are logarithmic. Dashed lines are least-squares fits to the rank–frequency relation, with the fitted exponent .
| Year | Models ( ) | Datasets ( ) |
|---|---|---|
| 2020 | 0.417 | 0.365 |
| 2021 | 0.549 | 0.716 |
| 2022 | 0.878 | 0.743 |
| 2023 | 1.130 | 0.801 |
| 2024 | 0.668 | 0.133 |
Table 2: Standard deviation of residuals ( ) from Zipf’s law fit per year for language models and datasets.
| Top removed | Slope | Pearson | |||
|---|---|---|---|---|---|
| 0 | 1732 | 0.312 | 0.304 | 0.551 | |
| 5 | 1727 | 0.299 | 0.286 | 0.535 | |
| 6 | 1726 | 0.297 | 0.283 | 0.532 | |
| 7 | 1725 | 0.296 | 0.280 | 0.530 | |
| 10 | 1722 | 0.289 | 0.272 | 0.522 | |
| 15 | 1717 | 0.288 | 0.268 | 0.517 |
Table 3: Sensitivity of the speaker–model correlation to exclusion of the most-spoken languages. Log–log OLS refit of the number of models against speaker population after removing the top languages by speaker population. The correlation declines only modestly and remains moderate and highly significant throughout, indicating that the relationship is not an artefact of the most-resourced languages.
Figure 9: The vast majority of dead languages with models and datasets are located in Europe.
Figure 10: The majority of overresourced languages with models and datasets are located in Europe.
Figure 11: The vast majority of underresourced languages with models and datasets are located in Africa.
Figure 12: Distribution of mid-tier languages, the intermediate group with model presence, comprising those not classified as dead, under-resourced, or among the top five per speaker bin. They span a wide range of speaker populations and are distributed across all inhabited continents.
Figure 13: Comparative visualization of language technology development across the United States, Australia, and Nigeria reveals an unexpected pattern. Nigeria, a comparatively less technologically developed country, has a higher proportion of languages covered by at least one language model (14.7%) than both the United States (14%) and Australia (6.8%). Paradoxically, the region with the largest number of languages left behind in the United States is California—the most technologically advanced area in the world.
Figure 14: The Gompertz model fitted to the adoption rate of various technologies. The left panel shows the displacement parameter ( ), which indicates the horizontal shift of the growth curve. The right panel displays the growth rate constant ( ), which represents the steepness and speed of adoption. Language Models exhibits the highest growth rate and the lowest displacement, indicating a very rapid, early-stage adoption trajectory.
Figure 15: Heatmap illustrating the pairwise Pearson correlation coefficients between 24 variables, ordered to display the distinct clustering of socioeconomic and digital infrastructure indicators on the one hand, and AI resources indicators on the other. The color scale, ranging from deep blue (strong negative correlation) to deep red (strong positive correlation), reveals two highly correlated internal blocks: one comprising AI resource availability indicators (e.g., cc_gb, datasets_n) and the other comprising socioeconomic and digital infrastructure indicators (e.g., admin_1_hdi, literacy_rate, avg_download_speed_kbps). Crucially, the coefficients across these two blocks are predominantly weak (near zero, white/light grey), quantitatively showcasing the structural independence of general societal development from the specific availability of AI-relevant resources.
Figure 16: Heatmap illustrating the rotated component loadings for 24 variables on the first two principal components (PC1 and PC2) following Varimax rotation. PC1 is defined by high positive loadings from social and infrastructural indicators, including the proportion of individuals using the internet (loading =0.35), Human Development Index (HDI; loading =0.35), and education level (loading =0.33). Conversely, PC2 is strongly characterized by high positive loadings for AI technology availability, with the highest values observed for the volume of data on CommonCrawl (loading =0.37), the number of datasets on HuggingFace (loading =0.37), and Wikipedia active users (loading =0.37). The color intensity reflects the magnitude of the loading, ranging from a strong negative correlation (deep blue, e.g., avg_latency_ms on PC1) to a strong positive correlation (deep red). Together, these two orthogonal components explain 58.4% of the total variance, revealing a clean separation between foundational societal development and the specific availability of AI-focused resources.
Figure 17: Standardized Regression Coefficients ( ) from Stepwise Model Selection. Bar plot displaying the standardized regression coefficients ( ) and confidence intervals (CI) for predictors retained in a final model after stepwise regression. Coefficients colored light blue are statistically significant ( ), indicating a reliable positive relationship with the dependent variable. Significant positive predictors include the number of speakers ( ), the volume of Opus data ( ), the volume of Xeus data ( ), and the strongest predictor by magnitude, the presence of a Bible translation ( ). Variables in light gray (e.g., , ) were retained but found to be statistically non-significant ( ), as indicated by the CI overlapping the zero line (dashed blue). The standardized nature of the coefficients allows for a direct comparison of the relative predictive strength of each significant variable.
| Institution/Organization | Country | Role | Count |
|---|---|---|---|
| LMU Munich | Germany | PhD Student | 5 |
| Uppsala University | Sweden | Professor | 1 |
| University of Cambridge | United Kingdom | Postdoc | 1 |
| Würzburg University | Germany | Professor | 1 |
| McGill University | Canada | Professor | 1 |
| Google DeepMind | United Kingdom | Research Scientist | 1 |
Table 4: Affiliations and roles of participants in Survey 1, the ranking of AI resources. Respondents were experts in multilingual language technology (n = 28).
| Institution/Organization | Country | Role | Count |
|---|---|---|---|
| University of Cambridge | United Kingdom | Postdoctoral Researcher | 2 |
| University of Bristol | United Kingdom | Professor | 2 |
| Cohere For AI | Canada & USA | Research Scientist | 2 |
| Stanford University | USA | PhD Student | 1 |
| University of Washington | USA | PhD Student | 1 |
| George Mason University | USA | Professor | 1 |
Table 5: Affiliations and roles of participants in Survey 2, the ranking of digital-infrastructure and socioeconomic factors. Respondents were experts in human–computer interaction and global technology adoption (n = 20).
| Rank | Resource |
|---|---|
| 1 | Amount of data in Common Crawl (bytes) |
| 2 | Amount of data in Wikipedia Articles (bytes) |
| 3 | Number of already available multilingual Language Models |
| 4 | Typological similarity to well resourced language families |
| 5 | Amount of data in OPUS (bytes) |
| 6 | Number of already available monolingual Language Models |
Table 6: Ranked AI resources in order of importance for building a language technology system.
| Rank | Resource |
|---|---|
| 1 | Percentage of households with phones |
| 2 | Percentage of individuals with home internet |
| 3 | Average network speed (upload, download, latency) |
| 4 | Percentage of households with home internet access |
| 5 | Existence of a nearby university or technical institution |
| 6 | Availability of local cloud computing infrastructure (e.g., data centers, regional hosting services) |
Table 7: Ranked digital infrastructure factors in order of importance for bridging digital divides that affect the possibility of development and long-term sustainability of language technologies in under-resourced communities.
| Rank | Resource |
|---|---|
| 1 | Language community’s Institutional Status |
| 2 | Number of Speakers |
| 3 | Language community’s Vitality Status |
| 4 | Online Active Wikipedia Users (Digital Vitality) |
| 5 | Literacy Rate |
| 6 | Gross Domestic Product |
Table 8: Ranked socioeconomic factors in order of importance for bridging digital divides that affect the possibility of development and long-term sustainability of language technologies in under-resourced communities.
More in this topic
Low-Resource Language Processing
Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.
See all 565 papers