cs.CYFeb 27, 2026

Tracing the Evolution of Word Embedding Techniques in Natural Language Processing

Authors: Minh Anh Nguyen, Kuheli Sai, Minh Nguyen

Organizations: Texas Tech University Lubbock, Texas · University of Pittsburgh Pittsburgh, Pennsylvania · Florida Atlantic University Boca Raton, Florida

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This work traces the evolution of word-embedding techniques within the natural language processing (NLP) literature. We collect and analyze 149 research articles spanning the period from 1954 to 2025, providing both a comprehensive methodological review and a data-driven bibliometric analysis of how representation learning has developed over seven decades. Our study covers four major embedding paradigms, statistical representation-based methods (one-hot encoding, bag-of-words, TF-IDF), static word embeddings (Word2Vec, GloVe, FastText), contextual word embeddings (ELMo, BERT, GPT), and sentence/document embeddings, critically discussing the strengths, limitations, and intellectual lineage connecting each category. Beyond the methodological survey, we conduct a formal era comparison using GPT-3's release as a dividing line, applying seven hypothesis tests to quantify shifts in research focus, collaboration patterns, and institutional involvement. Our analysis reveals a dramatic post-GPT-3 paradigm shift: contextual and sentence-level methods now dominate at 6.4X the odds of the pre-GPT-3 era, mean team sizes have grown significantly (p = 0.018), and 30 entirely new techniques have emerged while 54 pre-GPT-3 methods received no further attention. These findings, combined with evidence of rising industry involvement, provide a quantitative account of how the field's epistemic priorities have been reshaped by the advent of large language models.

Figures & tables

Figure 1: Data collection and annotation pipeline.
Figure 2: Distribution of paper publications per year (1954-2025).
Figure 3: Distribution of number of authors.
Figure 4: Top 20 affiliations ranked by author-affiliation count.
Figure 5: Top 20 nations ranked by author-affiliation count.
Figure 6: Number of papers by publication type.
Figure 7: The frequency of embedding method categories.
Figure 8: Original vs. Variation/Enhancement papers.
Figure 9: Coauthor network for the word-embedding literature. Node size encodes the number of publications per author, edge thickness encodes coauthorship frequency, and shaded regions indicate densely connected author clusters.
Figure 10: Coauthor graph with authors having at least 2 publications in the dataset. This filtered view highlights the core research communities.
Figure 11: Evolution of collaboration dynamics across three eras. Left: mean number of authors per paper. Right: percentage of papers with at least one industry affiliation.
Figure 12: Timeline of major word embedding techniques (1954–2022). The field evolved in four distinct phases: (1) Statistical methods (grey) focused on frequency counts; (2) Static embeddings (blue) introduced dense vector learning; (3) Contextual models (green) achieved dynamic representations culminating in LLMs; and (4) Sentence embeddings (purple) extended these principles to longer text units.
Figure 13: Distribution of embedding method categories before and after GPT-3. The post-GPT-3 era shows a dramatic shift away from static and statistical methods toward contextual and sentence/document embeddings.
Figure 14: Original vs. Variant/Enhancement by embedding category: Pre- vs. Post-GPT-3. Post-GPT-3 contextual embeddings are dominated by variant/enhancement work (62%), while the few remaining static embedding papers are entirely original contributions.
Figure 15: Industry involvement by embedding category: Pre- vs. Post-GPT-3. Contextual embeddings are heavily industry-driven in both eras, while static and statistical methods remain predominantly academic. The overall rise in industry involvement is largely a compositional effect of the field’s shift toward contextual methods.
Figure 16: Mean team size by embedding category and era. Contextual embeddings consistently require the largest teams, while static embedding team sizes actually decreased post-GPT-3, indicating the overall team size increase is driven by the field’s compositional shift toward resource-intensive categories.
MetricPre-GPT-3Post-GPT-3Testpp -valueSig.
Category distribution––Chi-square / Fisher’s<< 0.0001Yes
Team size (mean)3.854.81Welch’s tt0.018Yes
Abstract length (words)138163Welch’s tt0.0001Yes
Industry involvement (%)35.850.0Fisher’s exact0.119No
Original research (%)65.351.9Fisher’s exact0.120No
International collab. (%)16.825.9Fisher’s exact0.206No
Table 1: Summary of hypothesis tests comparing Pre-GPT-3 ( ≤\leq 2019, n=95n=95 ) vs. Post-GPT-3 ( ≥\geq 2020, n=54n=54 ) papers.
Figure 17: Summary dashboard comparing six key metrics across the Pre-GPT-3 and Post-GPT-3 eras. Each panel shows the pre- and post-GPT-3 values with the percentage change annotated. Green arrows ( ↑\uparrow ) indicate increases; red arrows ( ↓\downarrow ) indicate decreases.
More in this topic

Word Embeddings

Word embeddings map words or tokens to vectors that capture patterns of meaning, usage, and linguistic structure.

See all 40 papers

Explore similar work

CardsList
Jul 26, 2026

An empirical investigation into the properties of standard word embeddings

The embedding of word sequences into continuous vector spaces has been one of the most important developments in Natural Language Processing in the recent past. Such embeddings have found application in areas such as Automatic Speech Recognition, Machine Translation, Sentiment Analysis and many more. This essay reviews the various mechanisms that have been proposed for the calculation of word embeddings, investigates popular toolkits and embedding matrices that are available in the public domain, and experiments with one or more selected implementations to better understand their characteristics. La représentation vectorielle continue de mots a été l'un des développements les plus importants dans le domaine du traitement automatique du langage naturel au cours des dernières années. Ces représentations ont trouvé application dans des domaines tels que la reconnaissance vocale, la traduction automatique, l'analyse des sentiments, etc. Ce travail passe en revue les différents mécanismes proposés pour le calcul de ces vecteurs de mots, étudie les kits d'outils populaires et les matrices disponibles publiquement en ligne, et expérimente avec une ou plusieurs implémentations sélectionnées pour mieux comprendre leurs caractéristiques.

Salomon Kabongo

African Institute for Mathematical Sciences (AIMS)

Text EmbeddingsWord Embeddings
AskPDF
Jun 29, 2026

Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective

Most studies on technology development have been conducted from a thematic perspective, but the topics are coarse-grained and insufficient to accurately represent technology. The development of automatic entity recognition techniques makes it possible to extract technology-related entities on a large scale. Thus, we perform a more accurate analysis of technology development from an entity-centric perspective. To begin with, we extract technology-related entities such as methods, datasets, metrics, and tools in articles on Natural Language Processing (NLP), and we apply a semi-automatic approach to normalize the entities. Subsequently, we calculate the z-scores of entities based on their co-occurrence networks to measure their impact. We then analyze the development trends of new technologies in the NLP domain since the beginning of the 21st century. The findings of this paper include three aspects: Firstly, the continued increase in the average number of entities per paper implies a growing burden on researchers to acquire relevant technical background knowledge. However, the emergence of pre-trained language models has injected new vitality into the technological innovation of the NLP domain. Secondly, Methods dominate among the 179 high-impact entities. An analysis of the z-score trend about the top 10 entities reveals that pre-trained language models, exemplified by BERT and Transformer, have become mainstream in recent years. Unlike the trend of the other eight method entities, the impact of Wikipedia dataset and BLEU metric has continued to rise in the long term. Thirdly, in recent years, there has been a remarkable surge in popularity for new high-impact technologies than ever before, and their acceptance by researchers has accelerated at an unprecedented speed. Our study provides a new perspective on analyzing technology development in a specific domain.

Heng Zhang, Chengzhi Zhang, Yuzhuo Wang

Department of Information Management, Nanjing University of Science and Technology, Nanjing, 210094, China · Department of Management Science and Engineering, School of Management, Anhui University, Hefei, 230601, China

Topic pending
AskPDF
Aug 24, 2026

A Comprehensive Analysis of Arabic Natural Language Processing Research: Trends, Topic Evolution, and Research Gaps -- A Bibliometric and Topic-Based Study

Arabic Natural Language Processing (NLP) has grown rapidly over the past decade, driven by digital transformation in the Arab world, social media, and large language models (LLMs). Despite this growth, a comprehensive quantitative meta-analysis remains absent. This study presents a bibliometric and topic-based analysis of 7,120 Arabic NLP papers published between 1960 and 2026, sourced from five platforms (arXiv, ACL Anthology, Semantic Scholar, Crossref, OpenAlex) plus an additional targeted OpenAlex subset. We employ BERTopic for topic modeling, regression analysis, social network analysis, and geographic mapping. Our findings show a significant publication surge after 2020, driven by transformer models and LLMs. Topic modeling identifies 19 themes, the largest centered on text, speech, translation, and recognition (2,942 papers). Citation analysis reveals a positive correlation between paper age and citations (r = 0.245, p < 0.001); regression (R^2 = 0.105) shows that indexing in OpenAlex or Semantic Scholar and institutional affiliation are associated with higher citations. Saudi Arabia, the United States, and Egypt lead in research output. A task-dialect gap matrix identifies understudied areas, including summarization for Maghrebi, Iraqi, and Sudanese dialects. The largest topic has the highest H-index (90), followed by sentiment analysis (57). Our quantitative approach complements existing qualitative surveys and offers recommendations to prioritize under-resourced dialects and develop culturally aligned benchmarks.

Mullosharaf K. Arabov

Kazan Federal University, Institute of Computational Mathematics and Information Technologies, Kazan, Russia

Arabic NLPLow-Resource Language Processing
AskPDF