cs.CYFeb 27, 2026

Tracing the Evolution of Word Embedding Techniques in Natural Language Processing

Authors: Minh Anh Nguyen, Kuheli Sai, Minh Nguyen

Organizations: Texas Tech University Lubbock, Texas · University of Pittsburgh Pittsburgh, Pennsylvania · Florida Atlantic University Boca Raton, Florida

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This work traces the evolution of word-embedding techniques within the natural language processing (NLP) literature. We collect and analyze 149 research articles spanning the period from 1954 to 2025, providing both a comprehensive methodological review and a data-driven bibliometric analysis of how representation learning has developed over seven decades. Our study covers four major embedding paradigms, statistical representation-based methods (one-hot encoding, bag-of-words, TF-IDF), static word embeddings (Word2Vec, GloVe, FastText), contextual word embeddings (ELMo, BERT, GPT), and sentence/document embeddings, critically discussing the strengths, limitations, and intellectual lineage connecting each category. Beyond the methodological survey, we conduct a formal era comparison using GPT-3's release as a dividing line, applying seven hypothesis tests to quantify shifts in research focus, collaboration patterns, and institutional involvement. Our analysis reveals a dramatic post-GPT-3 paradigm shift: contextual and sentence-level methods now dominate at 6.4X the odds of the pre-GPT-3 era, mean team sizes have grown significantly (p = 0.018), and 30 entirely new techniques have emerged while 54 pre-GPT-3 methods received no further attention. These findings, combined with evidence of rising industry involvement, provide a quantitative account of how the field's epistemic priorities have been reshaped by the advent of large language models.

Figures & tables

Figure 1: Data collection and annotation pipeline.
Figure 2: Distribution of paper publications per year (1954-2025).
Figure 3: Distribution of number of authors.
Figure 4: Top 20 affiliations ranked by author-affiliation count.
Figure 5: Top 20 nations ranked by author-affiliation count.
Figure 6: Number of papers by publication type.
Figure 7: The frequency of embedding method categories.
Figure 8: Original vs. Variation/Enhancement papers.
Figure 9: Coauthor network for the word-embedding literature. Node size encodes the number of publications per author, edge thickness encodes coauthorship frequency, and shaded regions indicate densely connected author clusters.
Figure 10: Coauthor graph with authors having at least 2 publications in the dataset. This filtered view highlights the core research communities.
Figure 11: Evolution of collaboration dynamics across three eras. Left: mean number of authors per paper. Right: percentage of papers with at least one industry affiliation.
Figure 12: Timeline of major word embedding techniques (1954–2022). The field evolved in four distinct phases: (1) Statistical methods (grey) focused on frequency counts; (2) Static embeddings (blue) introduced dense vector learning; (3) Contextual models (green) achieved dynamic representations culminating in LLMs; and (4) Sentence embeddings (purple) extended these principles to longer text units.
Figure 13: Distribution of embedding method categories before and after GPT-3. The post-GPT-3 era shows a dramatic shift away from static and statistical methods toward contextual and sentence/document embeddings.
Figure 14: Original vs. Variant/Enhancement by embedding category: Pre- vs. Post-GPT-3. Post-GPT-3 contextual embeddings are dominated by variant/enhancement work (62%), while the few remaining static embedding papers are entirely original contributions.
Figure 15: Industry involvement by embedding category: Pre- vs. Post-GPT-3. Contextual embeddings are heavily industry-driven in both eras, while static and statistical methods remain predominantly academic. The overall rise in industry involvement is largely a compositional effect of the field’s shift toward contextual methods.
Figure 16: Mean team size by embedding category and era. Contextual embeddings consistently require the largest teams, while static embedding team sizes actually decreased post-GPT-3, indicating the overall team size increase is driven by the field’s compositional shift toward resource-intensive categories.
MetricPre-GPT-3Post-GPT-3Testpp -valueSig.
Category distribution––Chi-square / Fisher’s<< 0.0001Yes
Team size (mean)3.854.81Welch’s tt0.018Yes
Abstract length (words)138163Welch’s tt0.0001Yes
Industry involvement (%)35.850.0Fisher’s exact0.119No
Original research (%)65.351.9Fisher’s exact0.120No
International collab. (%)16.825.9Fisher’s exact0.206No
Table 1: Summary of hypothesis tests comparing Pre-GPT-3 ( ≤\leq 2019, n=95n=95 ) vs. Post-GPT-3 ( ≥\geq 2020, n=54n=54 ) papers.
Figure 17: Summary dashboard comparing six key metrics across the Pre-GPT-3 and Post-GPT-3 eras. Each panel shows the pre- and post-GPT-3 values with the percentage change annotated. Green arrows ( ↑\uparrow ) indicate increases; red arrows ( ↓\downarrow ) indicate decreases.
More in this topic

Word Embeddings

Word embeddings map words or tokens to vectors that capture patterns of meaning, usage, and linguistic structure.

See all 40 papers

Explore similar work

CardsList

  1. An empirical investigation into the properties of standard word embeddings

    Jul 26, 2026Salomon KabongoText EmbeddingsWord Embeddings

    African Institute for Mathematical Sciences (AIMS)

  2. Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective

    Jun 29, 2026Heng Zhang, Chengzhi Zhang, Yuzhuo Wang

    Department of Information Management, Nanjing University of Science and Technology, Nanjing, 210094, China · Department of Management Science and Engineering, School of Management, Anhui University, Hefei, 230601, China

  3. A Comprehensive Analysis of Arabic Natural Language Processing Research: Trends, Topic Evolution, and Research Gaps -- A Bibliometric and Topic-Based Study

    Aug 24, 2026Mullosharaf K. ArabovArabic NLPLow-Resource Language Processing

    Kazan Federal University, Institute of Computational Mathematics and Information Technologies, Kazan, Russia