Word Embeddings

Follow this topic Ask about this topic

Word embeddings represent words or tokens as vectors, allowing systems to compare meanings, retrieve related text, and use language features in downstream tasks. Static methods such as Word2Vec, GloVe, and FastText learn from word co-occurrences, while contextual transformers produce representations that vary with surrounding text; character and morphology-aware methods help with rare words and complex languages. Researchers study these vectors through analogy, similarity, retrieval, and linguistic-probing benchmarks, as well as their geometry, bias, memory use, and behavior during training. A central tension is that compact or stable vectors can support efficient comparison, but may miss context-dependent meaning or preserve unwanted associations; improvements on one benchmark do not always transfer to other tasks.

Latest papers

40 papers5 papers in the last four weeks, down 29% on the four weeks before. 0.0% of all new papers.

A KAN-based Non-linear Factorization for Embedding Layer Compression
Oct 21, 2026Oliver Giudice, Anna Russo RussoEmbedding Compression Low-Rank Compression
Applied Research Team, IT Dept. Banca d’Italia *−
A Statistical Inference Framework for PMI Estimation and SGNS Word Embeddings
Oct 4, 2026Zhongqi FanWord Embeddings Uncertainty Quantification
Beijing Normal-Hong Kong Baptist University
Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
Oct 2, 2026A. BochkovLLM Training Word Embeddings
Scaling and Distilling Text Embeddings for Better Diffusibility
Oct 1, 2026Zekai Zhang, Yunjie Tian, Yanjin He +4Word Embeddings Language Model Distillation
University of Michigan
Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky
Sep 30, 2026Liam Storan, Andreas Tolias, Nina MiolaneWord Embeddings Representation Geometry in Language Models
Stanford University · University of California, Santa Barbara
Comparative Evaluation of Static Embedding Models for HTTP Request Anomaly Detection
Sep 22, 2026Amanda Riverol, Gustavo Betarte, Rodrigo Martínez +1Anomaly Detection Unsupervised Anomaly Detection
Tilsor SA · Departamento de Informática e Inteligencia Artificial, Universidad Católica del Uruguay
Custom Named Entity Recognition and Topic Classification for Global Health Publications
Sep 21, 2026Genis Skura, Antoine Geissbühler, Jean-Luc FalconeNamed Entity Recognition Multi-Label Classification
University of Geneva
Analysing the Linearity of Linguistic Relations in Language Model Embedding Spaces
Sep 18, 2026Vasudevan Nedumpozhimana, Fathima Thekkekara, John KelleherRepresentation Geometry in Language Models Word Embeddings
ADAPT Research Centre Trinity College Dublin, Ireland · Indian Institute of Technology Bombay Mumbai, India
Generalization through Lexical Abstraction in Transformer Models: The Case of Functional Words
Sep 17, 2026Giuseppe Samo, Vivi Nastase, Paola MerloTransformer Interpretability Word Embeddings
Idiap Research Institute, Martigny, Switzerland · University of Geneva, Swizerland
Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Sep 10, 2026Lisa BylininaWord Embeddings Multimodal Embedding
Institute for Language Sciences Utrecht University
Reading a Legal Question Word by Word: Embedding Trajectories of 2,144 Vietnamese Legal Headlines
Sep 8, 2026Tran Minh QuanText Embeddings Word Embeddings
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization
Aug 31, 2026Junjie Yao, Liangkai Hang, Zhi-Qin John XuLanguage Model Pretraining Word Embeddings
School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China · Institute of Natural Sciences, School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China
Sparse Mutual Information Graph Averaging for Improving Random Indexing Embeddings
Aug 6, 2026Sriram Loganathan, Gokul Anand, Aung Bo Bo +2Word Embeddings Distributional Semantics
Department of Computer Science San Jose State University San Jose, CA, USA
Do Static Embeddings Add Value to Hybrid Dutch Retrieval?
Aug 3, 2026António Pereira BarataText Embeddings Word Embeddings
An empirical investigation into the properties of standard word embeddings
Jul 26, 2026Salomon KabongoText Embeddings Word Embeddings
African Institute for Mathematical Sciences (AIMS)
CAMMAR: Culture-Aware Matryoshka for Metaphorical Arabic Representations
Jul 17, 2026Suzan Awinat, Alfonso Ortega del PuenteRepresentation Geometry in Language Models Arabic NLP
Department of Computer Engineering Autonomous University of Madrid (UAM) School of Science and Technology IE University Madrid, Spain · Department of Computer Science Oviedo University Asturias, Spain
ROMEVA: Geometry-Preserving Vocabulary Expansion for Roman Urdu Language Models
Jun 21, 2026Mahnoor Khan, Afsheen Asif, Milhan Afzal Khan +2Multilingual Language Models Word Embeddings
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan · Department of Computer Science, University of Agriculture Faisalabad, Faisalabad, Pakistan
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Jun 17, 2026Tolga ŞakarWord Embeddings Subword Tokenization
Independent Researcher
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation
Jun 11, 2026Marek Šuppa, Andrej Ridzik, Daniel Hládek +2Text Embeddings Word Embeddings
Comenius University in Bratislava, Slovakia · Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · Technical University of Košice, Slovakia +1
What Does Debiasing Really Remove? A Geometric Study of PCA-Based Gender Debiasing in Word Embeddings
Jun 6, 2026Alexey Kresin, Tchifou M. Dieffi, Tomer CaspiNeural Representation Geometry Word Embeddings
Department of Computer Science, Hood College, Frederick, MD, United States · Department of Mathematics, Ben-Gurion University of the Negev, Be'er Sheva, Israel
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
May 28, 2026Rohan ShravanLanguage Model Pretraining LLM Compression
The School of AI Bengaluru, India
Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
May 22, 2026Andres Nava, Matthieu WyartNeural Representation Geometry Word Embeddings
Johns Hopkins University · EPFL & Johns Hopkins University
Domain-Independent Game Abstraction using Word Embedding Techniques
May 15, 2026Juho Kim, Tuomas SandholmWord Embeddings
Computer Science Department Carnegie Mellon University · Computer Science Department, CMU · Strategic Machine, Inc. +2
Embeddings for Preferences, Not Semantics
May 8, 2026Carter Blair, Ariel D. Procaccia, Milind TambeText Embeddings Word Embeddings
Harvard University
FastOmniTMAE: Parallel Clause Learning for Scalable and Hardware-Efficient Tsetlin Embeddings
May 7, 2026Ahmed K. Kadhim, Lei Jiao, Rishad Shafik +2Word Embeddings Text Embedding Models
Department of ICT University of Agder Grimstad, Norway · School of Engineering Newcastle University Newcastle upon Tyne, UK
Test-Time Safety Alignment
Apr 28, 2026Baturay Saglam, Dionysis KalogeriasWord Embeddings LLM Safety Alignment
Department of Electrical and Computer Engineering Yale University
ADE: Adaptive Dictionary Embeddings -- Scaling Multi-Anchor Representations to Large Language Models
Apr 27, 2026Orhan Demirci, Sezer Aptourachman, Aydın KayaText Embeddings Word Embeddings
Department of Computer Engineering Hacettepe University Ankara, Turkey
ORPHEAS: A Cross-Lingual Greek-English Embedding Model for Retrieval-Augmented Generation
Apr 22, 2026Ioannis E. Livieris, Athanasios Koursaris, Alexandra Apostolopoulou +2Retrieval-Augmented Generation Word Embeddings
Novelcore, Athens, GR 10436 · Department of Business Administration & Organization Administration, University of Peloponnese, Kalamata„ GR 24100
Community-Specific Slang and Entity Detection via Semantic Shift in Fine-Tuned Language Models
Apr 19, 2026Julia Kruk, Sanchita Porwal, Amitrajit Bhattacharjee +1Social Media Analysis Word Embeddings
Georgia Institute of Technology College of Computing Atlanta, GA, USA
Corpora deduplication or duplication in Natural Language Processing of few resourced languages ? A case of study: The Mexico's Nahuatl
Apr 8, 2026Juan-José Guzman-Landa, Juan-Manuel Torres-Moreno, Graham Ranger +4Low-Resource Language Processing Word Embeddings
{1LIA; 3LCTT}, Université d’Avignon. Avignon, France · 2LIFAT, Université de Tours. Tours, France · 5Instituto de Investigaciones en Educación, Universidad Veracruzana, Xalapa, Mexico +3

Oct 21, 2026J/K move · Enter open · S save

A KAN-based Non-linear Factorization for Embedding Layer Compression

Oliver Giudice, Anna Russo Russo

Applied Research Team, IT Dept. Banca d’Italia *−

Embedding Compression Low-Rank Compression Word Embeddings

Open paper AskPDF arXiv

Among existing embedding compression approaches, low-rank factorization provides a simple and mathematically principled solution by approximating the embedding matrix with a reduced-rank decomposition. However, its practical effectiveness is often limited, as the resulting linear bottleneck constrains representational capacity. To overcome these limitations, we propose an enhanced factorization scheme that increases expressive power by introducing learnable non-linear projections within the compressed embedding layer. Our method is inspired by the Kolmogorov Arnold Networks (KAN) paradigm, which replaces fixed linear transformations with learnable univariate functions. Despite promising results in other domains, applications of KANs in NLP remain scarce and have so far been largely limited to final or task specific layers, rather than core representational components such as embedding layers. We evaluate the proposed approach by integrating the compressed embedding layer into the Skip-gram framework and comparing it against a standard low-rank baseline under matched parameter budgets on the Word Analogy and Semantic Similarity benchmarks. We further introduce an effective initialization strategy from a pre-trained, uncompressed embedding layer. The behavior of the model is analyzed both immediately after initialization and after a brief post-initialization refinement phase, considering training with the standard Skip-gram objective as well as variants augmented with distillation from the original embedding layer. Overall, our results show that augmenting low-rank factorization with learnable non-linear projections effectively mitigates the limitations of linear bottlenecks, yielding more faithful compressed embeddings under fixed dimensionality and suggesting a promising direction for embedding layer compression beyond purely linear methods.