Word Embeddings

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 28

All topics

A Statistical Inference Framework for PMI Estimation and SGNS Word Embeddings
Oct 4, 2026Zhongqi FanWord Embeddings Uncertainty Quantification
Beijing Normal-Hong Kong Baptist University
Scaling and Distilling Text Embeddings for Better Diffusibility
Oct 1, 2026Zekai Zhang, Yunjie Tian, Yanjin He +4Word Embeddings Language Model Distillation
University of Michigan
Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky
Sep 30, 2026Liam Storan, Andreas Tolias, Nina MiolaneWord Embeddings Representation Geometry in Language Models
Stanford University · University of California, Santa Barbara
Custom Named Entity Recognition and Topic Classification for Global Health Publications
Sep 21, 2026Genis Skura, Antoine Geissbühler, Jean-Luc FalconeNamed Entity Recognition Multi-Label Classification
University of Geneva
Generalization through Lexical Abstraction in Transformer Models: The Case of Functional Words
Sep 17, 2026Giuseppe Samo, Vivi Nastase, Paola MerloTransformer Interpretability Word Embeddings
Idiap Research Institute, Martigny, Switzerland · University of Geneva, Swizerland
Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Sep 10, 2026Lisa BylininaWord Embeddings Multimodal Embedding
Institute for Language Sciences Utrecht University
Reading a Legal Question Word by Word: Embedding Trajectories of 2,144 Vietnamese Legal Headlines
Sep 8, 2026Tran Minh QuanText Embeddings Word Embeddings
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization
Aug 31, 2026Junjie Yao, Liangkai Hang, Zhi-Qin John XuLanguage Model Pretraining Word Embeddings
School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China · Institute of Natural Sciences, School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China
Sparse Mutual Information Graph Averaging for Improving Random Indexing Embeddings
Aug 6, 2026Sriram Loganathan, Gokul Anand, Aung Bo Bo +2Word Embeddings Distributional Semantics
Department of Computer Science San Jose State University San Jose, CA, USA
Do Static Embeddings Add Value to Hybrid Dutch Retrieval?
Aug 3, 2026António Pereira BarataText Embeddings Word Embeddings
An empirical investigation into the properties of standard word embeddings
Jul 26, 2026Salomon KabongoText Embeddings Word Embeddings
African Institute for Mathematical Sciences (AIMS)
CAMMAR: Culture-Aware Matryoshka for Metaphorical Arabic Representations
Jul 17, 2026Suzan Awinat, Alfonso Ortega del PuenteFigurative Language Understanding Arabic NLP
Department of Computer Engineering Autonomous University of Madrid (UAM) School of Science and Technology IE University Madrid, Spain · Department of Computer Science Oviedo University Asturias, Spain
ROMEVA: Geometry-Preserving Vocabulary Expansion for Roman Urdu Language Models
Jun 21, 2026Mahnoor Khan, Afsheen Asif, Milhan Afzal Khan +2Multilingual Language Models Word Embeddings
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan · Department of Computer Science, University of Agriculture Faisalabad, Faisalabad, Pakistan
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Jun 17, 2026Tolga ŞakarWord Embeddings Subword Tokenization
Independent Researcher
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation
Jun 11, 2026Marek Šuppa, Andrej Ridzik, Daniel Hládek +2Text Embeddings Word Embeddings
Comenius University in Bratislava, Slovakia · Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · Technical University of Košice, Slovakia +1
What Does Debiasing Really Remove? A Geometric Study of PCA-Based Gender Debiasing in Word Embeddings
Jun 6, 2026Alexey Kresin, Tchifou M. Dieffi, Tomer CaspiNeural Representation Geometry Word Embeddings
Department of Computer Science, Hood College, Frederick, MD, United States · Department of Mathematics, Ben-Gurion University of the Negev, Be'er Sheva, Israel
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
May 28, 2026Rohan ShravanLanguage Model Pretraining LLM Compression
The School of AI Bengaluru, India
Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
May 22, 2026Andres Nava, Matthieu WyartNeural Representation Geometry Word Embeddings
Johns Hopkins University · EPFL & Johns Hopkins University
Domain-Independent Game Abstraction using Word Embedding Techniques
May 15, 2026Juho Kim, Tuomas SandholmWord Embeddings
Computer Science Department Carnegie Mellon University · Computer Science Department, CMU · Strategic Machine, Inc. +2
Embeddings for Preferences, Not Semantics
May 8, 2026Carter Blair, Ariel D. Procaccia, Milind TambeText Embeddings Word Embeddings
Harvard University
FastOmniTMAE: Parallel Clause Learning for Scalable and Hardware-Efficient Tsetlin Embeddings
May 7, 2026Ahmed K. Kadhim, Lei Jiao, Rishad Shafik +2Word Embeddings Text Embedding Models
Department of ICT University of Agder Grimstad, Norway · School of Engineering Newcastle University Newcastle upon Tyne, UK
Test-Time Safety Alignment
Apr 28, 2026Baturay Saglam, Dionysis KalogeriasWord Embeddings LLM Safety Alignment
Department of Electrical and Computer Engineering Yale University
ADE: Adaptive Dictionary Embeddings -- Scaling Multi-Anchor Representations to Large Language Models
Apr 27, 2026Orhan Demirci, Sezer Aptourachman, Aydın KayaText Embeddings Word Embeddings
Department of Computer Engineering Hacettepe University Ankara, Turkey
ORPHEAS: A Cross-Lingual Greek-English Embedding Model for Retrieval-Augmented Generation
Apr 22, 2026Ioannis E. Livieris, Athanasios Koursaris, Alexandra Apostolopoulou +2Retrieval-Augmented Generation Word Embeddings
Novelcore, Athens, GR 10436 · Department of Business Administration & Organization Administration, University of Peloponnese, Kalamata„ GR 24100
Community-Specific Slang and Entity Detection via Semantic Shift in Fine-Tuned Language Models
Apr 19, 2026Julia Kruk, Sanchita Porwal, Amitrajit Bhattacharjee +1Social Media Analysis Word Embeddings
Georgia Institute of Technology College of Computing Atlanta, GA, USA
World Properties without World Models: Distributional Associations and the Interpretation of Decoding Results from Language Models
Mar 4, 2026Elan BarenholtzLLM World Models Word Embeddings
Department of Psychology & Center for Complex Systems and Brain Sciences Florida Atlantic University
Symmetry in language statistics shapes the geometry of model representations
Feb 16, 2026Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2Representation Learning Neural Representation Geometry
1UC Berkeley · 2EPFL · 3Johns Hopkins +1
Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management
Jun 2, 2025Yimin DuText Embeddings Word Embeddings
Independent Researcher Beijing, China

Oct 4, 2026, stat.MLJ/K move · Enter open · S save

A Statistical Inference Framework for PMI Estimation and SGNS Word Embeddings

Zhongqi Fan

Beijing Normal-Hong Kong Baptist University

Word Embeddings Uncertainty Quantification Distributional Semantics Mutual Information Estimation

Open paper AskPDF arXiv

Pointwise Mutual Information (PMI) is a core measure of testing word association, and Skip-gram with Negative Sampling (SGNS) is essentially a method that implicitly factorizes a shifted PMI matrix. However, a systematic and well-rounded characterization of finite-sample uncertainty in PMI estimation remains absent and imperative to venture into. We provide a statistical framework for PMI estimation and its connection to SGNS. We prove consistency, asymptotic unbiasedness, and asymptotic normality of the empirical PMI estimator, derive its variance via the Delta method, and, applying stochastic approximation theory, obtain a variance decomposition for SGNS-based PMI estimation that separates data variance from optimization variance. Simulation experiments validate the Delta method approximation. Real-data experiments on the Brown Corpus (d = 100) reveal that SGNS systematically deviates from the theoretical relationship PMI + log K. The empirical relationship shows an attenuated PMI coefficient, an amplified log K effect, and a positive intercept, indicating systematic bias. Word analogy validation confirms the models are effective. The failure to validate the variance decomposition under low-dimensional conditions does not diminish its theoretical value; rather, it identifies the unbiasedness assumption as the key bottleneck and clarifies the gap between asymptotic theory and practice, providing implications for both practice and theory.

Momentum

Related topics

Latest papers 28