Text Embeddings

Follow this topic Ask about this topic

Text embeddings represent words, sentences, or documents as vectors, enabling tasks such as semantic search, clustering, similarity measurement, and retrieval-augmented generation. Methods range from static word vectors to contextual transformer and language-model encoders, trained or adapted with contrastive examples, instructions, pooling, distillation, and multilingual or domain-specific data. Evaluation commonly uses retrieval and semantic-textual-similarity benchmarks such as MTEB, alongside tests of long-context, language, and transformation robustness. A central challenge is that high benchmark scores do not guarantee that vectors preserve the specific information a task needs: pooling can dilute meaning, and semantic similarity may confound style, preference, or other target properties.

Latest papers

56 papers11 papers in the last four weeks, up 83% on the four weeks before. 0.1% of all new papers.

Low-Bit N-Gram Embeddings for Language Models
Oct 22, 2026Stephen Chung, Wenyu DuLLM Compression Low-Bit Quantization
DualverseAI
ULME: Predictive Coupling for Mutually Reinforcing Embedding and Language Modeling
Oct 22, 2026Yang Liu, Zhiqiang Yuan, Shu-Dong Huang +5Text Embeddings Text Embedding Models
Pattern Recognition Center, WeChat AI, Tencent Inc. · College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education
SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum
Oct 19, 2026Zhongxin Chen, Songhao Wu, Heng Cui +3Representation Learning Matryoshka Representation Learning
Gaoling School of Artificial Intelligence, Renmin University of China · School of Computer Science, Wuhan University · Lenovo Group
Multi-author style detection with stylistic embeddings
Oct 16, 2026Man-Ling Sung, Chak Fai Li, Damianos Karakos +2Stylometry Authorship Attribution
RTX BBN Technologies Cambridge MA, USA
Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal
Oct 16, 2026Ailiang Lin, Zhuoyun Li, Yusong Wang +3Text Embeddings Cross-Lingual Representation Alignment
Institute of Science Tokyo · Tencent
SciTBERT: A family of chronologically consistent language models for scientific and technological language processing
Oct 8, 2026Thomas Gebhart, Russell J. FunkLanguage Modeling Text Embedding Models
Carlson School of Management, University of Minnesota
Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay
Oct 4, 2026Yiyuan Luo, Vaggos ChatziafratisText Embeddings Attention Mechanisms
University of California, Santa Cruz
Learning Query Encoders Can Be Hard Even When Vector Retrieval Is Geometrically Easy
Oct 2, 2026Anders Wikum, Nina Mishra, Amin Saberi +1Dense Retrieval Text Embeddings
Stanford University · Amazon AWS · Tel Aviv University
Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction
Sep 30, 2026Grayson Wycliffe Storer, Julia Witte ZimmermanText Embeddings Language Modeling
Department of Mathematics and Statistics. · Department of Computer Science, University of Vermont, Burlington, 05405, Vermont, United States. · Computational Story Lab. +1
Doc2LoRA Provides Decodable Representations of Scientific Ideas
Sep 29, 2026Chand Sahil Mansuri, Joel Zachariah, Sadamori KojakuText Embeddings Low-Rank Adaptation
School of Systems Science and Industrial Engineering Binghamton University, Binghamton, NY, USA
Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
Sep 29, 2026Zifeng Cheng, Jie Zheng, Zhiwei Jiang +4Text Embeddings Text Embedding Models
State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore · Nanjing University of Posts and Telecommunications
Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs
Sep 26, 2026Zifeng Cheng, Lingyun Qian, Zhiwei Jiang +5Text Embeddings Text Embedding Models
State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore
Embedding Models Measure in Peculiar Ways
Sep 17, 2026Juri Opitz, Andrianos MichailText Embeddings Text Embedding Evaluation
Dept. of Computational Linguistics University of Zurich
Beyond frequency measures: Can contextual embeddings capture meaning change in scientific texts?
Sep 16, 2026Jianying Liu, Kim Gerdes, Jean-Marc DeltornText Embeddings Semantic Change Detection
STL, BETA, CEIPI · LISN, Qatent, STL · CEIPI
Multi-Functional Embedding Models for Funder Name Disambiguation in Scientific Publication Records
Sep 9, 2026Kanyao Han, Zhiwen You, Jinseok Kim +1Text Embeddings Entity Resolution
University of Illinois at Urbana-Champaign, Champaign, USA · University of Michigan, Ann Arbor, USA · Technical University of Munich, Munich, Germany +2
Reading a Legal Question Word by Word: Embedding Trajectories of 2,144 Vietnamese Legal Headlines
Sep 8, 2026Tran Minh QuanText Embeddings Word Embeddings
When Can We Work in Embedding Space? What Text Embeddings Preserve
Aug 31, 2026Simon FreyaldenhovenText Embeddings Latent Variable Models
Federal Reserve Bank of Philadelphia
Task-Conditional Flow Matching for Balanced Multilingual Text Embedding Adaptation
Aug 6, 2026Tirth Bhatt, Naren Kumar S, Mayank SinghFlow Matching Text Embeddings
LINGO Research Group, Indian Institute of Technology Gandhinagar, India
Do Static Embeddings Add Value to Hybrid Dutch Retrieval?
Aug 3, 2026António Pereira BarataText Embeddings Word Embeddings
An empirical investigation into the properties of standard word embeddings
Jul 26, 2026Salomon KabongoText Embeddings Word Embeddings
African Institute for Mathematical Sciences (AIMS)
KeySI: An Interaction Framework for Tuning Text Embeddings Based on Human Feedback
Jul 17, 2026Yan Zhu, Y. Chen, Rebecca FaustText Embeddings Text Embedding Models
Yan Zhu is with Tulane University. · Y. Chen is with Tulane University. · Rebecca Faust* is with Tulane University.
Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization
Jul 1, 2026Dain Kim, Jinseo Kim, Sungyong BaikDiffusion Model Guidance Text Embeddings
Dept. of Artificial Intelligence, Hanyang University, South Korea · Dept. of Data Science, Hanyang University, South Korea
STEB: Style Text Embedding Benchmark
Jun 30, 2026Rafael Rivera Soto, Anna Wegmann, Cristina AggazzottiText Embeddings Text Embedding Evaluation
1Johns Hopkins University · 2Utrecht University
How Can AI Find My Model? A Model-Finding Experimental Study Considering Data Formats, Embeddings, and Retrieval Strategies
Jun 29, 2026Jhon G. Botello, Jose J. Padilla, Erika Frydenlund +2Text Embeddings Learning to Rank
Dept. of Computer Science, Old Dominion University, Norfolk, VA, USA · Virginia Modeling, Analysis, and Simulation Center, Old Dominion University, Norfolk, VA, USA · Virginia Digital Maritime Center, Old Dominion University, Norfolk, VA, USA +1
A Comparative Study on Affective Cues in Text Embeddings Across Psychological Emotion Theories
Jun 27, 2026Fabio Ciani, Harald Schweiger, Emilia Parada-Cabaleiro +1Text Embeddings Text Embedding Evaluation
Department of Electronics, Information, and Bioengineering, Polytechnic University of Milan · Multimedia Mining and Search Group, Institute of Computational Perception, Johannes Kepler University Linz · Department of Music Pedagogy, Nuremberg University of Music +1
EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory
Jun 19, 2026Chang Nie, Chaoyou Fu, Junlan Feng +1Long-Context Retrieval Text Embeddings
Nanjing University
Aligning Sentence Embeddings to Human Concepts via Sparse Autoencoders
Jun 19, 2026Wonseok Shin, Songkuk KimText Embeddings Sentence Embeddings
Yonsei University
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation
Jun 11, 2026Marek Šuppa, Andrej Ridzik, Daniel Hládek +2Text Embeddings Word Embeddings
Comenius University in Bratislava, Slovakia · Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · Technical University of Košice, Slovakia +1
STEDiff: Strengthening Text Embedding for Text-to-Image Alignment in Diffusion Model
Jun 9, 2026Hailan Zhang, Haipeng Liu, Bo Fu +1Text Embeddings Text-Image Alignment
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China · School of Computer and Artificial Intelligence, Liaoning Normal University, Dalian, China
Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
Jun 6, 2026Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides +3Text Embeddings LLM Compression
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · AIP, RIKEN, Tokyo, Japan.

Oct 22, 2026J/K move · Enter open · S save

Low-Bit N-Gram Embeddings for Language Models

Stephen Chung, Wenyu Du

DualverseAI

LLM Compression Low-Bit Quantization Text Embeddings LLM Quantization

Open paper AskPDF arXiv

Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although we find that a short fine-tuning phase recovers a significant part of this loss. In addition, reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.