Token Embeddings

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 6Week of Sep 21

Latest papers 57

All topics

Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
Sep 29, 2026Zifeng Cheng, Jie Zheng, Zhiwei Jiang +4Efficient Long-Context Inference Token Embeddings
State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore · Nanjing University of Posts and Telecommunications
CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion
Sep 28, 2026Genglin Wang, Wangsong Yin, Yeerzhati Abudunuer +3Key-Value Cache Cross-Attention
The Chinese University of Hong Kong · Peking University · The Hong Kong University of Science and Technology
Instance-Adaptive Prompts as Context for Time-Series Foundation Models
Sep 28, 2026Zehao Xiao, Shifeng Xie, Lei Zan +5Time Series Foundation Models Wireless Foundation Models
Huawei Noah’s Ark Lab, Paris, France · LIPADE, Universit´e Paris Cit´e, Paris, France · Huawei Noah’s Ark Lab, Shenzhen, China
Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings
Sep 23, 2026Jiaqi DengToken Embeddings Nonlinearity
Independent Researcher
Distilling Sequential Computation in Transformer Language Models
Sep 23, 2026Zixuan Lan, Jessica Yang, Yanhong Li +2Transformer Architectures Token Embeddings
The University of Chicago · Toyota Technological Institute at Chicago · Independent Researcher +1
Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning
Sep 16, 2026Dunyao Xue, Chengshuo Du, Zhengbo Wang +2Large Language Model Decoding Decoding
Institute of Statistics and Big Data, Renmin University of China, Beijing, China · Big Data and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing, China · Center for Applied Statistics, Institute of Statistics and Big Data, Renmin University of China, Beijing, China
CoGe-GCD: Reframing Generalized Category Discovery with Compositional Generalization
Sep 9, 2026Luyao Tang, Jiewei Zheng, Kunze Huang +3Generalized Category Discovery Compositional Generalization
The University of Hong Kong, Hong Kong SAR, China · Xiamen University, Xiamen, China · Shenzhen University, Shenzhen, China.
uFlowCSP: Crystal Structure Prediction using Mean flow generative models
Sep 9, 2026Sourin Dey, Dipannoy Das Gupta, Lai Wei +2Crystal Structure Prediction Crystal Structure
Department of Computer Science and Engineering University of South Carolina Columbia, SC 29201 · Department of Computer Science and Cybersecurity University of North Georgia Columbia, SC 29201
Vectorizer: Vectorizing NumPy Programs with Shape-Guided Rewrite
Sep 8, 2026Jingqian Liu, Xiaoyu Liu, Yuepeng WangVectorization Python
Simon Fraser University Burnaby, British Columbia, Canada
Content-Based Addressing for Long Context
Sep 7, 2026Mahesh GodavartiRotary Position Embedding Efficient Long-Context Inference
A Carrot, Inc.
Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling
Sep 1, 2026Stefano Leggio, Giulio Rossolini, Alessandro BiondiVision Transformer Token Embeddings
Dept. of Excellence in Robotics & AI, Scuola Superiore Sant’Anna Pisa, Italy
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization
Aug 31, 2026Junjie Yao, Liangkai Hang, Zhi-Qin John XuToken Embeddings Distributional Signatures
School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China · Institute of Natural Sciences, School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China
Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees
Aug 30, 2026Shulei WangNext-Token Prediction Token Embeddings
Department of Statistics, University of Illinois at Urbana-Champaign, 605 E. Springfield Ave., Champaign, IL 61820
A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddings
Aug 30, 2026Alexandre QuemyToken Embeddings Intrinsic Dimensionality
Hother Labs
Universal or Language-Family-Specific Script Unification for Cross-Lingual Transfer? A Case Study on Turkic Languages
Aug 10, 2026Zijie ZhangCross-Lingual Transfer Token Embeddings
The Chinese University of Hong Kong, Shenzhen
Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
Aug 6, 2026Haris Riaz, Hyungji Kim, Mihai SurdeanuToken Embeddings Transformer Architectures
Department of Computer Science University of Arizona Tucson, AZ, USA
Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
Aug 4, 2026Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan +1Large Language Model Pretraining Token Embeddings
NVIDIA
Token-Native Storage: Read and Write in your Agent's Language
Aug 3, 2026Kumar ShivenduByte-Pair Encoding Token Embeddings
Qdrant
Interpretable GOHR Agents via Sparse Autoencoders
Jul 27, 2026Shiwei Tan, Yusong Zhao, Weiyi Qin +6Autonomous Agents Hidden States
1Rutgers University · 3Paul B Kantor Consultant · University of Illinois Urbana-Champaign
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Replay-Based Continual Learning Catastrophic Forgetting
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
Tokens are All You Need: Dual-purpose Semantic IDs for Achieving LLM-Level I/O Efficiency in recommendation systems
Jul 26, 2026Baolei Li, Yiping Yuan, Yilin Zheng +6Token Embeddings Duality
YouTube · Google Deepmind
Evaluation design conditions the expert-vs-auto MeSH gap: a controlled comparison of bag-of-words and BiomedBERT on the Cohen benchmark
Jul 23, 2026Samuel M. Okoe-MensahText Corpora Token Embeddings
Accra, Ghana
Progressive Cramming: Reliable Token Compression and What It Reveals
Jul 23, 2026Dmitrii Tarasov, Timofei Lashukov, Elizaveta Goncharova +1Token Compression Data Compression Methods
FusionBrain Lab, AXXX, Moscow, Russia · HSE University, Moscow, Russia · Artificial Intelligence Institute, Innopolis University, Innopolis, Russia
Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Models
Jul 18, 2026Ye Lu, Yihan Yan, Zhaoyang Zhang +4Speech Encoder Token Embeddings
School of Cyberspace Science, Harbin Institute of Technology, Harbin 150001, China
Sticky Jump Diffusions: A Unifying View of Masked, Continuous, and Hybrid Diffusion
Jul 12, 2026Pascal Jutras-Dubé, Patrick Pynadath, Jeremy Lu +2Diffusion Dynamics Stochastic Differential Equations
Department of Computer Science Purdue University · Department of Mathematics Purdue University
VectorizationLLM: Smart Vectorization Based AI Assistant
Jul 8, 2026Ryan DukeMath Problems Tutors
Department of Electrical & Computer Engineering Technology, New York Institute of Technology, Old Westbury, NY, USA
Riemannian Geometry for Pre-trained Language Model Embeddings
Jul 8, 2026Szczepan Konior, Alexandre Quemy, Przemysław Klocek +2Token Embeddings Text Embeddings
IBM Automation and AI, Krakow, Poland · Hother, Krakow, Poland · University of Warsaw +1
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
Jul 6, 2026Hunter HeidenreichSubword Tokenization Unigram Tokenizer
Independent Researcher
Full Glyph Images Beat Token Embeddings: A Controlled Study for Transformers
Jul 4, 2026Shuyang Xiang, Hao GuanToken Embeddings Transformer Architectures
Independent Researcher. · Institute of Software, Chinese Academy of Sciences.
Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings
Jun 30, 2026Jonas Schäfer, Cezary Pilaszewicz, Gerhard WunderLarge Language Model Watermarks Watermarks
Department of Mathematics and Computer Science Freie Universität Berlin Berlin, Germany
VASAE: Naming SAE Dictionary Directions with Vocabulary-Aligned Anchoring
Jun 26, 2026Kairui Zhang, Ziwen Yu, Zahraa S. Abdallah +1Sparse Autoencoder Features Improving Sparse Autoencoders
Intelligent Systems Laboratory, University of Bristol, Bristol, UK · University of Bristol, Bristol, UK · University of Amsterdam, Amsterdam, Netherlands
Token-to-Token Alignment of Text Embeddings for Semantic Blending
Jun 22, 2026Saar Huberman, Ron Mokady, Or Patashnik +1Image-Text Alignment Text-To-Image
1Tel Aviv University · 2BRIA AI
STORM: Stepwise Token Optimization with Reward-Guided Beam Search
Jun 9, 2026Arthur Satouf, Giulio D'Erasmo, Yuxuan Zong +3Retrieval Layer Retrievers
MILA – Quebec AI Institute & ILLS, Canada · 5Sapienza, University of Rome, Italy · 4Sorbonne Université & ISIR & CNRS, France +2
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
Jun 8, 2026Ming-Hao Hsu, Yuxuan Hu, Shujie Liu +3Speech Language Models Token Embeddings
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research, Redmond, WA, USA. · Microsoft Research Asia, Hong Kong, China.
Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
Jun 6, 2026Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides +3Unigram Tokenizer Token Embeddings
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · AIP, RIKEN, Tokyo, Japan.
Generative Criticality in Large Language Model Temperature Scaling
Jun 4, 2026Huajian Ruan, Jinyang Li, Xingyu Guo +1Autoregressive Generation Token Embeddings
State Key Laboratory of Nuclear Physics and Technology, Institute of Quantum Matter, South China Normal University, Guangzhou 510006, China · Key Laboratory of Atomic and Subatomic Structure and Quantum Control (MOE), Guangdong-Hong Kong Joint Laboratory of Quantum Matter, Guangzhou 510006, China · Guangdong Basic Research Center of Excellence for Structure and Fundamental Interactions of Matter, Guangdong Provincial Key Laboratory of Nuclear Science, Guangzhou 510006, China +4
ColBERTSaR: Sparsified ColBERT Index via Product Quantization
Jun 4, 2026Eugene Yang, Andrew Yates, Dawn Lawrie +3Retrieval Layer Token Embeddings
Johns Hopkins University, Baltimore, MD, USA
Don't Forget Your Embeddings: Robust Knowledge Erasure via Precise Editing of Embeddings
Jun 2, 2026Clara Haya Suslik, Or Shafran, Mor GevaConcept Erasure Large Language Model Unlearning
Blavatnik School of Computer Science and AI, Tel Aviv University
LLMs Need Encoders for Semantic IDs Too
May 29, 2026Xiangyi Chen, Zelun Wang, Xinyi Li +3Multimodal Large Language Models Token Embeddings
Pinterest · United States
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
May 28, 2026Rohan ShravanToken Embeddings Accents
The School of AI Bengaluru, India
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisTransformer Attention Positional Encoding
Independent Researcher, Athens, Greece.
Spectral Retrieval: Multi-Scale Sinc Convolution over Token Embeddings for Localized Retrieval in LLM Multi-Agent Systems
May 23, 2026Andrea MorandiToken Embeddings
Cisco Systems, Inc.
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisTransformer Attention Token Embeddings
Independent Researcher, Athens, Greece.
Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models
May 12, 2026Runhe Lai, Xinhua Lu, Yanqi Wu +3Citation Hallucination Detection Object Hallucination
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Peng Cheng Laboratory, Shenzhen, China · Key Laboratory of Machine Intelligence and Advanced Computing, MOE, Guangzhou, China.
Conditional Memory Enhanced Item Representation for Generative Recommendation
May 12, 2026Ziwei Liu, Yejing Wang, Shengyu Zhou +2Autoregressive Generation Token Embeddings
City University of Hong Kong Hong Kong, China · Independent Researcher Beijing, China · Tsinghua University Beijing, China
SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization
May 9, 2026Yan Sun, Guoxia Wang, Jinle Zeng +6Large Language Model Pretraining Pretraining
Baidu Inc. · Nanyang Technological University
Accurate and Efficient Statistical Testing for Word Semantic Breadth
May 8, 2026Yo EharaTwo-Sample Testing Contextual Embeddings
Tokyo Gakugei University
TIDE: Every Layer Knows the Token Beneath the Context
May 7, 2026Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +3Token Embeddings Efficient Long-Context Inference
Apple
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Transformer Architectures Token Embeddings
Stanford University
MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation
May 7, 2026Shichao Kan, Xuyang Zhang, Haojie Zhang +7Image Difference Captioning Similarity and Metrics
School of Computer Science and Engineering, Central South University, Changsha, 410083, China · School of Computer Science and Technology and the State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University, Beijing, 100044, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, 101408, China +1
Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations
May 2, 2026Pratyush Acharya, Nuraj Rimal, Habish DhakalToken Embeddings Spectral
Lost in State Space: Probing Frozen Mamba Representations
Apr 30, 2026Bhagyashree Wagh, Akash SinghToken Embeddings Recurrent State
University of Washington · University of Florida
Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings
Apr 30, 2026Tomomasa Hara, Hiroto Kurita, Masaaki Imaizumi +2Token Embeddings Text Embeddings
Tohoku University · The University of Tokyo · Kyoto University +3
Towards Intelligent Legal Document Analysis: CNN-Driven Classification of Case Law Texts
Apr 20, 2026Moinul Hossain, Sourav Rabi Das, Zikrul Shariar Ayon +4Large Language Model Classification Term Frequency-Inverse Document Frequency
Department of Computer Science and Engineering, American International University–Bangladesh, Dhaka 1229, Bangladesh · Department of Computer Science and Engineering, Prime University, Dhaka, Bangladesh · Department of Mechanical Engineering, Shajalal University of Science and Technology, Sylhet, Bangladesh +2
DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs
Apr 19, 2026Lexuan Liang, Tao Zou, Xuxiang Ta +1Text-Attributed Graph Graph Neural Networks
School of Computer Science and Engineering Beihang University Beijing, China 100191
A framework for auditing grounding claims
Dec 5, 2025Daniel Quigley, Eric MaynardContextual Grounding Model Auditing
Center for Possible Minds Indiana University Bloomington Bloomington, IN 47408 · Eruditis Milwaukee, WI 53207
Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management
Jun 2, 2025Yimin DuUnigram Tokenizer Token Embeddings
Independent Researcher Beijing, China

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings

Zifeng Cheng, Jie Zheng, Zhiwei Jiang, Shuwen Wang+3

State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore · Nanjing University of Posts and Telecommunications

Efficient Long-Context Inference Token Embeddings

Open paper AskPDF arXiv

Large language models (LLMs) have shown strong potential as training-free text encoders for long-context embeddings. Existing approaches primarily improve information flow under causal attention and typically construct embeddings by uniformly averaging all token representations. However, for long documents, such mean pooling can dilute salient semantic information with abundant redundant or weakly informative content. To this end, we propose SCSP, a training-free framework that leverages semantic compression for informative token selection in long-context embedding. Specifically, SCSP first partitions a document into sentence-aware chunks and appends a semantic compression prompt to each chunk. A prompt-isolated attention mask preserves information flow among document tokens while restricting each prompt to its corresponding local context. We then use the attention patterns elicited by these prompts to estimate token importance, select informative tokens, and aggregate their intermediate-layer representations into the final embedding. Extensive experiments on long-context embedding benchmarks demonstrate that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.

Momentum

Related topics

Latest papers 57