Memory-Efficient Inference

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 157% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 101

All topics

Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models
Apr 20, 2026Tobias Grantner, Emanuel Sallinger, Martin FlechlMemory-Efficient Inference Text Embedding Models
Dynatrace Research · TU Wien · University of Oxford
Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks
Apr 18, 2026Nges Brian Njungle, Eric Jahns, Michel A. KinsyEfficient Neural Network Inference Memory-Efficient Inference
STAM Center, Ira A. Fulton Schools of Engineering · Arizona State University, Tempe, AZ 85281, USA
MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
Apr 16, 2026Xinyu Liu, Xin Liu, Bo Jin +8Multi-Token Prediction KV Caching
School of Computer Science and Engineering, Northeastern University, China · Meituan Inc. · NiuTrans Research, Shenyang, China
VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination
Feb 11, 2026Muyan Hu, Ahan Gupta, Jiachen Yuan +7Efficient Neural Network Inference Memory-Efficient Inference
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Microsoft
FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
Jan 22, 2026Byeongju Kim, Jungwan Lee, Donghyeon Han +2Edge Inference Mixture-of-Experts Inference
NOSA: Native and Offloadable Sparse Attention
Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache Offloading GPU Acceleration
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1
Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management
Jun 2, 2025Yimin DuText Embeddings Word Embeddings
Independent Researcher Beijing, China
ReFreeKV: Towards Threshold-Free KV Cache Compression
Feb 24, 2025Xuanfan Ni, Liyan Xu, Chenyang Lyu +6KV Caching Memory-Efficient Optimization
Nanjing University of Aeronautics and Astronautics · WeChat AI, Tencent · Independent Researcher +1
FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices
Jan 13, 2025Yuji Chai, Mujin Kwun, David Brooks +1LLM Quantization On-Device Language Model Inference
Harvard John A. Paulson School Of Engineering And Applied Sciences Cambridge, Massachusetts, USA
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Date pendingQingxiu Liu, Yongchao He, Runhan Jiang +4Expert Offloading GPU Acceleration
CUHK · Unaffiliated · SCITIX +1
Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
Date pendingZhenhe Wu, Yaping Jin, Qinghua Xing +6Mixture of Experts Memory-Efficient Optimization
Huawei Technologies · Beihang University · Tianjin University +2

Apr 20, 2026, cs.CLJ/K move · Enter open · S save

Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models

Tobias Grantner, Emanuel Sallinger, Martin Flechl

Dynatrace Research · TU Wien · University of Oxford

Memory-Efficient Inference Text Embedding Models State Space Models

Open paper AskPDF arXiv

Transformer-based embedding models suffer from quadratic computational and linear memory complexity, limiting their utility for long sequences. We propose recurrent architectures as an efficient alternative, introducing a vertically chunked inference strategy that enables fast embedding generation with memory usage that becomes constant in the input length once it exceeds the vertical chunk size. By fine-tuning Mamba2 models, we demonstrate their viability as general-purpose text embedders, achieving competitive performance across a range of benchmarks while maintaining a substantially smaller memory footprint compared to transformer-based counterparts. We empirically validate the applicability of our inference strategy to Mamba2, RWKV, and xLSTM models, confirming consistent runtime-memory trade-offs across architectures and establishing recurrent models as a compelling alternative to transformers for efficient embedding generation.

Momentum

Related topics

Latest papers 101