Long-Context Language Model Inference

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 275% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 166

All topics

Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets
Jul 23, 2026James E. AllchinLLM Pruning Long-Context Language Model Inference
Independent Researcher
SALT: Salience-Aware Lexical Trie for Long-Context Compression
Jul 20, 2026Oteo Mamo, Hyunjin Yi, Joydhriti Choudhury +2LLM Compression Long-Context Language Model Inference
Florida State University
Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheel
Jul 15, 2026Sietse SchelpeMemory-Augmented Language Models Knowledge Augmentation for Language Models
Corbenic AI
PReM: Learning What to Preserve and When to Refresh for Context Compression
Jul 15, 2026Bohan Yu, Lei Shen, Chenxi Zhou +5Memory-Augmented Language Models Long-Context Language Model Inference
Alibaba Group
Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference
Jul 14, 2026Soumil MandalKV Caching Long-Context Language Model Inference
ServiceNow
Global Merger-Arbitrage Forecasting with Language Models
Jul 10, 2026Hinal Jajal, Michal Mucha, Charles Sweat +3Long-Context Language Model Inference LLM Fine-Tuning
Balyasny Asset Management
Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference
Jul 8, 2026Vladimir GusevResidual VQ KV Caching
Independent Researcher
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
Jul 7, 2026Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Jul 7, 2026Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling
Jul 6, 2026Archie ChaudhuryLong-Context Language Model Inference Long-Context Language Modeling
Axionic Labs
Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
Jul 5, 2026Akhiad Bercovich, Talor Abramovich, Daniel Afrimi +67Mixture-of-Experts Pruning LLM Pruning
Uncertainty-gated selection for block-sparse attention
Jul 4, 2026Thomas RossiLong-Context Language Model Inference LLM Inference Acceleration
Eonpass
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Long-Context Language Model Inference Long-Context Language Modeling
Tencent HY Team · ShanghaiTech University · The Hong Kong University of Science and Technology +1
Training Hybrid Block Diffusion Language Models with Partial Bidirectionality
Jul 2, 2026Pranshu Chaturvedi, Parth Shroff, Tarun Suresh +2Mamba Long-Context Language Model Inference
Stanford University
Gemma 4 Technical Report
Jul 2, 2026Gemma Team, Sherif El Abd, Vaibhav Aggarwal +298Efficient Multimodal Inference Unified Multimodal Models
Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
Jul 2, 2026Wenchen Han, Gingfung Matthew Yeung, Marco Barletta +3LLM Inference KV Caching
University College London · Huawei
Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
Jul 1, 2026Aryo Pradipta Gema, Beatrice Alex, Pasquale MinerviniLong-Context Retrieval Attention Head Analysis
QUniversity of Edinburgh · KHeriot-Watt University · V Miniml.AI
MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
Jul 1, 2026Sheng Qiang, Ruiwei Chen, Yinpeng Wu +5KV Caching Long-Context Language Model Inference
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University, Shanghai, China
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3KV Caching Long-Context Language Model Inference
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
Jun 30, 2026Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1KV-Cache Offloading KV Caching
University of British Columbia · Microsoft Research
Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding
Jun 29, 2026Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +4Long-Context Language Model Inference LLM Inference Acceleration
University of Pittsburgh, Pittsburgh, PA, USA · HPE Labs, Milpitas, CA, USA
Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
Jun 28, 2026Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi +1Long-Context Language Model Inference KV-Cache Eviction
RBC Borealis
HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression
Jun 27, 2026Yuxuan Yang, Feiyang Ren, Bowen Zeng +4KV Caching Long-Context Language Model Inference
The State Key Laboratory of Blockchain and Data Security, Zhejiang University
NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation
Jun 26, 2026Qiong Tang, Xiangkun Hu, Xiangyang Liu +2Long-Context Language Model Inference Sliding-Window Attention
FARS
Information-Aware KV Cache Compression for Long Reasoning
Jun 25, 2026Jushi Kai, Zhuiri Xiao, Alexandra Birch +1KV Caching Long-Context Language Model Inference
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Jiao Tong University · School of Informatics, University of Edinburgh
PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
Jun 25, 2026Muhammad AhmedLLM Serving GPU Kernel Optimization
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
Jun 23, 2026WenHung Lee, Jian-Jia Chen, Xiaolin Lin +6Long-Context Language Model Inference KV-Cache Management
National Yang Ming Chiao Tung University · Technische Universität Darmstadt · Cornell University
Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets
Jun 22, 2026Duc Duong, Hoang Anh Duy Le, Jianwen Xie +2KV Caching Long-Context Language Model Inference
Department of Computer Science, Grinnell College · Department of Computer Science, Rice University · Lambda, Inc +1
Unlimited OCR Works
Jun 22, 2026Youyang Yin, Huanhuan Liu, YY +14Efficient Transformer Inference Long-Context Language Model Inference
Baidu Inc.
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunLong-Context Language Model Inference LLM Inference Acceleration
KAIST

Jul 23, 2026, cs.LGJ/K move · Enter open · S save

Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets

James E. Allchin

Independent Researcher

LLM Pruning Long-Context Language Model Inference Causal Interventions in Language Models

Open paper AskPDF arXiv

Pruning a long context means committing to the blocks a model will keep, and the usual selector is distilled from a dense teacher's attention. That assumes attention shows which context the answer depends on. We test the assumption on retrieval tasks where the evidence is known exactly, by masking context and measuring whether the answer changes. Attention and causal dependence disagree, and selectors inherit the disagreement: teachers attend to outdated facts that answer-preserving restrictions drop, and attend differently across training runs that use the same evidence. In a two-step reference task, a selector distilled from attention routes at 36% to 98% depending on the training run, the same selector trained on causal evidence sets reaches 99% or better on every one, and dense accuracy does not say which teacher you have. On the topologies our estimator covers, the causal sets need no annotation: masking alone recovers them from the frozen teacher. Pretrained models show the same conflict. Qwen2.5-3B attends more to an outdated fact than its replacement on 58% of the examples it answers correctly (48% past a first-token attention sink), a causal router lifts Gemma-2-9B from 56% to 98%, and no readout we test recovers a two-hop chain that all three tested models solve. On our teachers a one-pass cross-layer trace supervises chains as well as the annotated sets, so the readout fails rather than the weights; interventions alone separate current from obsolete evidence, survive redundancy, and stay stable across training runs.

Momentum

Related topics

Latest papers 166