Efficient Long-Context Inference

Ask about this topic

Momentum

31 papers in the last four weeks, up 107% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 229

All topics
CardsList

  1. OVAL: Output-Aware Local Page Bases for KV Cache Retrieval

    Oct 5, 2026Ashkan Shahbazi, Chayne Thrash, Soheil KolouriEfficient Long-Context Inference

    College of Connected Computing Vanderbilt University Nashville, TN, USA

  2. ReMem: Streaming Video Understanding With Long Context Retention

    Oct 5, 2026Li Yiheng, He Xu, Wang Shaobo +2Streaming Video UnderstandingEfficient Long-Context Inference

    Nanyang Technological University · Shanghai Jiaotong University · University of the Chinese Academy of Sciences

  3. HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

    Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Kimi Delta AttentionEfficient Long-Context Inference

    Monash University · Zhejiang University

  4. Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

    Sep 30, 2026Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane +2LLM Reasoning StrategiesEfficient Long-Context Inference

    Department of Computer Science, The University of Alabama

  5. RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

    Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengEfficient Long-Context InferenceRotary Position Embedding

    University of Illinois at Urbana-Champaign, USA

  6. CommunityKV: Efficient Long-Context Decoding via Graph Partitioning

    Sep 30, 2026Joe McKenna, Anastasios Alexandridis, Nathan Susanj +1Dynamic Sparse AttentionEfficient Long-Context Inference

    Amazon AGI

  7. PatchKV: Weight-Space Compensation of KV Cache

    Sep 30, 2026Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi +2Key-Value Cache CompressionKey-Value Cache

    KAIST

  8. StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning

    Sep 30, 2026Naen Xu, Wanqing Cui, Yibo Hu +5LLM Reasoning StrategiesDialogue Benchmarks

    Zhejiang University · Taobao & Tmall Group of Alibaba

  9. Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

    Sep 29, 2026Eric Enouen, Sainyam GalhotraPositional EncodingEfficient Long-Context Inference

    Cornell University

  10. Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings

    Sep 29, 2026Zifeng Cheng, Jie Zheng, Zhiwei Jiang +4Efficient Long-Context InferenceToken Embeddings

    State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore · Nanjing University of Posts and Telecommunications

  11. Cartridges++: KV Cache Compression without Off-Context Derailment

    Sep 28, 2026Sonia Laguna, Joao Monteiro, Marco Cuturi +2Key-Value Cache CompressionEfficient Long-Context Inference

    Apple

  12. When Can Attention Heads Be Statically Defined?

    Sep 28, 2026Weixian Waylon Li, Yintao Tai, Marcio Fonseca +1Efficient Long-Context InferenceModel Fine-Tuning

    University of Edinburgh, United Kingdom · Chamber of Deputies, Brazil

  13. The Model Knows When to Stop: Training-Free Early Stopping for Long-Context Reading

    Sep 28, 2026Muath Alyobi, Mohamed Eltahir, Almoayyad Abuljdail +3Efficient Long-Context InferenceLong-Context Reasoning

    King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England

  14. Rethinking Contextualization by Reinterpreting Attention Head Channels

    Sep 27, 2026Hakaze Cho, Haolin Yang, Zhun Sun +3Efficient Long-Context InferenceChannel

    RIKEN · Tohoku University · New York University +2

  15. DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregation

    Sep 27, 2026Guanzheng Chen, Viet Dac Lai, Subhojyoti Mukherjee +5Efficient Long-Context InferenceLLM Reasoning Strategies

    National University of Singapore · Adobe Research

  16. Memory Attention

    Sep 23, 2026Jiale KangEfficient Long-Context InferenceLanguage Modeling

  17. HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

    Sep 22, 2026Jianyu Wei, Yizhao Gao, Qihao Zhang +12Dynamic Sparse AttentionEfficient Long-Context Inference

    LLM-Core Xiaomi

  18. ARM: Attention with Routed-Memory for Learnable Sparse Control

    Sep 21, 2026Qiuhao Zeng, Jerry Huang, Peng Lu +7Efficient Long-Context InferencePeak Memory

    University of Toronto, Canada · Vector Institute, Canada · RIKEN AIP, Japan +3

  19. MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing

    Sep 20, 2026Zeyu Yang, Xinyu Zhang, Zibo Bi +5Multilingual BenchmarkAcoustic Latent Space

    The Chinese University of Hong Kong, Shenzhen · Alibaba Token Hub, Alibaba Group · Xi’an Jiaotong University

  20. On-Demand Attention: Language Models Know When to Recall

    Sep 17, 2026Haibo Feng, Ruiqi Liang, Hanyang Peng +1Efficient Long-Context InferenceDynamic Attention

    Southern University of Science and Technology · Peking University · Peng Cheng Laboratory

  21. Foreground Voice Activity Detection: Learning Speaker Selectivity from Supervision

    Sep 17, 2026Guangzhao Yang, Muhammad Huzaifah, Yu Pan +2SpeakerTalk

    R&D Team, Recho Inc., Tokyo, Japan

  22. Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training

    Sep 16, 2026Tarun Suresh, Pranshu Chaturvedi, Hangoo Kang +4Diffusion Language ModelsEfficient Long-Context Inference

    Stanford University

  23. Long-Context Demonstration Selection Using State Space Models

    Sep 15, 2026Ziniu Zhang, Zhenshuo Zhang, Ruoxuan Xiong +2Efficient Long-Context InferenceIn-Context Learning

    Northeastern University · Emory University

  24. SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

    Sep 14, 2026Zhiwei Li, Lei Zhu, Hao Gu +6Dynamic Sparse AttentionBlock Sparse Flash Attention

    Tencent HY LLM Frontier · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology

  25. Distance generalization in transformers: why bother with positional encoding?

    Sep 11, 2026Daniel Henrik Nevermann, Claudius GrosPositional EncodingTransformer Architectures

    Institute for Theoretical Physics, Goethe University Frankfurt, Germany

  26. PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations

    Sep 10, 2026Hyojeong Yu, Hyukhun Koh, Minsung Kim +2Conversational MemoryInteraction History

    Dept. of ECE, Seoul National University · IPAI, Seoul National University

  27. ConvMem: Convolutional Memory for Long-Context Reasoning

    Sep 9, 2026Hongming Zhang, Zhaozhen Gu, Fengshuo Bai +6Efficient Long-Context InferenceLarge Language Model Memory

    Institute of Automation, Chinese Academy of Sciences

  28. Pushing the Boundaries of Streaming Multi-Speaker ASR: A Systematic Study of Architectural Trade-offs

    Sep 9, 2026Taejin Park, Ivan Medennikov, Kunal Dhawan +3Automatic Speech RecognitionStreaming

    NVIDIA, Santa Clara, CA, United States

  29. Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?

    Sep 9, 2026Fumihiko Tachibana, Daisuke Miyashita, Jun DeguchiKey-Value CacheCache

    Kioxia Corporation

  30. CausalChapter: Improving Long-Video Chaptering with Interventional Dependency Modeling

    Sep 8, 2026Xinran Duan, Guozhang Li, Yaoyao Zhong +3Video UnderstandingEfficient Long-Context Inference

    School of Artificial Intelligence, Beijing Normal University

  31. Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?

    Sep 8, 2026Sara Rizwan, Samaanah Abdus SalamEfficient Long-Context Inference

    Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India

  32. Content-Based Addressing for Long Context

    Sep 7, 2026Mahesh GodavartiRotary Position EmbeddingEfficient Long-Context Inference

    A Carrot, Inc.

  33. Separating Stream Stability from Long-Term Recall in Language Models

    Sep 7, 2026Peipei Cao, Xin Zhang, Jie Tang +3Large Language Model MemoryEfficient Long-Context Inference

    School of Computer and Electronic Information, Guangxi University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Guangdong University of Technology

  34. CEDAR: Error-Bounded Residual Routing for Efficient Long-Context Attention

    Sep 7, 2026Siyu Li, Dong Wang, Jie Zhou +3Dynamic Sparse AttentionEfficient Long-Context Inference

    School of Computer Science & Technology, Shanghai University · School of Computer Science & Technology, East China Normal University · School of Computer Engineering, Jiangsu Ocean University

  35. SGD-KV: Summarization Guided KV Cache Compression

    Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5Key-Value Cache CompressionEfficient Long-Context Inference

    USC, USA · KAIST, Korea · Amazon AGI, USA

  36. HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models

    Sep 2, 2026Renjie Xie, Juncheng Yang, Aoting Hu +4Efficient Long-Context InferenceKey-Value Cache

    Nanjing University of Posts and Telecommunications · Tylogi AI Lab / TAIL · Anhui University of Technology +2

  37. LatentPress: Context Compression Beyond Text and Vision

    Sep 1, 2026Zhengze Zhou, Hejian SangEfficient Long-Context InferenceLarge Language Model Compression

    1Cornell University · 2Iowa State University

  38. Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding

    Aug 31, 2026Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5Efficient Long-Context InferenceLarge Language Model Decoding

    Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Harbin Institute of Technology, Harbin, China

  39. TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories

    Aug 31, 2026Daniel Agyei Asante, Yang LiEfficient Long-Context InferenceLLM Inference Optimization

    1Iowa State University, United States

  40. Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs

    Aug 31, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +2Kimi Delta AttentionLinear Attention

    Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University

  41. CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration

    Aug 31, 2026Haoyun Jiang, Haolin Li, Jianwei Zhang +7Efficient Long-Context InferenceLLM Inference Optimization

    1CMIC, Shanghai Jiao Tong University · 2Alibaba Group · 3Fudan University

  42. REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

    Aug 30, 2026Devrim Çavuşoğlu, Emre AkbaşEfficient Long-Context InferenceRetrieval Layer

    Middle East Technical University · OBSS AI

  43. TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy

    Aug 27, 2026Hong Chen, Yudong Zeng, Yongwei Huang +4Key-Value Cache EvictionCache

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Bosum Institute of Management Science, Shenzhen, China

  44. Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

    Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Dynamic Sparse AttentionEfficient Long-Context Inference

    Amazon Web Services · University of Amsterdam · Amazon +1

  45. Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

    Aug 12, 2026Arda Uzunoglu, Benjamin Van Durme, Daniel KhashabiEfficient Long-Context InferenceLarge Language Model Training

    Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA

  46. MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

    Aug 12, 2026Ming Zhang, Kaisen Yang, Shu Yu +6Efficient Long-Context InferenceRecurrent Model

    Shanghai AI Laboratory · Tsinghua University · Fudan University

  47. Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Models

    Aug 11, 2026Yuhang Song, Bor-Jiun Lin, Jiaxu Liu +3Efficient Long-Context InferenceTransformer Architectures

    Department of Computer Science, University of Liverpool · Department of Computer Science, National Tsinghua University · Imperial College London +1

  48. Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension

    Aug 10, 2026Amanda Bertsch, Luca Soldaini, Matthew R. Gormley +4Efficient Long-Context InferenceTransformer Architectures

    Ai2 · Carnegie Mellon University · University of Washington

  49. MixFormer: Linear Transformer with Mixture of Memory Experts

    Aug 10, 2026Yu Guo, Lei DuanKimi Delta AttentionEfficient Long-Context Inference

    School of Computer Science, Sichuan University Chengdu, China · School of Artificial Intelligence, Sichuan University Chengdu, China

  50. VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling

    Aug 9, 2026Yuqi Zhang, Cheng Chen, Yuyu Guo +6Recent Vision-Language ModelsEfficient Long-Context Inference

    Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University, China · Shanghai Innovation Institute, China · Ant Group, China

  51. QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

    Aug 5, 2026Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya +3Key-Value Cache EvictionEfficient Long-Context Inference

    Mehta Family School of Data Science and Artificial Intelligence, Indian Institute of Technology Roorkee · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi

  52. OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

    Aug 5, 2026Indraneil Paul, Falko Helm, Goran Glavaš +1Efficient Long-Context InferenceCodebases

    UKP Lab & Center for Applied Cybersecurity ATHENE, TU Darmstadt · CAIDAS, University of Würzburg

  53. Training-Free Hashing-Based Attention via Binary Principal Components

    Aug 5, 2026Daohai Yu, Zhanpeng Zeng, Keyu Chen +6Dynamic Sparse AttentionBlock Sparse Flash Attention

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Tencent YouTu Lab, Shenzhen, China · Sino-Russian Research Center for Digital Economy.

  54. TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning

    Aug 4, 2026Wonpyo Park, Seung-won HwangKey-Value Cache CompressionKey-Value Cache Eviction

    Seoul National University · Google

  55. Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

    Aug 3, 2026Anusha Madan Gopal, Aras Pirbadian, Kristofor D. Carlson +2Large Language Model MemoryEfficient Long-Context Inference

    BrainChip Inc. 23041 Avenida de la Carlota, Laguna Hills, CA

  56. DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling

    Aug 3, 2026Yixiao Qian, Song Chen, Pengkai Wang +3Recurrent StateKimi Delta Attention

    College of Control Science and Engineering, Zhejiang University · Department of Mathematics, National University of Singapore · Hong Kong Polytechnic University +1

  57. Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

    Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1Dynamic Sparse AttentionBlock Sparse Flash Attention

    The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA

  58. Learning What to Remember: Test-Time Training via Context Distillation

    Aug 3, 2026Zixuan Wang, Xingyu Dang, Rui-Jie Zhu +4Efficient Long-Context InferenceTest-Time Training

    Princeton University · UC Santa Cruz · Carnegie Mellon University +1

  59. LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

    Aug 3, 2026Wen Zan, Jiaqi Zhang, Jianchao Tan +11Dynamic Sparse AttentionEfficient Long-Context Inference

    Meituan LongCat Team

  60. ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

    Jul 31, 2026Yuhang Zhan, Lisi Chen, Shuo ShangKey-Value Cache CompressionCache

    University of Electronic Science and Technology of China