Kimi Delta Attention

Ask about this topic

Momentum

10 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 66

All topics
CardsList

  1. HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

    Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Kimi Delta AttentionEfficient Long-Context Inference

    Monash University · Zhejiang University

  2. Switching Linear Attention

    Sep 30, 2026Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet +3Kimi Delta AttentionLinear Attention

    Stanford University · Unconventional AI. Work done at Stanford University.

  3. STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

    Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6Recurrent StateKimi Delta Attention

    Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3

  4. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

    Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10Large Language Model QuantizationKimi Delta Attention

    UC Berkeley · University of Washington · MIT +2

  5. Quasi Linear Kernel Attention with Infinite Capacity

    Sep 28, 2026Nicolaj Rux, Johannes Hertrich, Sebastian NeumayerKimi Delta AttentionKernel Method

    Faculty of Mathematics, Chemnitz University of Technology, 09126 Chemnitz, Germany · Institute of Computer Science, University of Göttingen, 37073 Göttingen, Germany

  6. Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs

    Sep 27, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +6Kimi Delta AttentionCache

    Institute of Artificial Intelligence, China Telecom (TeleAI) China · Shanghai Jiao Tong University · Individual Researcher +1

  7. DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video

    Sep 23, 2026Taeyoun Kwon, Seungjin Kim, Hyeonyu Kim +1Kimi Delta AttentionLinear Attention

    Maum AI Inc. · Seoul National University · Yonsei University

  8. Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

    Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Kimi Delta AttentionExpressivity

  9. RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States

    Sep 14, 2026Luca Herranz-Celotti, Vincent GuigueKimi Delta AttentionRecurrent State

    ISIR Sorbonne, France

  10. OneLA: Scaling Linear-Attention Decoding to Large Beams in Generative Recommendation

    Sep 14, 2026Xiangrui Yang, Cheng Peng, Yunfeng Zhao +9Kimi Delta AttentionLinear Attention

    Kuaishou Technology China · Peking University China

  11. Kalman Delta Networks: Uncertainty-aware Associative Memory

    Sep 7, 2026Ngoc Bui, Tinglin Huang, Rex YingKimi Delta AttentionLinear Attention

    Department of Computer Science, Yale University

  12. DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving

    Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4Kimi Delta AttentionRecurrent State

    East China Normal University · Meituan · South China University of Technology

  13. Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs

    Aug 31, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +2Kimi Delta AttentionLinear Attention

    Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University

  14. CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration

    Aug 31, 2026Haoyun Jiang, Haolin Li, Jianwei Zhang +7Efficient Long-Context InferenceLLM Inference Optimization

    1CMIC, Shanghai Jiao Tong University · 2Alibaba Group · 3Fudan University

  15. Sliding-window beats linear attention

    Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1Kimi Delta AttentionLarge Language Model Memory

    Microsoft Applied Sciences Group (ASG) · Independent

  16. MixFormer: Linear Transformer with Mixture of Memory Experts

    Aug 10, 2026Yu Guo, Lei DuanKimi Delta AttentionEfficient Long-Context Inference

    School of Computer Science, Sichuan University Chengdu, China · School of Artificial Intelligence, Sichuan University Chengdu, China

  17. Linearized 2-Simplicial Attention

    Aug 10, 2026Aritra Das, Dhruman Gupta, Debayan GuptaKimi Delta Attention

    1Truth Audit Labs

  18. Retrofitting Linear Attention into Diffusion Language Models

    Aug 6, 2026Jinha Kim, Younghun Roh, Jaeyeon KimDiffusion Language ModelsKimi Delta Attention

    Apple · *Work done at MIT · Google DeepMind +1

  19. DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling

    Aug 3, 2026Yixiao Qian, Song Chen, Pengkai Wang +3Recurrent StateKimi Delta Attention

    College of Control Science and Engineering, Zhejiang University · Department of Mathematics, National University of Singapore · Hong Kong Polytechnic University +1

  20. DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows

    Aug 2, 2026Guangfu Guo, Xiaoqian Lu, Linsey Pang +4Diffusion Language ModelsKimi Delta Attention

    1Clemson University · University of Ottawa · 3PayPal +1

  21. LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs

    Jul 31, 2026Yirui Liu, Ruoling Qi, Longwen Wang +5Kimi Delta AttentionKey-Value Cache

    Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University · Xi’an Jiaotong University +1

  22. Subtract, Transport, or Replay? Auditable Deletion from Language-Model Memory

    Jul 30, 2026Vishwajith RameshLarge Language Model MemoryDeletion

    Vy Labs, Inc.

  23. Raven: High-Recall Sequence Modeling with Sparse Memory Routing

    Jul 28, 2026Arshia Afzal, Aviv Bick, Eric P. Xing +2Sequence ModelingKimi Delta Attention

    EPFL · Carnegie Mellon University · MBZUAI +1

  24. Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

    Jul 19, 2026Ayoub Ghriss, Sourav ChakrabortyKimi Delta AttentionLinear Attention

    Department of Computer Science, University of Colorado Boulder

  25. SpecLA: Efficient Speculative Decoding for Linear-Attention Models

    Jul 18, 2026Zhibin Wang, Xuying Han, Zhaohua Yang +5Speculative DecodingKimi Delta Attention

    State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China

  26. Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

    Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Kimi Delta AttentionLinear Attention

    1ETH Zurich, D-INFK · 2ETH AI Center, ETH Zurich

  27. The Key to Going Linear: Analysis-Driven Transformer Linearization

    Jul 8, 2026Anna Kuzina, Paul N. Whatmough, Babak Ehteshami BejnordiKimi Delta AttentionTransformer Architectures

    Qualcomm AI Research∗ · Qualcomm AI Research

  28. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Kimi Delta AttentionEfficient Long-Context Inference

    Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen

  29. A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

    Jul 2, 2026Wanyun CuiKimi Delta AttentionLinear Attention

    Shanghai University of Finance and Economics

  30. Precomputing the Future-Offset Average in TriAttention

    Jul 1, 2026Amarnath MukherjeeKimi Delta AttentionKey-Value Cache

    Hozhoke, Inc.

  31. SAFE-DiT: Semantics-Aware Fast-path Execution for High-Resolution Diffusion Transformers

    Jun 28, 2026Xuanhua Yin, Yuxuan Jia, Chuanzhi Xu +1Diffusion TransformersSpatio-Temporal Attention

    School of Computer Science, The University of Sydney, Australia

  32. CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

    Jun 25, 2026Sayak DuttaRecurrent ModelKimi Delta Attention

  33. Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention

    Jun 25, 2026Xiao Li, Chengruidong Zhang, Hao Luo +15Kimi Delta AttentionRecurrent Model

    1Qwen Team · 2Nanjing University · 3Zhejiang University

  34. ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory

    Jun 23, 2026Habibullah AkbarEfficient Long-Context InferenceKimi Delta Attention

    Kreasof AI Jakarta, Indonesia

  35. Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels

    Jun 12, 2026Tiantian ZhangKimi Delta AttentionRecurrent State

    Columbia University

  36. Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning

    Jun 10, 2026Kai Liu, Peijie Dong, Xinchen Xie +5Mathematical Reasoning BenchmarksLarge Language Model Reinforcement Learning

    Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Hong Kong University of Science and Technology (Guangzhou) · Shanghai AI Laboratory

  37. Dynamic Linear Attention

    Jun 9, 2026Xin Wang, Hui Shen, Boyuan Zheng +7Kimi Delta AttentionEfficient Long-Context Inference

    The Ohio State University · University of Michigan · ByteDance Seed

  38. Q-Delta: Beyond Key-Value Associative State Evolution

    Jun 7, 2026Sumin Park, Seojin Kim, Noseong ParkKimi Delta AttentionRecurrent State

    Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.

  39. When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet

    Jun 4, 2026Luoming Zhang, Yuwei Ren, Kui Zhang +7Matrix MultiplicationKimi Delta Attention

    Qualcomm AI Research, an initiative of Qualcomm Technologies, Inc.

  40. SinkRec: Mitigating Semantic State Sink in Long Sequence Recommendation with Memory-Conditioned Gated Delta Networks

    Jun 3, 2026Zhuang Zhuang, Zhipeng Wei, Ji Dai +4Recurrent StateKimi Delta Attention

    Kuaishou Technology, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China · Independent Researcher

  41. Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

    May 31, 2026Dong Le, Thong Nguyen, Cong-Duy Nguyen +1Kimi Delta AttentionLinear Attention

    Nanyang Technological University · National University of Singapore · VinUniversity

  42. Parallax: Parameterized Local Linear Attention for Language Modeling

    May 27, 2026Yifei Zuo, Dhruv Pai, Zhichen Zeng +3Kimi Delta AttentionLanguage Modeling

    Northwestern University · Tilde Research · University of Washington

  43. Approaching I/O-optimality for Approximate Attention

    May 22, 2026Pál András Papp, Aleksandros Sobczyk, Anastasios ZouziasKimi Delta AttentionMatrix Multiplication

    Computing Systems Lab, Huawei Technologies, Zurich, Switzerland

  44. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

    May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzKimi Delta AttentionGated Deltanet

    NVIDIA

  45. Tensor Cache: Eviction-conditioned Associative Memory for Transformers

    May 21, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2CacheKimi Delta Attention

    Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA

  46. Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers

    May 20, 2026Aleksandros Sobczyk, Gioele Gottardo, Christos K. Matzoros +4Kimi Delta AttentionMatrix Multiplication

    Computing Systems Lab Huawei Technologies2026 Switzerland

  47. LT2: Linear-Time Looped Transformers

    May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Kimi Delta AttentionTransformer Architectures

    1Rice University · 2Apple · 3UC Santa Cruz +1

  48. KVBuffer: IO-aware Serving for Linear Attention

    May 18, 2026Longwei Zou, Lin ZhongKimi Delta AttentionDepthweave-Kv

    Department of Computer Science Yale University New Haven, CT 06511

  49. Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders

    May 15, 2026Tomasz SteiferKimi Delta AttentionGated Deltanet

    Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland

  50. OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention

    May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Kimi Delta AttentionLinear Attention

    Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1

  51. Exact Linear Attention

    May 13, 2026Weinuo OuKimi Delta AttentionTransformer Attention

    Wuyi University

  52. The Routing and Filtering Structure of Attention

    May 12, 2026Shafayeth Jamil, Rehan KapadiaKimi Delta AttentionDecomposition

    Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089

  53. δδ-mem: Efficient Online Memory for Large Language Models

    May 12, 2026Jingdi Lei, Di Zhang, Junxian Li +7Peak MemoryEfficient Long-Context Inference

    Nanyang Technological University · Mind Lab · Fudan University +3

  54. Variational Linear Attention: Stable Associative Memory for Long-Context Transformers

    May 11, 2026Vishal Pandey, Gopal SinghKimi Delta AttentionLinear Attention

    Independent Researcher London, UK · Metriqual Athens, GR

  55. Kaczmarz Linear Attention

    May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuKimi Delta AttentionRecurrent Model

    School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China

  56. Adaptive Memory Decay for Log-Linear Attention

    May 7, 2026Yaxita Amin, Helen Zichen Li, Mengfan Zhang +1Kimi Delta AttentionMultilayer Perceptrons

    University of Maryland

  57. MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

    May 7, 2026Yulong Huang, Xiang Liu, Hongxiang Huang +5Kimi Delta AttentionLinear Attention

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

  58. ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers

    Apr 26, 2026Chih-Chung Hsu, Xin-Di Ma, Wo-Ting Liao +1Kimi Delta AttentionGumbel-Softmax Relaxation

    Advanced Computer Vision Laboratory, National Yang Ming Chiao Tung University

  59. Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences

    Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusKimi Delta AttentionRecurrent Model

    1MIT · 2Liquid AI

  60. FG2^2-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control

    Apr 21, 2026Pingwei Sun, Yuxuan Hu, Jianchao Tan +6Kimi Delta AttentionLinear Attention

    Meituan · Beijing, China