Kimi Delta Attention

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 66

All topics

Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Prefill Efficient Long-Context Inference
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University
Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention
Mar 6, 2026Haiqing Hao, Zhipeng Sui, Rong Zou +4Object Detection Dynamic Sparse Attention
State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China · Robotics and Perception Group, University of Zurich, Zurich, Switzerland · ShanghaiTech University, Shanghai, China
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Feb 2, 2026Weikang Meng, Liangyu Huo, Yadan Luo +4Kimi Delta Attention
SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland +1
Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Dec 14, 2025Jingdi Lei, Di Zhang, Soujanya PoriaKimi Delta Attention Exact
Nanyang Technological University · Fudan University
A Systematic Analysis of Hybrid Linear Attention
Jul 8, 2025Dustin Wang, Rui-Jie Zhu, Steven Abreu +9Kimi Delta Attention Transformer Architectures
1UC Santa Cruz · 2ByteDance Seed · University of Groningen +3
RotRNN: Modelling Long Sequences with Rotations
Jul 9, 2024Kai Biegun, Rares Dolga, Jake Cunningham +1Recurrent Model Sequence Modeling
UCL AI Centre London, UK · UCL AI Centre & UiPath London, UK

Apr 20, 2026, cs.AIJ/K move · Enter open · S save

Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling

Yujie Chen, Tailai Chen, Yifeng Gao, Zoe Wanying He+3

Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University

Prefill Efficient Long-Context Inference Kimi Delta Attention Visual Token Pruning Redundancy

Open paper AskPDF arXiv

Prefilling computational costs pose a significant bottleneck for Large Language Models (LLMs) and Large Multimodal Models (LMMs) in long-context settings. While token pruning reduces sequence length, prior methods rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention. In this work, we observe that tokens evolve toward \textit{semantic fixing points}, making further processing redundant. To this end, we introduce Delta Attention Selective Halting (DASH), a training-free policy that monitors the layer-wise update dynamics of the self-attention mechanism to selectively halt stabilized tokens. Extensive evaluation confirms that DASH generalizes across language and vision benchmarks, delivering significant prefill speedups while preserving model accuracy and hardware efficiency. Code will be released at https://github.com/verach3n/DASH.git.

Momentum

Related topics

Latest papers 66

Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling

Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention

STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

A Systematic Analysis of Hybrid Linear Attention

RotRNN: Modelling Long Sequences with Rotations