Cross-Layer Attention

Latest papers

13 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention
Oct 3, 2026Zhaohui Wang, Zhixin Pan, Fanxu Meng +1Cross-Layer Attention LLM Inference Acceleration
Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology
WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing
Aug 19, 2026Wenbo Zhang, Xiang RenEfficient Transformer Inference Cross-Layer Attention
University of Southern California
The Wiola Architecture for Efficient Small Language Models
Jul 1, 2026Aryuemaan Kumar Chowdhury, Afreen Shaik, Yaparla Bhargavi +1Rotary Positional Embeddings Token Merging
Research and Development, Oscowl Ai IIT Hyderabad Hyderabad, India · Research and Development, Oscowl Ai Hyderabad, India
Enhancing Layer Interaction Using Key-Correlated Layer Attention
Jun 24, 2026Jianlong Xiong, ChuanBo Xie, Le Yu +2Efficient Attention Linear Attention
College of Computer Science, Sichuan University, Chengdu, 610065, China · Department of Ultrasound, Zigong Maternal and Child Health Care Hospital, China
LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation
Jun 10, 2026Liwen Yi, Xianlin Zhang, Yue Zhang +2Weakly Supervised Image Segmentation Cross-Layer Attention
Beijing University of Posts and Telecommunications Beijing, China
You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
Jun 4, 2026Yutao Sun, Yanqi Zhang, Li Dong +2Long-Context Language Model Inference KV-Cache Management
Microsoft Research · Tsinghua University
Depth-Attention: Cross-Layer Value Mixing for Language Models
Jun 3, 2026Boyi Zeng, Yiqin Hao, Zitong Wang +7Efficient Transformer Inference Cross-Layer Attention
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute +2
Delta Attention Residuals
May 13, 2026Cheng Luo, Zefan Cai, Junjie HuCross-Layer Attention Attention Residual Connections
Independent Researcher · University of Wisconsin–Madison
HAViT: Historical Attention Vision Transformer
Mar 19, 2026Swarnendu Banik, Manish Das, Shiv Ram Dubey +1Cross-Layer Attention Vision Transformer
Computer Vision and Biometrics Lab, Indian Institute of Information Technology, Allahabad
Mixture-of-Depths Attention
Mar 16, 2026Lianghui Zhu, Yuxin Fang, Bencheng Liao +10Cross-Layer Attention Transformer Attention
School of EIC, Huazhong University of Science & Technology · ByteDance Seed
Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
Feb 28, 2026April FuCross-Layer Attention Self-Correction in VLMs
Independent Researcher
CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill
Feb 17, 2026Bradley McDanel, Steven Li, Harshit KhaitanCross-Layer Attention LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
You Do Not Fully Utilize Transformer's Representation Capacity
Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Representation Collapse Memory-Augmented Language Models
1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University

Oct 3, 2026J/K move · Enter open · S save

LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention

Zhaohui Wang, Zhixin Pan, Fanxu Meng, Muhan Zhang

Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology

Cross-Layer Attention LLM Inference Acceleration Efficient Attention Sparse Attention Multi-Head Latent Attention

Open paper AskPDF arXiv

Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers reduces this overhead but constrains multiple layers to the same token set. We introduce LatentIndex, which extends the latent-sharing principle of Multi-head Latent Attention across indexer layers. Each layer group constructs a shared latent cache from its first layer's hidden states, while layer-specific scoring enables independent token selection. Absorbing key decoders into queries enables direct scoring of the shared cache without reconstructing historical per-layer keys. We develop training-free calibration and investigate a training-aware instantiation of this principle. To balance quality and computation, a hierarchical selection (HS) variant lets followers independently refine a shared candidate set proposed by the anchor. With four-layer sharing, LatentIndex reduces logical indexer-cache storage by 61.1% on DeepSeek-V3.2. Across DeepSeek-V3.2 and GLM-5, training-free LatentIndex improves head-wise attention-mass recall over IndexCache by up to 3.28 percentage points while maintaining RULER and LongBench performance close to native DSA. HS further achieves 2.30-2.72 times decode indexer speedups over DSA across 8K-128K contexts, retaining most of LatentIndex's recall. LatentIndex offers a new perspective on cross-layer indexing: sharing continuous representations rather than discrete selections enables efficient reuse while preserving layer-specific token selection.