Hierarchical Attention

Follow this topic Ask about this topic

Latest papers

12 papers0 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Subquadratic Attention with a Hierarchical Tree-Structured Indexer
Oct 30, 2026Leon Lufkin, Cutter Dawes, Alexander Speicher +3Efficient Attention Long-Context Language Modeling
Zyphra Research
HIDRA: Hierarchical Dual-Routing Attention for Replay-Free Lifelong Imitation Learning
Oct 26, 2026Fanqi Yu, Matteo Tiezzi, Cigdem Beyan +2Hierarchical Attention Robot Imitation Learning
AI for Good (AIGO) Research Unit, Istituto Italiano di Tecnologia, Genoa, Italy · DITEN, University of Genoa, Genoa, Italy · Pattern Analysis and Computer Vision (PAVIS), Istituto Italiano di Tecnologia, Genoa, Italy +1
Long-Context Fine-Tuning with Limited VRAM
Jul 16, 2026Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko +1Fine-Tuning KV-Cache Offloading
BMW Group
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Long-Context Language Model Inference Long-Context Language Modeling
Tencent HY Team · ShanghaiTech University · The Hong Kong University of Science and Technology +1
Hierarchical Global Attention (HGA)
Jun 29, 2026Woernle Frank, Fedosov Vladimir, Grinenko ArtemiyKV-Cache Offloading Long-Context Modeling
BMW Group
Patch Hierarchical Attention Transformer for Efficient Particle Jet Tagging
May 20, 2026Aaron Wang, Zihan Zhao, Alan Xia +5Jet Tagging Hierarchical Attention
University of Illinois Chicago Chicago, IL 60607, USA · University of California San Diego La Jolla, CA 92093, USA · California Institute of Technology Pasadena, CA 91125, USA +1
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
May 18, 2026Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti +5Self-Attention Hierarchical Attention
1Tsinghua University · 2Instituto Superior Técnico, Universidade de Lisboa. · 3Instituto de Telecomunicações. +6
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleAttention Mechanisms Self-Attention
Nous Research
Angle-I2P: Angle-Consistent-Aware Hierarchical Attention for Cross-Modality Outlier Rejection
May 6, 2026Muyao Peng, Shun Zou, Pei An +2Hierarchical Attention Point Cloud Registration
School of Electronic Information and Communications, Huazhong University of Science and Technology
Hierarchical Kernel Transformer: Multi-Scale Attention with an Information-Theoretic Approximation Analysis
Apr 10, 2026Giansalvo CirrincioneAttention Approximation Transformer Attention
Laboratoire LTI, Université de Picardie Jules Verne, 33 rue Saint-Leu, Amiens, 80039, France
HiCI: Hierarchical Construction-Integration for Long-Context Attention
Mar 21, 2026Xiangyu Zeng, Qi Xu, Yunke Wang +1Hierarchical Attention Long-Context Language Modeling
School of Computer Science, University of Sydney, Sydney, Australia · 2Westlake University, Hangzhou, China.
MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
Mar 21, 2026Dong Liu, Yanxuan Yu, Ben Lengerich +1Hierarchical Attention Attention Mechanisms
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA

Oct 30, 2026J/K move · Enter open · S save

Subquadratic Attention with a Hierarchical Tree-Structured Indexer

Leon Lufkin, Cutter Dawes, Alexander Speicher, Kamesh Krishnamurthy+2

Zyphra Research

Efficient Attention Long-Context Language Modeling LLM Inference Acceleration Sparse Attention Hierarchical Attention

Open paper AskPDF arXiv

The quadratic scaling of attention is a fundamental bottleneck limiting the long-context capabilities of large language models (LLMs). A variety of approaches have attempted to circumvent this, but they either fail to match attention's expressiveness (e.g., sliding-window or linear attention variants with fixed-size state) or only achieve constant-factor speedups (e.g., sparse attention methods whose indexers themselves scale quadratically). In this paper, we derive an asymptotically subquadratic sparse attention method based on a hierarchical, tree-structured indexer, which we call Zyphra Hierarchical Attention (ZHA). ZHA constructs a tree of recursively compressed key summaries over the KV cache, which future queries traverse from the top down to retrieve a sparse subset of keys. Crucially, by iteratively growing the tree's depth, the total cost including routing and final attention grows sublinearly in sequence length, resulting in overall subquadratic (but superlinear) scaling for attention. We demonstrate that ZHA can be cheaply applied to existing pretrained models with only a short KL distillation training stage over a few billion tokens, and maintains performance on long-context retrieval and challenging reasoning, mathematical, and agentic coding evaluation tasks. Moreover, we show that our method is highly robust to tree geometry including very deep trees of up to 6 levels. Finally, we develop a detailed performance model and show that our approach scales subquadratically, dramatically outperforms competing methods beyond 1 million context, and ultimately has the potential to extend effective context windows to the hundreds of millions or billions of tokens.