Long-Context Language Modeling

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 71% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 112

All topics

Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleLanguage Model Pretraining Self-Attention
Nous Research
Matrix-Decoupled Concentration for Autoregressive Sequences: Dimension-Free Guarantees for Sparse Long-Context Rewards
May 7, 2026Pei-Sen LiLong-Context Language Modeling Autoregressive Language Modeling
School of Mathematics and Statistics, Beijing Institute of Technology, Beijing 100872, China
Training Transformers for KV Cache Compressibility
May 7, 2026Yoav Gelberg, Yam Eitan, Michael Bronstein +2Long-Context Language Modeling KV-Cache Compression
University of Oxford · 2Technion – Israel Institute of Technology · 3AITHYRA +1
Caracal: Causal Architecture via Spectral Mixing
Apr 30, 2026Bingzheng Gan, Tianyi Zhang, Yusu Li +4Long-Context Language Modeling Autoregressive Language Modeling
Huawei Technologies Co., Ltd., China.
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
Apr 27, 2026Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh +7Long-Context Language Model Inference Long-Context Language Modeling
1AMD
Kwai Summary Attention Technical Report
Apr 27, 2026Chenglong Chu, Guorui Zhou, Guowang Zhang +35Self-Attention KV Caching
Kuaishou
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusLong-Context Language Modeling Recurrent Neural Networks
1MIT · 2Liquid AI
Sessa: Selective State Space Attention
Apr 20, 2026Liubomyr HorbatkoSelf-Attention Long-Context Language Modeling
OPSDL: On-Policy Self-Distillation for Long-Context Language Models
Apr 19, 2026Xinsen Zhang, Zhenkai Ding, Tianjun Pan +4Long-Context Language Modeling On-Policy Self-Distillation
Baidu Inc.
Screening Is Enough
Apr 1, 2026Ken M. NakanishiSelf-Attention Long-Context Language Modeling
Center for Emergent Matter Science (CEMS), RIKEN · Graduate School of Science, The University of Tokyo
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State
Feb 12, 2026Kaicheng Xiao, Haotian Li, Liran Dong +1Memory-Augmented Neural Networks Long-Context Language Modeling
The Chinese University of Hong Kong
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Self-Attention Long-Context Language Modeling
University of Oxford · Carnegie Mellon University · Snap Inc.
TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish
Dec 28, 2025Melikşah Türker, A. Ebrar Kızıloğlu, Onur Güngör +1Language Model Pretraining Long-Context Language Modeling
Department of Computer Engineering, Boğaziçi University, Türkiye · VNGRS, Istanbul, Türkiye · Technical University of Munich (TUM), Munich, Germany
Controllably Efficient Language Models
Nov 7, 2025Jatin Prakash, Aahlad Puli, Rajesh RanganathEfficient Transformer Inference Long-Context Language Modeling
New York University
Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
Sep 25, 2025Shiju Wang, Yujie Wang, Fangcheng Fu +6Long-Context Language Modeling Efficient Language Model Training
Tsinghua University · Peking University · Shanghai Jiao Tong University +1
Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
Sep 12, 2025Rupert Mitchell, Kristian KerstingLanguage Model Pretraining Softmax Attention
Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany +1
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Sep 4, 2025Peter RacioppoSelf-Attention Long-Context Language Modeling
Independent Researcher, Los Angeles, CA, USA.
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer Inference LLM Evaluation
University of Edinburgh · Cohere · Meta
LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers
Feb 20, 2025Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev +4Transformer Interpretability Long-Context Language Modeling
AIRI · Skoltech · Lomonosov Moscow State University +1
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Date pendingShaowen Wang, Yuke Zheng, Tansheng Zhu +4Rotary Positional Embeddings Self-Attention
Tsinghua University · Hunyuan Team, Tencent · Xiongan AI Institute +1

May 7, 2026, cs.CLJ/K move · Enter open · S save

Long Context Pre-Training with Lighthouse Attention

Bowen Peng, Subho Ghosh, Jeffrey Quesnelle

Nous Research

Language Model Pretraining Self-Attention Long-Context Language Modeling Efficient Attention Attention Mechanisms

Open paper AskPDF arXiv

Training causal transformers at extreme sequence lengths is bottlenecked by the quadratic time and memory of scaled dot-product attention (SDPA). In this work, we propose Lighthouse Attention, a training-only symmetrical selection-based hierarchical attention algorithm that wraps around ordinary SDPA and can be easily removed towards the end of the training. Our hierarchical selection is also gradient-free, which exempts us from dealing with a complicated and potentially inefficient backward pass kernel. Our contribution is three-fold: (i) A subquadratic hierarchical pre- and post-processing step that does adaptive compression and decompression of the sequence. (ii) A symmetrical compression strategy that pools queries, keys and values at the same time, while preserving left-to-right causality, which greatly improves parallelism. (iii) A two stage training approach which we pre-train for the majority of the time with Lighthouse Attention and recover a full attention model at the end with a short training. We run preliminary small scale LLM pre-training experiments that show the effectiveness of our method compared to full attention training with all other settings matched, where we achieve a faster total training time and lower final loss after the recovery phase. Full code is available at: https://github.com/ighoshsubho/lighthouse-attention

Momentum

Related topics

Latest papers 112