Context Length

Follow this topic Ask about this topic

Recent momentum

+0%

5 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Context Length.

Period ending 2026-09-07

1 new paper

A weekly snapshot of new work published in Context Length.

64 papers

Latest in Context Length

Open your feed →

Distance generalization in transformers: why bother with positional encoding?
Sep 11, 2026Daniel Henrik Nevermann, Claudius GrosPositional Encoding Context Length
Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
Sep 9, 2026Fumihiko Tachibana, Daisuke Miyashita, Jun DeguchiKv-Cache Steering Key-Value Cache
Positional task conditioning for scalable defect detection across product families in large product catalogs
Sep 9, 2026Soham Satyadharma, Gabriel Roccabruna, Suleiman A. KhanDefect Detection Products
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
Sep 2, 2026Renjie Xie, Juncheng Yang, Aoting Hu +4Efficient Long-Context Inference Key-Value Cache
REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling
Aug 30, 2026Devrim Çavuşoğlu, Emre AkbaşRetrieval Benchmarks Context Length
Motif 3: Technical Report
Aug 10, 2026Junghwan Lim, Joon Son Chung, Sungmin Lee +24Mixture-Of-Expert Language Models Context Length
VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling
Aug 9, 2026Yuqi Zhang, Cheng Chen, Yuyu Guo +6Efficient Long-Context Inference Context Length
K-EXAONE 2.0 Technical Report
Aug 5, 2026Eunbi Choi, Kibong Choi, Sehyun Chun +74Context Length
Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks
Aug 4, 2026Mohsen ArjmandiContext Length Retrieval Benchmarks
REFACT: Adaptive Fact Restatement for Compact and Faithful Chain-of-Thought Reasoning
Jul 23, 2026Zhensheng Jin, Xin Dai, Zhenghao Liu +5LLM Reasoning Methods Reasoning Trajectory
SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
Jul 11, 2026Abhigya Verma, Khyati Mahajan, Amit Kumar Saha +4Visual Document Retrieval Multimodal Benchmarks
Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
Jul 9, 2026Siddhartha Jain, Ameya VelingkerContext Length Large Language Model Benchmarks
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Jul 7, 2026Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo +4Key-Value Cache Compression LLM Inference Efficiency
LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
Jul 7, 2026Chenhao Yuan, Yinhao Xu, Shuwen Xu +8Efficient Long-Context Inference Context Length
CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents
Jul 6, 2026Yujiang Li, Zhenyu Hou, Yi Jing +2Compaction Agentic Reinforcement Learning
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Jul 4, 2026Mary Letey, Yue M. Lu, Cengiz Pehlevan +1In-Context Learning Context Length
BamiBERT: A New BERT-based Language Model for Vietnamese
Jul 2, 2026Dat Quoc Nguyen, Thinh Pham, Chi Tran +1Bert Models Protein Language Model
Morphing into Hybrid Attention Models
Jun 29, 2026Disen Lan, Jianbin Zheng, Yuxi Ren +5Hybrid Transformer Self-Attention Mechanism
EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory
Jun 19, 2026Chang Nie, Chaoyou Fu, Junlan Feng +1Efficient Long-Context Inference Retrieval Benchmarks
The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications
Jun 18, 2026Austin Hamilton, Ryan Singh, Michael Wise +6Retrieval-Augmented Generation Context Length
KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing
Jun 15, 2026Mufei Li, Shikun Liu, Dongqi Fu +5Kv-Cache Steering Key-Value Cache
SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
Jun 15, 2026Nathanaël LangloisRetrieval-Augmented Generation Context Length
Training and Evaluating Diffusion Policies with Long Context Lengths
Jun 15, 2026Abhinav Agarwal, Adam Wei, Taylan Kargin +6Imitation Learning Diffusion Policies
A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization
Jun 14, 2026Weixiao Zhou, Gengyao Li, Xianfu Cheng +3Summarization Evaluation Benchmarks
Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL
Jun 11, 2026Shu Tong Luo, Wenqin Liu, Rui Liu +2LLM Reasoning Methods Context Length
Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents
Jun 8, 2026Abhilasha Lodha, Mahsa Pahlavikhah Varnosfaderani, Abir Chakraborty +1Context Length Large Language Model Calls
End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12Large Language Model Compression Efficient Long-Context Inference
EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering
Jun 5, 2026Xiaopeng Yuan, Zebin Wang, Suwen Wang +3Context Length Qwen3
InfoMem: Training Long-Context Memory Agents with Answer-Conditioned Information Gain
Jun 2, 2026Tiancheng Han, Yong Li, Wuzhou Yu +2Agent Memory Memforest
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
May 31, 2026Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li +1Efficient Long-Context Inference Context Length
CoMem: Context Management with A Decoupled Long-Context Model
May 29, 2026Yuwei Zhang, Chengyu Dong, Shuowei Jin +11Efficient Long-Context Inference Long-Horizon Agents
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
May 28, 2026Pierre-Antoine Lequeu, Camille Barboule, Benjamin PiwowarskiPositional Encoding Transformer Encoder
Periodic RoPE for Infinite Context LLMs
May 27, 2026Simin HuoEfficient Long-Context Inference Context Length
Separating Semantic Competition from Context Length in RAG Reading
May 26, 2026Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh +4Retrieval-Augmented Generation Reading
EchoSR: Efficient Context Harnessing for Lightweight Image Super-Resolution
May 17, 2026Hanli Zhao, Binhao Wang, Shihao Zhao +3Real-World Image Super-Resolution Multiscale
EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
May 14, 2026Han Tian, Luxuan Chen, Xinran Chen +10Efficient Long-Context Inference Context Length
Classifier Context Rot: Monitor Performance Degrades with Context Length
May 12, 2026Sam Martin, Fabien RogerContext Length Agentic Benchmarks
On Problems of Implicit Context Compression for Software Engineering Agents
May 11, 2026Kirill Gelvan, Igor Slinko, Felix Steinbauer +3Context Length Software Engineering
Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition
May 11, 2026Haoren Xu, Guanhua FangSoftmax Attention In-Context Learning
MemReread: Enhancing Agentic Long-Context Reasoning via Memory-Guided Rereading
May 11, 2026Baibei Ji, Xiaoyang Weng, Juntao Li +3Context Length Retrieval Benchmarks
Route Before Retrieve: Activating Latent Routing Abilities of LLMs for RAG vs. Long-Context Selection
May 11, 2026Yiwen Chen, Kuan Li, Fuzhen Zhuang +6Large Language Model Routing Retrieval-Augmented Generation
Scaling Limits of Long-Context Transformers
May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Softmax Attention Efficient Long-Context Inference
RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
May 8, 2026Junkai Zhang, Hang Guo, Luca Benini +1Key-Value Cache Compression Key-Value Cache
MiA-Signature: Approximating Global Activation for Long-Context Understanding
May 7, 2026Yuqing Li, Jiangnan Li, Mo Yu +3Model Activations Signatures
SCOUT: Active Information Foraging for Long-Text Understanding with Decoupled Epistemic States
May 6, 2026Zhenliang Zhang, Wenqing Wang, Yong Hu +4Context Length Epistemic States
Retrieval and Multi-Hop Reasoning in 1M-Token Context Windows: Evaluating LLMs on Classical Chinese Text
May 4, 2026Eric H. C. ChowContext Length Retrieval Benchmarks
Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks
May 2, 2026Sunil Kumar Maurya, Xin LiuReasoning Graphs Large Language Model Evaluation
On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility
May 2, 2026Zhitao He, Haolin Yang, Rui Min +2Long-Form Generation Efficient Long-Context Inference
NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus
Apr 30, 2026Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman +8Portuguese Modernbert
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Efficient Long-Context Inference Deepseek-R1
BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
Apr 24, 2026Jinghong Chen, Jingbiao Mei, Guangyu Yang +1Retrieval-Augmented Generation Knowledge-Based Visual Question Answering
SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference
Apr 24, 2026Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16Dynamic Sparse Attention Efficient Long-Context Inference
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusRecurrent Model Softmax Attention
GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0)
Apr 18, 2026Jiaqing Liang, Jinyi Han, Weijia Li +15Large Language Model Agents Self-Evolving Agents
LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
Apr 16, 2026Bowen Ping, Zijun Chen, Tingfeng Hui +4Efficient Long-Context Inference Context Length
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
Mar 11, 2026Lin Chen, Bolin Ni, Qi Yang +5Large Multimodal Models Cross-Modal Attention
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Mar 5, 2026Xin Teng, Canyu Zhang, Shaoyi Zheng +3Kv-Cache Steering Retrieval-Augmented Generation
How to Train Your Long-Context Visual Document Model
Feb 16, 2026Austin VeselkaEfficient Long-Context Inference Recent Vision-Language Models
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Jan 12, 2026Xin Cheng, Rui Tian, Wangding Zeng +18Sparsity Lookup Tables
CompLLM: Compression for Long Context Q&A
Sep 23, 2025Gabriele Berton, Jayakrishnan Unnikrishnan, Son Tran +1Efficient Long-Context Inference Large Language Model Compression

Sep 11, 2026 · cs.CLJ/K move · Enter open · S save

Distance generalization in transformers: why bother with positional encoding?

Daniel Henrik Nevermann, Claudius Gros

Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such as RoPE and ALiBi improve distance resolution relative to no positional encoding (NoPE)? (B) How does data diversity, the number of inter-token distances seen in training, affect performance? (C) When is distance transfer learning positive or negative? We present a thorough investigation, finding that it is paramount to improve our understanding of the underlying mechanisms.

Positional Encoding Context Length Distance

Open paper Ask about it PDF