Efficient Long-Context Inference

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

6 new papers

A weekly snapshot of new work published in Efficient Long-Context Inference.

Inside this field

Focused directions

Focused topic

Efficient Long-Context Inference

Within Efficient Long-Context Inference

229 papersExplore

Focused topic

Context Length

Within Efficient Long-Context Inference

61 papersExplore

Focused topic

Longbench

Within Efficient Long-Context Inference

60 papersExplore

316 papers

Latest in Efficient Long-Context Inference

Browse all topics →

Soft-NBCE: Entropy-Weighted Chunk Fusion for Long-Context
May 31, 2026Shihao Ji, Mingyu Li, Zihui SongEfficient Long-Context Inference Bayesian
WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering
May 30, 2026Jinnan Yang, Yan Wang, Zhen Bi +5Efficient Long-Context Inference Key-Value Cache
Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
May 30, 2026Petr NyomaEfficient Long-Context Inference State Space Models
GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
May 29, 2026Junjie Peng, You Wu, Haoyi Wu +4Key-Value Cache Compression Key-Value Cache
CoMem: Context Management with A Decoupled Long-Context Model
May 29, 2026Yuwei Zhang, Chengyu Dong, Shuowei Jin +11Efficient Long-Context Inference Long-Horizon Agents
Loong: A Human-Like Long Document Translation Agent with Observe-and-Act Adaptive Context Selection
May 28, 2026Yutong Wang, Xuebo Liu, Derek F. Wong +6Machine Translation Models Efficient Long-Context Inference
DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark
May 28, 2026Ruofan Hu, Menghui Zhu, Jieming Zhu +8Multimodal Retrieval Multimodal Documents
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
May 28, 2026Pierre-Antoine Lequeu, Camille Barboule, Benjamin PiwowarskiPositional Encoding Transformer Encoder
Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
May 28, 2026Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir SinghKey-Value Cache Compression Efficient Long-Context Inference
STAP: A Shuffle-Tokenized App Predictor with Ultra Long Context for Vocabulary-Free Mobile App Prediction
May 28, 2026Chengyu Fan, Hang LiuNext-Token Prediction Device
Thinking as Compression: Your Reasoning Model is Secretly a Context Compressor
May 27, 2026Guoxin Ma, Yibing Liu, Chengzhengxu Li +7Efficient Long-Context Inference Context Compression
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
May 27, 2026Xiuying Wei, Caglar GulcehreMemory-Augmented Large Language Models Efficient Long-Context Inference
ZipRL: Adaptive Multi-Turn Context Compression with Hindsight Response Replay
May 27, 2026Zhexin Hu, Li Wang, Xiaohan Wang +4Reinforcement Learning Post-Training Context Compression
Periodic RoPE for Infinite Context LLMs
May 27, 2026Simin HuoEfficient Long-Context Inference Context Length
VoiceGiraffe: A Benchmark for Extreme Long-Context Audio-Language Understanding
May 27, 2026Jashin Ye, Dongxiao Wang, Yixuan Ye +10Large Audio Language Models Efficient Long-Context Inference
Separating Semantic Competition from Context Length in RAG Reading
May 26, 2026Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh +4Context Length
Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
May 26, 2026Aaron Branson Cigres Li, Zhaowei Wang, Yu Zhao +9Multimodal Retrieval Large Vision Language Models
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
May 26, 2026Hong Chen, Xiang Liu, Yubo Gao +5Key-Value Cache Compression Efficient Long-Context Inference
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
May 25, 2026Xintong Yang, Hao Gu, Binxing Xu +6Efficient Long-Context Inference Key-Value Cache Eviction
Polynomial Context-Truncation Sensitivity in Autoregressive Language Models: Sequential Wyner-Ziv Bounds for KV Cache Compression
May 24, 2026Munsik KimKey-Value Cache Compression Autoregressive Language Models
Interdomain Attention: Beyond Token-Level Key-Value Memory
May 23, 2026Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin +4Efficient Long-Context Inference Key-Value
Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems
May 22, 2026Stefano Cirillo, Domenico Desiato, Giuseppe Polese +1Global Embeddings Longbench
ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
May 21, 2026Joe SharrattEfficient Long-Context Inference Quantizer
VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding
May 21, 2026Haichen He, Jiayi Zhou, Sifeng Shang +3Video Understanding Efficient Long-Context Inference
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzEfficient Long-Context Inference Large Language Model Decoder
FastTab: A Fast Table Recognizer with a Tiny Recursive Module and 1D Transformers
May 21, 2026Laziz Hamdi, Amine Tamasna, Pascal Boisson +1Transformer Encoder Structure-Aware
Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression
May 21, 2026Wei Luo, Yi Huang, Songchen Ma +3Key-Value Cache Compression Key-Value Cache
ACC: Compiling Agent Trajectories for Long-Context Training
May 21, 2026Qisheng Su, Zhen Fang, Shiting Huang +8Efficient Long-Context Inference Large Language Model Agents
When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
May 20, 2026Doeun Lee, Muge Zhang, Yi Yu +11Clinical Reasoning Multi-Hop Question Answering Benchmarks
Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization
May 20, 2026Zhiqi Gao, Albert Ge, Alexander Berenbeim +2Supervised Fine-Tuning Longbench
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoEfficient Long-Context Inference Dynamic Sparse Attention
Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers
May 20, 2026Aleksandros Sobczyk, Gioele Gottardo, Christos K. Matzoros +4Quadratic Attention Full-Precision Performance
MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
May 20, 2026Junhao Ruan, Abudukeyumu Abudula, Bei Li +8Retrieval-Augmented Generation Systems Conversational Benchmarks
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Dynamic Sparse Attention Diffusion Language Models
GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment
May 19, 2026Minxuan Lv, Tiehua Mei, Tanlong Du +9Efficient Long-Context Inference Deep Reinforcement Learning
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
May 18, 2026Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti +5Dynamic Sparse Attention Efficient Long-Context Inference
Context Memorization for Efficient Long Context Generation
May 18, 2026Yasuyuki Okoshi, Hao Mark Chen, Guanxi Lu +3Efficient Long-Context Inference LLM Inference Efficiency
KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5Key-Value Cache Efficient Long-Context Inference
LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs
May 17, 2026Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin +5Vision Encoders Context Length
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Dynamic Sparse Attention Efficient Long-Context Inference
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1Prefill Efficient Long-Context Inference
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
May 15, 2026Shuaiyi Li, Zhisong Zhang, Yan Wang +5Global Attention Mechanism Efficient Long-Context Inference
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Position Embedding Efficient Long-Context Inference
DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts
May 14, 2026Jiading Gai, Shuai Zhang, Xiang Song +2Quadratic Attention Efficient Long-Context Inference
EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
May 14, 2026Han Tian, Luxuan Chen, Xinran Chen +10Efficient Long-Context Inference Context Length
SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference
May 13, 2026Anay Chauhan, Gurucharan Marthi Krishna Kumar, Arion Das +4Key-Value Cache Key-Value
Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
May 13, 2026Zhaowei Wang, Lishu Luo, Haodong Duan +9Efficient Long-Context Inference Large Vision Language Models
LongBEL: Long-Context and Document-Consistent Biomedical Entity Linking
May 13, 2026Adam Remaki, Xavier Tannier, Christel GérardinMedical Ontology Longbench
KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference
May 12, 2026Alireza Nadali, Patrick Cooper, Ashutosh Trivedi +1Efficient Long-Context Inference Key-Value Cache
Classifier Context Rot: Monitor Performance Degrades with Context Length
May 12, 2026Sam Martin, Fabien RogerContext Length Agentic Benchmarks
$δ$ -mem: Efficient Online Memory for Large Language Models
May 12, 2026Jingdi Lei, Di Zhang, Junxian Li +7Efficient Long-Context Inference
Training-Inference Consistent Segmented Execution for Long-Context LLMs
May 12, 2026Xianpeng Shang, Jiang Li, Zehua Duo +2Efficient Long-Context Inference LLM Inference Efficiency
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
May 11, 2026Vishal Pandey, Gopal SinghQuadratic Attention Dense Associative Memory
On Problems of Implicit Context Compression for Software Engineering Agents
May 11, 2026Kirill Gelvan, Igor Slinko, Felix Steinbauer +3Context Length Software Engineering
Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing
May 11, 2026Jinchang Zhu, Jindong Li, Chengyu Zou +4Efficient Long-Context Inference Token-Level Supervision
Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition
May 11, 2026Haoren Xu, Guanhua FangSoftmax Attention In-Context Learning
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Christopher Irwin +2Rotary Position Embedding Positional Encoding
Route Before Retrieve: Activating Latent Routing Abilities of LLMs for RAG vs. Long-Context Selection
May 11, 2026Yiwen Chen, Kuan Li, Fuzhen Zhuang +6Large Language Model Routing Context Length
FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning
May 11, 2026Zehua Pei, Hui-Ling Zhen, Xianzhi Yu +3Efficient Long-Context Inference Bilevel Optimization
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahQuadratic Attention Efficient Long-Context Inference

May 31, 2026 · cs.LGJ/K move · Enter open · S save

Soft-NBCE: Entropy-Weighted Chunk Fusion for Long-Context

Shihao Ji, Mingyu Li, Zihui Song

Chunjiang Intelligence · Beijing Normal University

The quadratic complexity of self-attention remains a bottleneck for Large Language Models (LLMs) processing ultra-long contexts. The Naive Bayes Cognitive Engine (NBCE) parallelizes long-context inference by chunking documents and routing to the lowest-entropy chunk at each decoding step. This hard-selection strategy causes semantic fragmentation during cross-chunk reasoning, as abrupt routing changes between adjacent tokens disrupt the model's contextual grounding. We present Soft-NBCE, a lightweight extension that replaces discrete chunk selection with soft entropy-weighted chunk fusion. A temperature-scaled Softmax over predictive entropies assigns continuous weights to all chunks, enabling log-space aggregation across chunk-conditioned distributions. To partially compensate for the conditional independence assumption introduced by chunking, we propose Consistency Distillation, a LoRA-based self-distillation that constrains the chunked logit distribution toward a full-context teacher via KL-divergence. On LongBench multi-hop benchmarks, Soft-NBCE with Consistency Distillation improves consistently over NBCE-style baselines (MuSiQue F1: 0.310 vs.\ 0.275 for Vanilla NBCE; HotpotQA F1: 0.479 vs.\ 0.427) while maintaining retrieval accuracy (NIAH-32K: 0.909) at O(L^2/n) peak memory.

Efficient Long-Context Inference Bayesian

Open paper Ask about it PDF