Long-Context Language Model Evaluation

Follow this topic Ask about this topic

This field tests whether language models can retrieve evidence, combine information across distant passages, follow long conversations, and answer reliably as inputs grow. Researchers use benchmarks with varied context lengths, evidence positions, distractors, reasoning steps, and tasks such as summarization, code question answering, and multi-turn memory; some also test audio, images, or real-world procedures. Controlled comparisons, evidence annotations, agent harnesses, and mechanistic diagnostics help separate retrieval failures from problems with reasoning, attention, or context compression. A central finding is that advertised context length does not guarantee effective use of the whole input, while benchmark scores can be confounded by task difficulty, position, and distractor design.

Latest papers

68 papers14 papers in the last four weeks, up 17% on the four weeks before. 0.1% of all new papers.

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering
Oct 22, 2026Aman Behera, Namit Solanki, Mehul AnandState Space Language Models Long-Context Language Modeling
IIT Roorkee · AISSMS IOIT, Pune · Independent Researcher
LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking
Oct 19, 2026Howard Yen, Nitya Mani, Alex Renda +1Long-Context Language Model Evaluation Long-Context Reasoning in Language Models
Princeton Language and Intelligence, Princeton University · Jane Street
DiaLong: Benchmarking Reminiscence in Long-Context Dialogues
Oct 18, 2026Owen Bianchi, Samer Aslan, Ryunosuke SaitoConversational Memory Long-Context Language Model Evaluation
Center for Language and Speech Processing, Johns Hopkins University
LongForge: Measuring and Improving Long-Context Instruction Data via Evidence-Assertion Reasoning Graphs
Oct 16, 2026Ziyang Chen, Junlong Jia, Xing Wu +2Synthetic Data Generation for Language Models Instruction Tuning
Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
Oct 15, 2026Shiwen Qin, Cameron Barker, Henry Gouk +4Long-Context Reasoning in Language Models Long-Context Language Model Evaluation
University of Edinburgh, Edinburgh, United Kingdom
Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute
Oct 7, 2026Sietse SchelpePersistent Memory for Language Models KV-Cache Reuse
Corbenic AI
LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads?
Oct 2, 2026Xinyi Liu, Rinat Khaziev, Dilek Hakkani-Tür +1Social Reasoning Long-Context Language Model Evaluation
University of Illinois Urbana-Champaign · Semantic Code Lab
TPBench: A Turning-Point Benchmark for Dialogue Compression
Oct 2, 2026Minji Park, Seunghyun Yoon, Hyuk LimMulti-Turn Dialogue Evaluation Spoken Dialogue Systems
Korea Institute of Energy Technology (KENTECH)
Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning
Sep 30, 2026Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane +2Long-Context Language Model Evaluation Clinical Reasoning
Department of Computer Science, The University of Alabama
RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures
Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional Embeddings Long-Context Language Model Evaluation
University of Illinois at Urbana-Champaign, USA
Staying on Task: Testing the Foundations of Long-Horizon Agent Reliability
Sep 30, 2026Jeffrey Willette, Krishna C. Puvvada, Boris GinsburgLong-Context Language Model Evaluation Long-Horizon Agent Evaluation
NVIDIA
LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
Sep 29, 2026Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen +1Long-Context Language Model Evaluation Long-Context Language Model Inference
University of Alberta · Alberta Machine Intelligence Institute (Amii)
CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles
Sep 28, 2026Mir Tafseer Nayeem, Susmoy Chakraborty, Davood RafieiMultilingual Language Model Evaluation Long-Context Language Model Evaluation
University of Alberta · Independent Researcher
The Model Knows When to Stop: Training-Free Early Stopping for Long-Context Reading
Sep 28, 2026Muath Alyobi, Mohamed Eltahir, Almoayyad Abuljdail +3Early Stopping Long-Context Language Model Evaluation
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England
VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models
Sep 26, 2026Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden +1Audio-Language Model Evaluation Long-Term Conversational Memory
University of Melbourne · University of New South Wales
Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension
Sep 26, 2026Kohei Kajikawa, Lin Ai, Tatsuki Kuribayashi +1Computational Psycholinguistics Long-Context Language Modeling
Department of Linguistics, Georgetown University, USA · Computing and Mathematical Sciences Division, MBZUAI, UAE · Center for Language AI Research, Tohoku University, Japan
No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow
Sep 24, 2026Prasann Singhal, Amanda Bertsch, Jacob Steinhardt +1Long-Context Language Model Evaluation LLM Reasoning
UC Berkeley · Carnegie Mellon University · Allen Institute for AI
MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors
Sep 23, 2026Wei He, Aline Villavicencio, Rodrigo Wilkens +1Multilingual Language Model Evaluation Long-Context Language Model Evaluation
University of Exeter · University of Cambridge
The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence
Sep 22, 2026Xiaoyu Yang, Jie Lu, Wei Duan +1Long-Context Language Model Evaluation Evidence Retrieval
Australian Artificial Intelligence Institute (AAII), Faulty of Engineering and Information Technology, University of Technology Sydney, Australia.
Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models
Sep 18, 2026Xize Cheng, Wenxu Jia, Chenyuhao Wen +4Long-Context Language Model Evaluation Long-Context Modeling
Zhejiang University
Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
Sep 11, 2026Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie +2Long-Horizon LLM Reasoning Long-Context Language Model Evaluation
Manulife Pittsburgh, United States · Manulife Toronto, Canada · Manulife Bangalore, India
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sep 8, 2026Sara Rizwan, Samaanah Abdus Salam, Mohammed Mudassir UddinLong-Context Language Model Evaluation Long-Context Language Modeling
Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India
Separating Stream Stability from Long-Term Recall in Language Models
Sep 7, 2026Peipei Cao, Xin Zhang, Jie Tang +3Memory-Augmented Language Models Long-Context Language Model Evaluation
School of Computer and Electronic Information, Guangxi University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Guangdong University of Technology
HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Care
Sep 7, 2026Yuchen Niu, Yanan Ma, Srinivasan Nandakumar +7Clinical Language Model Evaluation Wearable Health Monitoring
Nanyang Technological University · Mayo Clinic · The University of Manchester +2
Polish ModernBERT: The Long and Short of Polish Language Understanding
Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model Pretraining Long-Context Language Model Evaluation
National Information Processing Institute, Warsaw, Poland
VoiceLongMemEval: Do Assistants Remember How You Sounded?
Sep 1, 2026Ramit Pahwa, Parivesh Priye, Apoorva BeeduAudio-Language Model Evaluation Long-Context Language Model Evaluation
Rivian VW Tech
MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects
Aug 30, 2026Jason Luo, Saibilila Abudukelimu, Judy Song +4Document QA Language Model Robustness
LongPIBench: A Long-Context Benchmark for Prompt Injection
Aug 28, 2026Yupei Liu, Yuqi Jia, Neil Zhenqiang Gong +1Prompt Injection Prompt Injection Attacks
The Pennsylvania State University · Duke University
LongGuard: Mechanistic Analysis and Training-Free Mitigation of Long-Context Failure in Safety Guardrails
Aug 27, 2026Ziyang Chen, Xing Wu, Songlin HuLanguage Model Safety Evaluation Long-Context Language Model Evaluation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
MGAL: A Multilingual Granularity-Aware Long-Context Benchmark
Aug 21, 2026Chunhan Li, Chenglin Xu, Zongyang Zhang +8Long-Context Language Model Evaluation Multilingual Language Model Evaluation
The Hong Kong University of Science and Technology(Guangzhou) · China University of Petroleum (East China) · Northeastern University at Qinhuangdao +1

Oct 22, 2026J/K move · Enter open · S save

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

Aman Behera, Namit Solanki, Mehul Anand

IIT Roorkee · AISSMS IOIT, Pune · Independent Researcher

State Space Language Models Long-Context Language Modeling Long-Context Language Model Evaluation Long-Context QA Language Modeling Hybrid Mamba-Transformer Architectures

Open paper AskPDF arXiv

Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.