Long-Context Language Model Evaluation

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 31

All topics

Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning
Sep 30, 2026Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane +2Long-Context Language Model Evaluation Clinical Reasoning
Department of Computer Science, The University of Alabama
RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures
Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional Embeddings Long-Context Language Model Evaluation
University of Illinois at Urbana-Champaign, USA
Staying on Task: Testing the Foundations of Long-Horizon Agent Reliability
Sep 30, 2026Jeffrey Willette, Krishna C. Puvvada, Boris GinsburgLong-Context Language Model Evaluation Long-Horizon Agent Evaluation
NVIDIA
LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
Sep 29, 2026Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen +1Long-Context Language Model Evaluation Long-Context Language Model Inference
University of Alberta · Alberta Machine Intelligence Institute (Amii)
CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles
Sep 28, 2026Mir Tafseer Nayeem, Susmoy Chakraborty, Davood RafieiMultilingual Language Model Evaluation Long-Context Language Model Evaluation
University of Alberta · Independent Researcher
The Model Knows When to Stop: Training-Free Early Stopping for Long-Context Reading
Sep 28, 2026Muath Alyobi, Mohamed Eltahir, Almoayyad Abuljdail +3Early Stopping Long-Context Language Model Evaluation
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England
No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow
Sep 24, 2026Prasann Singhal, Amanda Bertsch, Jacob Steinhardt +1Long-Context Language Model Evaluation LLM Reasoning
UC Berkeley · Carnegie Mellon University · Allen Institute for AI
MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors
Sep 23, 2026Wei He, Aline Villavicencio, Rodrigo Wilkens +1Multilingual Language Model Evaluation Long-Context Language Model Evaluation
University of Exeter · University of Cambridge
The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence
Sep 22, 2026Xiaoyu Yang, Jie Lu, Wei Duan +1Long-Context Language Model Evaluation Evidence Retrieval
Australian Artificial Intelligence Institute (AAII), Faulty of Engineering and Information Technology, University of Technology Sydney, Australia.
Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
Sep 14, 2026Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie +2Long-Context Language Model Evaluation
Manulife Pittsburgh, United States · Manulife Toronto, Canada · Manulife Bangalore, India
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sep 8, 2026Sara Rizwan, Samaanah Abdus SalamLong-Context Language Model Evaluation Long-Context Language Modeling
Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India
Separating Stream Stability from Long-Term Recall in Language Models
Sep 7, 2026Peipei Cao, Xin Zhang, Jie Tang +3Memory-Augmented Language Models Long-Context Language Model Evaluation
School of Computer and Electronic Information, Guangxi University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Guangdong University of Technology
Polish ModernBERT: The Long and Short of Polish Language Understanding
Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model Pretraining Long-Context Language Model Evaluation
National Information Processing Institute, Warsaw, Poland
VoiceLongMemEval: Do Assistants Remember How You Sounded?
Sep 1, 2026Ramit Pahwa, Parivesh Priye, Apoorva BeeduAudio-Language Model Evaluation Long-Context Language Model Evaluation
Rivian VW Tech
LongGuard: Mechanistic Analysis and Training-Free Mitigation of Long-Context Failure in Safety Guardrails
Aug 27, 2026Ziyang Chen, Xing Wu, Songlin HuLanguage Model Safety Evaluation Long-Context Language Model Evaluation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
Aug 10, 2026Chen Qiu, Ziwu Liu, Chao Fei +2Long-Context Language Model Evaluation KV Caching
KAUST Thuwal, Saudi Arabia
MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc
Aug 4, 2026Mingtian ZhangLLM Evaluation Long-Context Language Model Evaluation
PageIndex
Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks
Aug 4, 2026Mohsen ArjmandiLong-Context Language Model Evaluation Benchmark Design
evolutionID GmbH
Measuring Alignment With Reader Highlights Net of Position and Length
Jul 30, 2026Kazuki Nakayashiki, Keisuke WatanabeLLM Evaluation Long-Context Language Model Evaluation
Glasp Inc.
Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models
Jul 21, 2026Netanel EliavLLM Evaluation Language Model Generation Evaluation
Machine Human Intelligence Lab (MHIL)
Efficient Sequential Evaluation of Large Language Models
Jul 19, 2026Chia-Yu Hsu, Shubhanshu ShekharLLM Evaluation Confidence Estimation in Language Models
EECS Department, University of Michigan
Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
Jul 9, 2026Siddhartha Jain, Ameya VelingkerLLM Evaluation Long-Context Language Model Evaluation
NVIDIA
Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
Jul 1, 2026Aryo Pradipta Gema, Beatrice Alex, Pasquale MinerviniLong-Context Retrieval Attention Head Analysis
QUniversity of Edinburgh · KHeriot-Watt University · V Miniml.AI
TriggerBench: Investigating Prospective Memory for Large Language Models
Jun 22, 2026Tianhua Zhang, Xinjiang Wang, Qianxi Zhang +6LLM Evaluation Long-Context Language Model Evaluation
The Chinese University of Hong Kong · Microsoft Research Asia
A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization
Jun 14, 2026Weixiao Zhou, Gengyao Li, Xianfu Cheng +3LLM Evaluation Long-Context Language Model Evaluation
CCSE, Beihang University · Fanyu AI Laboratory · MAIS, CASIA
ATLAS: All-round Testing of Long-context Abilities across Scales
May 27, 2026Deli Huang, Cunguang Wang, Hongyin Tang +15LLM Evaluation Benchmark Construction
Meituan · Fudan University
Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks
May 22, 2026Chuyifei Zhang, Hongyu Cui, Xiaowen Huang +1LLM Evaluation Long-Context Language Model Evaluation
Beijing Jiaotong University · Central South University of Forestry and Technology
Classifier Context Rot: Monitor Performance Degrades with Context Length
May 12, 2026Sam Martin, Fabien RogerLanguage Model Safety Evaluation Long-Context Language Model Evaluation
Anthropic Fellows Program · Anthropic
Retrieval and Multi-Hop Reasoning in 1M-Token Context Windows: Evaluating LLMs on Classical Chinese Text
May 4, 2026Eric H. C. ChowLong-Context Retrieval In-Context Retrieval
School of Humanities, The University of Hong Kong
Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization
Jan 7, 2026Yao Dou, Benjamin Mamut, Wei XuLegal NLP LLM Evaluation
Georgia Institute of Technology

Sep 30, 2026, cs.CLJ/K move · Enter open · S save

Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane, Sudip Mittal+1

Department of Computer Science, The University of Alabama

Long-Context Language Model Evaluation Clinical Reasoning LLM Context Management Medical LLMs

Open paper AskPDF arXiv

Longitudinal clinical reasoning requires large language models (LLMs) to identify and integrate relevant evidence distributed across extended patient histories. Although long-context models can process increasingly large amounts of information, providing more history does not necessarily make relevant evidence more accessible or improve reasoning. We compare five context strategies (Full, Recent, Episodic, Semantic, and Hybrid) on MedLoCoMo across four open-weight LLMs, examining answer correctness, robustness to query-evidence distance, and abstention on questions with unsupported premises. Episodic and Hybrid generally achieve the strongest overall accuracy, while Recent Context degrades most as supporting evidence becomes more distant; Episodic and Hybrid maintain the highest accuracy at long distances. Analysis of adversarial questions further shows that strong performance on answerable questions does not necessarily translate to successful abstention when the available history does not support the requested conclusion. These findings show that reliable longitudinal reasoning depends not only on how much history an LLM can access, but critically on how relevant evidence is selected and presented for reasoning.

Momentum

Related topics

Latest papers 31