Long-Context Language Model Evaluation

Overview

This field tests whether language models can retrieve evidence, combine information across distant passages, follow long conversations, and answer reliably as inputs grow. Researchers use benchmarks with varied context lengths, evidence positions, distractors, reasoning steps, and tasks such as summarization, code question answering, and multi-turn memory; some also test audio, images, or real-world procedures. Controlled comparisons, evidence annotations, agent harnesses, and mechanistic diagnostics help separate retrieval failures from problems with reasoning, attention, or context compression. A central finding is that advertised context length does not guarantee effective use of the whole input, while benchmark scores can be confounded by task difficulty, position, and distractor design.

Latest papers

CardsList

75 papers7 papers in the last four weeks, down 59% on the four weeks before. 0.1% of all new papers.

  1. Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)

    Nov 17, 2026Allen G. Roush, Minh Nhat Nguyen, Ravid Shwartz-Ziv +2Language Model DecodingText Generation

  2. Where in the Input Does a One-Pass Decision Encoder Look?

    Nov 12, 2026Mohammed Nayeem Khan, Allan Kevin TLong-Context Language Model Evaluation

  3. CoRA-Bench: Evaluating Evidence Preservation by Context Selectors Under Token Budgets

    Nov 6, 2026Huzaifa Pardawala, Siddhant Sukhani, Sudheer Chava +1Evidence SelectionLong-Context Language Model Evaluation

  4. ASCENT: Scale-Complementary External State for Frozen Long-Context Language Models

    Nov 6, 2026Xingjian ZhangMemory-Augmented Language ModelsLong-Context Language Modeling

  5. Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

    Nov 3, 2026Yinuo LiuKnowledge Retrieval in Language ModelsLong-Context Language Model Evaluation

  6. Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

    Oct 22, 2026Aman Behera, Namit Solanki, Mehul AnandState Space Language ModelsLong-Context Language Modeling

  7. LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking

    Oct 19, 2026Howard Yen, Nitya Mani, Alex Renda +1State Tracking in Language ModelsLong-Context Language Model Evaluation

  8. DiaLong: Benchmarking Reminiscence in Long-Context Dialogues

    Oct 18, 2026Owen Bianchi, Samer Aslan, Ryunosuke SaitoConversational MemoryLong-Context Language Model Evaluation

  9. Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

    Oct 15, 2026Shiwen Qin, Cameron Barker, Henry Gouk +4Long-Context Reasoning in Language ModelsLong-Context Language Model Evaluation

  10. LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads?

    Oct 2, 2026Xinyi Liu, Rinat Khaziev, Dilek Hakkani-Tür +1Social ReasoningLong-Context Language Model Evaluation

  11. TPBench: A Turning-Point Benchmark for Dialogue Compression

    Oct 2, 2026Minji Park, Seunghyun Yoon, Hyuk LimMulti-Turn Dialogue EvaluationSpoken Dialogue Systems

  12. Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

    Sep 30, 2026Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane +2Long-Context Language Model EvaluationClinical Reasoning

  13. LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

    Sep 29, 2026Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen +1Long-Context Language Model EvaluationLong-Context Language Model Inference

  14. The Model Knows When to Stop: Training-Free Early Stopping for Long-Context Reading

    Sep 28, 2026Muath Alyobi, Mohamed Eltahir, Almoayyad Abuljdail +3Early StoppingLong-Context Language Model Evaluation

  15. VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

    Sep 26, 2026Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden +1Audio-Language Model EvaluationLong-Term Conversational Memory

  16. Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension

    Sep 26, 2026Kohei Kajikawa, Lin Ai, Tatsuki Kuribayashi +1Computational PsycholinguisticsLong-Context Language Modeling

  17. No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow

    Sep 24, 2026Prasann Singhal, Amanda Bertsch, Jacob Steinhardt +1Long-Context Language Model EvaluationLLM Reasoning

  18. MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors

    Sep 23, 2026Wei He, Aline Villavicencio, Rodrigo Wilkens +1Multilingual Language Model EvaluationLong-Context Language Model Evaluation

  19. The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence

    Sep 22, 2026Xiaoyu Yang, Jie Lu, Wei Duan +1Long-Context Language Model EvaluationEvidence Retrieval

  20. Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models

    Sep 11, 2026Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie +2Long-Horizon LLM ReasoningLong-Context Language Model Evaluation

  21. Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?

    Sep 8, 2026Sara Rizwan, Samaanah Abdus Salam, Mohammed Mudassir UddinLong-Context Language Model EvaluationLong-Context Language Modeling

  22. Separating Stream Stability from Long-Term Recall in Language Models

    Sep 7, 2026Peipei Cao, Xin Zhang, Jie Tang +3Memory-Augmented Language ModelsLong-Context Language Model Evaluation

  23. HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Care

    Sep 7, 2026Yuchen Niu, Yanan Ma, Srinivasan Nandakumar +7Clinical Language Model EvaluationWearable Health Monitoring