Long-Context Reasoning in Language Models

Latest papers

41 papers11 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Internalized Reasoning for Long-Context Visual Document Understanding
Mar 31, 2026Austin VeselkaMultimodal Understanding Document Image Understanding
LightOn
Probing How Scalable Table Data Enhances General Long-Context Reasoning
Mar 23, 2026Huaibing Xie, Guoliang Zhao, Yang Liu +8Long-Context Reasoning in Language Models Synthetic Data Generation for Language Models
Large Language Model Department, Tencent · Xi’an Jiaotong University, Xi’an, China · Fudan University, Shanghai, China
HiCI: Hierarchical Construction-Integration for Long-Context Attention
Mar 21, 2026Xiangyu Zeng, Qi Xu, Yunke Wang +1Long-Context Language Modeling Long-Context Modeling
School of Computer Science, University of Sydney, Sydney, Australia · 2Westlake University, Hangzhou, China.
The $\mathbf{Y}$ -Combinator for LLMs: Solving Long-Context Rot with $λ$ -Calculus
Mar 20, 2026Amartya Roy, Rasul Tutunov, Xiaotong Ji +2Long-Context Reasoning in Language Models Long-Context Language Model Inference
The School of Interdisciplinary Research, Indian Institute of Technology (IIT) Delhi, Robert Bosch GmbH, India · Huawei Noah’s Ark Lab · Huawei Noah’s Ark +1
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
Recursive Models for Long-Horizon Reasoning
Mar 2, 2026Chenxiao Yang, Nathan Srebro, Zhiyuan LiRecursive Language Models Language Modeling
Toyota Technological Institute at Chicago
DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models
Feb 25, 2026Xi Ye, Wuwei Zhang, Fangcong Yin +2Long-Context Reasoning in Language Models Language Model Decoding
Princeton Language and Intelligence, Princeton University · Department of Computer Science, New York University
The Limits of Long-Context Reasoning in Automated Bug Fixing
Feb 17, 2026Ravi Raju, Mengmeng Ji, Shubhangi Upasani +2Long-Context Reasoning in Language Models Long-Context Language Model Evaluation
SambaNova Systems San Jose, CA 95134, USA
LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards
Feb 5, 2026Qize Yu, Zijun Chen, Chen Wu +6Long-Context Reasoning in Language Models RL for Language Model Reasoning
Shanghai Jiao Tong University · Tsinghua University · Beijing University of Posts and Telecommunications, Beijing, China +1
Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
Nov 4, 2025Amanda Bertsch, Adithya Pratapa, Teruko Mitamura +2LLM Evaluation Long-Context Language Model Evaluation
Carnegie Mellon University, Language Technologies Institute
PERK: Long-Context Reasoning as Test-Time Learning
Jul 8, 2025Zeming Chen, Angelika Romanou, Gail Weiss +1Memory-Augmented Language Models Long-Context Modeling
EPFL

Mar 31, 2026J/K move · Enter open · S save

Internalized Reasoning for Long-Context Visual Document Understanding

Austin Veselka

LightOn

Multimodal Understanding Document Image Understanding Long-Context VLMs Long-Context Modeling VLM Reasoning Multimodal CoT Reasoning+2

Open paper AskPDF arXiv

Visual long-document understanding is critical for enterprise, legal, and scientific applications, yet the best performing open recipes have not explored reasoning, a capability which has driven leaps in math and code performance. We introduce a synthetic data pipeline for reasoning in long-document understanding that generates thinking traces by scoring each page for question relevance, extracting textual evidence and ordering it from most to least relevant. We apply SFT to the resulting traces within <think> tags, gated by a <cot> control token, and the resulting reasoning capability is internalized via low-strength model merging. We study Qwen3 VL 32B and Mistral Small 3.1 24B. With Qwen3 VL, we achieve 58.3 on MMLongBenchDoc, surpassing the 7

\times

larger Qwen3 VL 235B A22B (57.0). With Mistral, we show that synthetic reasoning outperforms distillation from the Thinking version's traces by 3.8 points on MMLBD-C, and internalized reasoning exhibits 12.4

\times

fewer mean output tokens compared to explicit reasoning. We release our pipeline for reproducibility and further exploration.