Long-Context VLMs

VLM: Vision-Language Model

Long-context VLMs study how models can find and reason over relevant visual evidence in long documents, videos, and ongoing multimodal conversations. Approaches include extending context during training, retrieving relevant pages or frames, compressing visual tokens, and adding structured or recurrent memory and tool-using agents. Evaluation spans document question answering benchmarks such as MMLongBench-Doc and long-video benchmarks such as VideoMME and LongVideoBench, with growing attention to evidence grounding and multi-session memory. The central tension is how to reduce context and memory costs without discarding fine-grained details or the connections needed for reliable reasoning.

Latest papers

46 papers9 papers in the last four weeks, up 13% on the four weeks before. 0.1% of all new papers.

Memory Internalization via Test-Time Training for Long-Video Understanding
Oct 17, 2026Haonan Zhao, Jiaxuan Chen, Qingyang Liu +4Memory-Augmented Video Understanding Long-Video QA
Shanghai Jiao Tong University · Tongji University
DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding
Oct 15, 2026Zhuowen Liang, Xiaotian Lin, Yuyu Luo +2Document QA Multimodal Agents
The Hong Kong University of Science and Technology (Guangzhou) · EvenUp, USA
WinARDet: Compressed Window Auto-Regression for Long-Horizon Vision-Language Grounding
Oct 14, 2026Chenxu Wang, Hangyi Kuang, Bo-Wen Yin +6Vision-Language Grounding Autoregressive Decoding
Nankai University · Beijing Normal University
MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation
Oct 13, 2026Xueyao Wei, Yifei Wang, Delun Li +4Vision-Language Navigation Memory-Augmented VLMs
Beijing Institute of Technology · Zhongguancun Academy · Peking University
Heading Hierarchy Reconstruction for Complex and Long Documents with Level-Wise Multimodal Memory
Oct 13, 2026Qian Chen, Xianyin Zhang, Lifan Guo +1Document Layout Analysis Memory-Augmented VLMs
Qwen DianJin Team, Alibaba Cloud Computing
TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering
Oct 7, 2026Jingwen Huang, Yiming Fang, Hanxiao Wu +2Remote Sensing VQA Visual Question Answering
Yunnan University of Finance and Economics, Kunming, China
ReMem: Streaming Video Understanding With Long Context Retention
Oct 5, 2026Li Yiheng, He Xu, Wang Shaobo +2Memory-Augmented VLMs Streaming Video Understanding
Nanyang Technological University · Shanghai Jiaotong University · University of the Chinese Academy of Sciences
VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding
Sep 29, 2026Susan Liang, Jianmin Wu, Daxiang DongLong-Video Understanding LLM Agent Harnesses
Baidu, Inc.
FocusVTC: Efficient and High-Performance Visual Text Compression with Adaptive Resolution
Sep 29, 2026FangZhi Zhong, Xuerui Qiu, Yuqi Pan +4Efficient VLM Inference VLM Reasoning
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding
Sep 26, 2026Zihan Chen, Xuejian Rong, Xiaojuan Wang +4Long-Video Understanding Memory-Augmented Video Understanding
University of Virginia · Google
An Empirical Study of VLM Pipelines for Long-Document QA
Sep 24, 2026Kenan E. Ak, Jay Mohta, Gwang Gook Lee +2VLM Evaluation Tool Use in VLMs
PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding
Sep 20, 2026Siru Zhong, Qiongyan Wang, Xiaohui Lv +5Memory-Augmented VLMs Recurrent Neural Networks
The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University
Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?
Sep 15, 2026Zhaoyang Wei, Zipeng Wang, Yushe Cao +10Audio-Visual Understanding Video-Language Model Evaluation
University of Chinese Academy of Sciences, China · Tsinghua University, China · Tencent, China
VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding
Sep 14, 2026Weixin Xu, Zhenyu Yang, Bing Wang +2Video Understanding Long-Video Understanding
Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering
Sep 7, 2026Amirul Rahman, Aisha Karim, Kenji Nakamura +1Document QA Long-Document VQA
University of Malaya
Ambient @ EgoLongQA 2026: Distilling Long-Video perception into a Sub-2B Model
Sep 7, 2026Logesh Kumar UmapathiEfficient VLM Inference Egocentric Video QA
Team Ambient
Dual-Latent Memory Routing for Vision-Language Reasoning
Sep 2, 2026Hao-Xuan Ma, Jin-Fei Qi, Yicheng Xiao +1Memory-Augmented VLMs VLM Reasoning
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models
Aug 25, 2026Guoyang Xu, Hao ChenProgram Synthesis Long-Video Understanding
Tencent
VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference
Aug 25, 2026Lyuke Wang, Zhuo Li, Guangxu ZhuEfficient VLM Inference Long-Context VLMs
Shenzhen International Center for Industrial and Applied Mathematics · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen +1
LongEarth-R1: Benchmarking and Aligning Vision-Language Models for Long-Horizon Earth Observation Reasoning
Aug 13, 2026Yupan Ding, Jing Xiao, Zhenyuan Zhang +4Remote Sensing Remote Sensing VQA
School of Artificial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · Xi’an University of Electronic Science and Technology, Xi’an, China +1
VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling
Aug 9, 2026Yuqi Zhang, Cheng Chen, Yuyu Guo +6Vision-Language Models Efficient VLM Inference
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University, China · Shanghai Innovation Institute, China · Ant Group, China
Linear Multi-Timescale Retention as a Memory-Efficient Vision-Language Bridge
Aug 3, 2026Ashfak Yeafi, Mehedi Hasan, Md Khairul IslamMemory-Augmented VLMs Vision-Language Models
Department of Electrical and Electronic Engineering Khulna University of Engineering & Technology Khulna-9203, Bangladesh · Department of Computer Science and Engineering Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science Hobart and William Smith Colleges Geneva, NY, USA
ReToken: Improving Long-Context VLMs with Visual Retrieval Token
Jul 30, 2026Yao Xiao, Reuben Tan, Zhen Zhu +3Vision-Language Models Vision-Language Retrieval
University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind
HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding
Jul 1, 2026Ji Ha Jang, Hayeon Kim, Chulwon Lee +2VLM Robustness Vision-Language Models
1Dept. of Electrical and Computer Engineering, 2IPAI, 3INMC & AIIS, Seoul National University, Republic of Korea
Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
May 26, 2026Aaron Branson Cigres Li, Zhaowei Wang, Yu Zhao +9Vision-Language Models VLM Interpretability
1HKUST · University of Edinburgh · 4CUHK +2
MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
May 14, 2026Xiyu Ren, Zhaowei Wang, Yiming Du +11VLM Evaluation Memory-Augmented VLMs
CSE Deparment, HKUST · CUHK · OmniMemory (Shenzhen) Intelligent Technology Co., Ltd. +1
Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
May 13, 2026Zhaowei Wang, Lishu Luo, Haodong Duan +9Visual Question Answering Vision-Language Models
CSE Department, HKUST · ByteDance Seed
One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
Apr 15, 2026Zheyu Zhang, Ziqi Pang, Shixing Chen +3Long-Video Understanding Video Token Compression
University of Illinois Urbana-Champaign · Amazon Prime Video
MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning
Apr 15, 2026Jiahang Lin, Kai Hu, Binghai Wang +12Long-Document VQA Multi-Turn RL
Fudan University · Shanghai Qiji Zhifeng Co., Ltd
DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding
Apr 14, 2026Hao Yan, Yuliang Liu, Xingchen Liu +5Document Understanding Long-Document VQA
Huazhong University of Science and Technology · Huawei Inc.

Oct 17, 2026J/K move · Enter open · S save

Memory Internalization via Test-Time Training for Long-Video Understanding

Haonan Zhao, Jiaxuan Chen, Qingyang Liu, Shuochen Chang+3

Shanghai Jiao Tong University · Tongji University

Memory-Augmented Video Understanding Long-Video QA Long-Context VLMs Test-Time Adaptation of VLMs Test-Time Training

Open paper AskPDF arXiv

Long-video understanding often requires answering many questions about a single video whose evidence is distributed across extended temporal contexts. Conventional video-language models repeatedly encode frames or retrieve visual evidence for every question. Parameterized video memory offers a write-once alternative, but existing methods rely primarily on synthesized text and fixed adaptation configurations, leaving open where video knowledge should be written and how visual evidence should shape the memory. We introduce Memory Internalization via Test-Time Training (MIT^3), a per-video adaptation framework that writes each long video into compact, video-specific adaptive parameters for video-free, text-only question answering. Under compact multi-scale supervision, causal tracing measures the causal memory utility of each model layer for video-grounded answering and selects video-specific adaptation locations. Visual memory distillation then trains the text-only memory to reproduce the representation change induced by the relevant video clip, preserving the internal effect of visual evidence beyond textual supervision. Across LVBench, LongVideoBench, and Video-MME, MIT^3 achieves data-efficient video-free question answering with substantially fewer synthesized samples than prior parameterized video-memory methods, demonstrating that effective video internalization depends on both where knowledge is written and how visual evidence is distilled into the model.