LVMem: Image-Augmented Memory for Long-Horizon Question Answering

Victoria Khoruzhaya; Maksim Kalmykov; Tuong Do Minh; Tatiana Zemskova; Yixin Zhu; Dmitry Yudin

cs.CVOct 19, 2026

LVMem: Image-Augmented Memory for Long-Horizon Question Answering

Authors: Victoria Khoruzhaya, Maksim Kalmykov, Tuong Do Minh, Tatiana Zemskova, Yixin Zhu, Dmitry Yudin

Organizations: Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia · Cognitive AI Systems Lab, Moscow, Russia · Peking University, Beijing, China

Abstract

Reliable language-guided perception is essential for embodied agents, where questions about objects, events, and their locations must be answered from long observation histories. Unlike conventional visual question answering, realistic robotic settings require reasoning over extended temporal horizons and persistent scene memory. We build upon the view-based memory paradigm and propose an efficient framework for long-horizon spatio-temporal question answering. While existing view-based systems typically store either text captions or visual embeddings, we extend the ReMEmbR-style memory with image paths that link each stored description to its original visual content. To make the stored annotations more informative, we replace uniform sampling during memory construction with an optical-flow-based frame selection strategy (PSFR) and store an object-centric descriptive scene representation instead of plain captions. We further equip a VLM-based agent with an adaptive retrieval tool that returns the image associated with the most relevant caption, allowing the agent to decide when visual inspection is needed. On the OC-NaVQA benchmark, LVMem outperforms ReMEmbR on binary and positional metrics with both backbones and matches or exceeds a 4D scene graph DAAAM baseline on binary accuracy, while operating on a lightweight view-based memory that requires no explicit graph construction.

Figures & tables

Fig. 1: Proposed LVMem approach. A long video sequence is divided into fixed-length segments, from which PSFR selects the most informative frames. An object-centric textual scene representation generator produces one scene description per segment from its selected frames, which is stored in memory alongside the segment’s timestamp, pose, and image path. For a given query, the VLM agent calls the image retrieval tool to obtain the image linked to the most relevant description, enabling joint reasoning over visual and textual evidence.

Fig. 2: Overview of the proposed pipeline. Each temporal segment is processed by PSFR, which selects the most informative frames; the first selected frame is stored as an image path alongside the resulting object-centric scene description. At query time, the VLM agent invokes a visual retrieval tool that returns the image paired with the most relevant description, enabling joint reasoning over textual and visual evidence to reason jointly over both modalities.

Method	Binary [%] $\uparrow$	Text [%] $\uparrow$	Pos. [m] $\downarrow$	Temp. [min] $\downarrow$	Dur. [min] $\downarrow$	Latency [s] ↓
ReMEmbR (GPT-5-Mini)	56.72 $\pm$ 5.38	36.36 $\pm$ 3.03	52.69 $\pm$ 5.49	5.19 $\pm$ 1.01	3.66 $\pm$ 0.99	-
DAAAM (GPT-5-Mini)	73.30 $\pm$ 2.94	42.30 $\pm$ 21.35	56.37 $\pm$ 6.38	3.83 $\pm$ 1.50	2.58 $\pm$ 1.72	-
LVMem (Ours, GPT-5-Mini)	76.62 $\pm$ 3.76	41.41 $\pm$ 6.31	34.68 $\pm$ 0.36	4.81 $\pm$ 0.61	4.88 $\pm$ 0.51	-
ReMEmbR (Qwen3.5-9B)	65.17 $\pm$ 3.11	24.24 $\pm$ 3.03	68.67 $\pm$ 12.76	16.95 $\pm$ 20.46	4.96 $\pm$ 2.22	19.19 $\pm$ 5.05
DAAAM (Qwen3.5-9B)	43.50 $\pm$ 10.97	23.09 $\pm$ 2.97	65.92 $\pm$ 7.92	7.86 $\pm$ 1.98	3.87 $\pm$ 2.99	30.67 $\pm$ 38.76
LVMem (Ours, Qwen3.5-9B)	76.12 $\pm$ 2.59	39.39 $\pm$ 3.03	50.12 $\pm$ 4.96	4.23 $\pm$ 0.54	3.67 $\pm$ 2.30	20.91 $\pm$ 5.33

TABLE I: Comparison on OC-NaVQA. Mean

\pm

std. dev. over 3 runs. ReMEmbR uses its original configuration, LVMem our proposed representation. Segment parameters are identical (3 seconds, 6 frames). Latency is measured on a single H100 GPU and reported only for local inference, as API-based runs incur variable network and queueing overhead.

Fig. 3: A question asking for the color of a horse sculpture. Text-based baselines describe the sculpture as “metal” and “reflective” and cannot determine its color. LVMem retrieves the corresponding frame and answers correctly based on the visible brown tone.

Sampling	Binary [%] $\uparrow$	Text [%] $\uparrow$	Pos. [m] $\downarrow$	Temp. [min] $\downarrow$	Dur. [min] $\downarrow$
uniform	55.72 $\pm$ 5.65	22.22 $\pm$ 1.75	89.98 $\pm$ 9.05	5.60 $\pm$ 2.26	4.27 $\pm$ 0.12
PSFR	64.68 $\pm$ 4.80	25.25 $\pm$ 1.75	68.16 $\pm$ 12.96	4.71 $\pm$ 2.66	3.83 $\pm$ 0.99

TABLE II: Effect of PSFR on OC-NaVQA performance for LVMem with Qwen3.5-9B. Segment parameters: 10 seconds, 6 frames

Mode	$k$	Img	Bin. [%] $\uparrow$	Text [%] $\uparrow$	Pos. [m] $\downarrow$	Temp. [min] $\downarrow$	Dur. [min] $\downarrow$
top-M	5	0	66.70 $\pm$ 0.90	34.30 $\pm$ 7.60	60.30 $\pm$ 0.20	4.16 $\pm$ 0.23	5.92 $\pm$ 2.02
top-M	20	0	67.20 $\pm$ 3.00	30.30 $\pm$ 3.00	54.50 $\pm$ 16.90	3.39 $\pm$ 0.67	4.53 $\pm$ 0.12
tool	5	1	69.65 $\pm$ 1.72	28.28 $\pm$ 1.75	68.15 $\pm$ 4.72	5.43 $\pm$ 1.69	4.99 $\pm$ 1.73
top-M	5	1	69.20 $\pm$ 3.80	38.40 $\pm$ 1.70	78.20 $\pm$ 3.90	2.03 $\pm$ 0.18	3.79 $\pm$ 0.10
top-M	20	1	68.20 $\pm$ 2.30	32.30 $\pm$ 6.30	72.50 $\pm$ 6.70	6.50 $\pm$ 2.68	4.19 $\pm$ 0.67
tool	5	3	74.13 $\pm$ 2.28	22.22 $\pm$ 1.75	67.53 $\pm$ 4.82	4.75 $\pm$ 1.27	3.78 $\pm$ 0.34

TABLE III: Joint effect of text retrieval depth

k

and the number of images

M

on OC-NaVQA with LVMem (Qwen3.5-9B). In the top-M mode, images from the top-

M

retrieved descriptions are always appended directly to the generation prompt. In the tool mode, the agent instead invokes the adaptive visual retrieval tool, which returns the images linked to the top retrieved descriptions. Segment parameters: 3 seconds, 6 frames.

Scene description model	Prompt	Images	Binary [%] $\uparrow$	Text [%] $\uparrow$	Pos. [m] $\downarrow$	Temp. [min] $\downarrow$	Dur. [min] $\downarrow$
VILA1.5-3b	ReMEmbR	0	66.70 $\pm$ 0.90	34.30 $\pm$ 7.60	60.30 $\pm$ 0.20	4.16 $\pm$ 0.23	5.92 $\pm$ 2.02
		1	69.20 $\pm$ 3.80	38.40 $\pm$ 1.70	78.20 $\pm$ 3.90	2.03 $\pm$ 0.18	3.79 $\pm$ 0.10
		3	73.00 $\pm$ 1.70	33.30 $\pm$ 3.00	77.10 $\pm$ 8.00	4.43 $\pm$ 0.77	4.34 $\pm$ 0.67
		5	77.30 $\pm$ 2.50	34.10 $\pm$ 1.20	71.70 $\pm$ 3.70	4.73 $\pm$ 0.46	4.25 $\pm$ 1.03
Qwen3.5-9B	LVMem	0	76.10 $\pm$ 2.10	31.10 $\pm$ 1.50	43.60 $\pm$ 3.00	15.98 $\pm$ 22.23	4.46 $\pm$ 1.43
		1	81.00 $\pm$ 5.40	31.80 $\pm$ 3.90	53.30 $\pm$ 4.60	3.82 $\pm$ 1.10	4.68 $\pm$ 0.65

TABLE IV: Effect of different scene description models and prompts within our LVMem pipeline on OC-NaVQA at

k{=}5

. Images from top-

M

retrieved documents are attached at generation; the visual retrieval tool is not used. Segment parameters: 3 seconds, 6 frames.

Embodied QA

Embodied question answering studies how agents use perception, memory, and action to answer questions grounded in the physical world.

See all 39 papers

PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agents
Aug 2, 2026Jingyu Sun, Yan Lin, Yuyang Xue +10Memory-Augmented VLMs Multimodal Memory
The University of Manchester · The University of Newcastle · The University of Edinburgh +5
RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memory
Jun 23, 2026Yixun Hu, Zhicheng Zheng, Lihan Zha +5Robot Navigation Embodied QA
Princeton University · University of Pennsylvania
VL-MemKnG: Hybrid Memory with a Spatio-Temporal Knowledge Graph for Question Answering over Long Egocentric Navigation Trajectories
Jun 15, 2026Svetlana Lukina, Mohamad Al Mdfaa, Gloria Haro +2Egocentric Video QA Video QA
Mobile Robotics Laboratory, Artificial Intelligence Center, Moscow, Russia · Intelligent Multimodal Vision Analysis Group, Department of Engineering, Universitat Pompeu Fabra, Barcelona, Spain · Independent Researcher

Aug 2, 2026J/K move · Enter open · S save

PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agents

Jingyu Sun, Yan Lin, Yuyang Xue, Yifan Wang+9

The University of Manchester · The University of Newcastle · The University of Edinburgh+5

Memory-Augmented VLMs Multimodal Memory Persistent Memory for Language Models Multimodal QA Long-Horizon LLM Agents Memory Consolidation

Open paper AskPDF arXiv

Long-term memory is essential for LVLM agents to maintain consistency and integrate information across extended multimodal interactions. Existing agent memory systems, however, often reduce visual experiences into textual summaries or rely on static retrieve-then-reason pipelines, which are inefficient at query time and brittle when questions require image-text binding, temporal updates, or visual details. We propose Prospective Multimodal Memory Compilation, a framework that shifts part of the memory reasoning process from query time to memory consolidation time. Given accumulated multimodal interactions, a Questioner predicts future question candidates, a Planner compiles question-conditioned multimodal memory programs, and a Doubter verifies whether the planned evidence path can support the predicted answer. The verified question-program pairs form a structured question bank for efficient query-time routing and evidence retrieval. Experiments on multimodal long-term memory benchmarks show that our method improves answer quality and visual evidence recall while reducing query-time token and latency costs. Extensive ablations analyze the effects of self-feedback, dynamic planning, raw-image access, and question bank coverage.

Abstract

Embodied QA

Explore similar work

PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agents

RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memory

VL-MemKnG: Hybrid Memory with a Spatio-Temporal Knowledge Graph for Question Answering over Long Egocentric Navigation Trajectories