Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge

Giuseppe Lando; Rosario Forte; Antonino Furnari

cs.CVFeb 25, 2026

Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge

Authors: Giuseppe Lando, Rosario Forte, Antonino Furnari

Organizations: Department of Mathematics and Computer Science, University of Catania, Italy

Abstract

We investigate the feasibility of using Multimodal Large Language Models (MLLMs) for real-time online episodic memory question answering. While cloud offloading is common, it raises privacy and latency concerns for wearable assistants, hence we investigate implementation on the edge. We integrated streaming constraints into our question answering pipeline, which is structured into two asynchronous threads: a Descriptor Thread that continuously converts video into a lightweight textual memory, and a Question Answering (QA) Thread that reasons over the textual memory to answer queries. Experiments on the QAEgo4D-Closed benchmark analyze the performance of Multimodal Large Language Models (MLLMs) within strict resource boundaries, showing promising results also when compared to clound-based solutions. Specifically, an end-to-end configuration running on a consumer-grade 8GB GPU achieves 51.76% accuracy with a Time-To-First-Token (TTFT) of 0.41s. Scaling to a local enterprise-grade server yields 54.40% accuracy with a TTFT of 0.88s. In comparison, a cloud-based solution obtains an accuracy of 56.00%. These competitive results highlight the potential of edge-based solutions for privacy-preserving episodic memory retrieval.

Figures & tables

Figure 1: High-level overview of the proposed Edge-based OEM-VQA system. The user wears smart glasses continuously streaming video to a local unit (GPU). The video is continuously processed into a textual memory M by the Descriptor Thread, allowing the user to ask questions that are ingested into the QA Thread that leverages the textual memory to reply to the user, sending the answer without storing raw video frames.

Figure 2: Overview of the Streaming OEM-VQA Framework. The architecture is organized into two asynchronous threads: Descriptor Thread : Processes handles the continuous streamed video clips ( $c_{k}$ ) of $s$ seconds. A Video LLM Descriptor generates a textual description ( $d_{k}$ ) for each clip in execution time $T_{\text{des}}$ , incrementally populating the semantic Memory $M$ . QA Thread : Activated upon user query, this thread utilizes the stored textual Memory $M$ and a Reasoner model to deduce the Answer in time $T_{\text{ans}}$ .

Figure 3: Overview of the adopted prompting strategy. The Descriptor prompt consists of four main components: 1) Task Description : instructs the model on the specific task to perform; 2) Detailed Instructions : provides specific guidelines, such as prioritizing actions or spatial positioning; 3) Question Template : primes the model with potential future questions; and 4) In-Context Learning Examples : provides a full clip description example to encourage adherence to output guidelines. Additionally, the 5) Reasoner Prompt is used at query time, providing the model with the question, candidate answers, and the accumulated memory history.

Size	BS	Time/Clip (s) $\downarrow$	Tok/s $\uparrow$	Peak Mem (GB) $\downarrow$
2B	1	7.30 $\pm$ 1.62	32.68 $\pm$ 3.91	6.22 $\pm$ 0.07
2B	2	5.89 $\pm$ 1.04	50.25 $\pm$ 11.28	8.51 $\pm$ 0.12
2B	4	7.10 $\pm$ 0.86	48.24 $\pm$ 0.81	13.24 $\pm$ 0.01
4B	1	9.76 $\pm$ 2.12	29.71 $\pm$ 4.98	10.73 $\pm$ 0.03
4B	2	7.18 $\pm$ 0.71	35.25 $\pm$ 2.07	13.81 $\pm$ 0.01
4B	4	6.19 $\pm$ 0.75	54.40 $\pm$ 5.59	19.96 $\pm$ 0.02

Table 1: Configuration selection on the enterprise setting (L40S, 48GB). The model is Qwen3-VL. We report mean

\pm

std for time per clip, generation throughput, and peak GPU memory. The bold row denotes the selected configuration.

Size	BS	Time/Clip (s) $\downarrow$	Tok/s $\uparrow$	Peak Mem. (GB) $\downarrow$
2B	1	7.30 $\pm$ 1.62	32.68 $\pm$ 3.91	6.22 $\pm$ 0.07
2B	2	5.89 $\pm$ 1.04	50.25 $\pm$ 11.28	8.51 $\pm$ 0.12
2B	4	7.10 $\pm$ 0.86	48.24 $\pm$ 0.81	13.24 $\pm$ 0.01
4B	1	9.76 $\pm$ 2.12	29.71 $\pm$ 4.98	10.73 $\pm$ 0.03
4B	2	7.18 $\pm$ 0.71	35.25 $\pm$ 2.07	13.81 $\pm$ 0.01
4B	4	6.19 $\pm$ 0.75	54.40 $\pm$ 5.59	19.96 $\pm$ 0.02

Table 2: Configuration selection on the enterprise setting (L40S, 48GB). The model is Qwen3-VL. We report mean

\pm

std for time per clip, throughput, and peak GPU memory. The bold row denotes the selected configuration.

GPU	Model	TTFT (s) $\downarrow$
RTX 3070 (8GB)	Qwen3-VL-2B	0.41 $\pm$ 0.27
	Qwen3-VL-4B	OOM
	Qwen3-VL-8B	OOM
L40S (48GB)	Qwen3-VL-2B	0.20 $\pm$ 0.14
	Qwen3-VL-4B	0.49 $\pm$ 0.31
	Qwen3-VL-8B	0.88 $\pm$ 0.57

Table 3: Query-time responsiveness measured as time-to-first-token (TTFT, mean

\pm

std) under different deployment settings. Larger models exceed the available memory on the consumer-grade GPU and result in out-of-memory (OOM) errors.

Descriptor	Reasoner	On Edge	Accuracy (%) $\uparrow$
2B	2B	Yes	51.76 $\pm$ 0.91
2B	8B	No	52.54 $\pm$ 1.11
8B	2B	No	51.32 $\pm$ 1.33
8B	8B	No	54.40 $\pm$ 0.88

Table 4: Main results on QAego4D-Closed test set (500 questions). All configurations use Qwen3-VL Instruct models (2B: Qwen3-VL-2B-Instruct, 8B: Qwen3-VL-8B-Instruct). Accuracy is reported as mean

\pm

std over 10 runs with different seeds. “On Edge” indicates whether the full pipeline can run end-to-end on an RTX 3070 (8GB).

Method	Accuracy (%) $\uparrow$
Ground VQA [ 4 ]	48.70
RekV-LLavaOneVision 0.5 [ 5 ]	50.00
RekV-LLaVaOneVision 7B [ 5 ]	56.00
LLaVaOneVision-based [ 9 ]	51.88
Gemini-based [ 9 ]	56.00
Ours (Edge-best: 2B Desc + 2B Reasoner)	51.76 $\pm$ 0.91

Table 5: Comparison with state-of-the-art methods on QAego4D-Closed in terms of accuracy. We report mean

\pm

std only for our runs (10 seeds).

Multimodal QA

Multimodal question answering combines language with images, video, audio, or documents to answer questions using evidence across modalities.

See all 195 papers

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models
Sep 1, 2026Yijun Chen, Yaqi Zheng, Yanya Li +11Multimodal RAG Efficient Multimodal Inference
Zhejiang University · South China University of Technology · Lenovo Group Limited
Retrieval-Augmented Online Textual Memory for Episodic Memory Video Question Answering
Oct 22, 2026Raffaele Cali, Giuseppe Lando, Rosario Forte +1Egocentric Video QA Video QA
Department of Mathematics and Computer Science, University of Catania, Italy
One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA
Jun 9, 2026Zhi Zheng, Ziqiao Meng, Hao Luan +2Multimodal RAG Multimodal Memory
School of Computing, National University of Singapore

Sep 1, 2026J/K move · Enter open · S save

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models

Yijun Chen, Yaqi Zheng, Yanya Li, Boyi Xiao+10

Zhejiang University · South China University of Technology · Lenovo Group Limited

Multimodal RAG Efficient Multimodal Inference Multimodal Memory Memory-Augmented Video Understanding Long-Video QA

Open paper AskPDF arXiv

Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).

Abstract

Multimodal QA

Explore similar work

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models

Retrieval-Augmented Online Textual Memory for Episodic Memory Video Question Answering

One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA