Proactive Memory for Ad-Hoc Recall over Streaming Dialogues

Bingbing Wang; Jing Li; Ruifeng Xu

cs.AIMar 5, 2026

Proactive Memory for Ad-Hoc Recall over Streaming Dialogues

Authors: Bingbing Wang, Jing Li, Ruifeng Xu

Organizations: Department of Computing The Hong Kong Polytechnic University · The School of Computer Science and Technology Harbin Institute of Technology (Shenzhen)

Abstract

Real-world dialogue usually unfolds as an infinite stream. It thus requires bounded-state memory mechanisms to operate within an infinite horizon. However, existing read-then-think memory is fundamentally misaligned with this setting, as it cannot support ad-hoc memory recall while streams unfold. To explore this challenge, we introduce STEM-Bench, the first benchmark for STreaming Evaluation of Memory. It comprises over 14K QA pairs in dialogue streams that assess perception fidelity, temporal reasoning, and global awareness under infinite-horizon constraints. The preliminary analysis on STEM-Bench indicates a critical textit{fidelity-efficiency dilemma}: retrieval-based methods use fragment context, while full-context models incur unbounded latency. To resolve this, we propose ProStream, a proactive memory framework for streaming dialogues built on a hierarchical structure. It enables ad-hoc memory recall on demand by reasoning over continuous streams with multi-granular distillation. Moreover, it employs Adaptive Spatiotemporal Optimization to dynamically optimize retention based on expected utility. It enables a bounded knowledge state for lower inference latency without sacrificing reasoning fidelity. Experiments show ProStream delivers higher reasoning fidelity than prior baselines while maintaining substantially lower latency than full-context alternatives.

Figures & tables

Figure 1: Comparison of the read-then-think paradigm (left) and the streaming memory paradigm (right) based on The Big Bang Theory (TBBT) dialogues.

Figure 2: Overview of STEM-Bench Benchmark Curation. (a) Data construction pipeline of the STEM-Bench dataset. (b) Statistics of the STEM-Bench dataset with samples examining HFP, SLR, and DGA. TBBT: The Big Bang Theory.

Figure 3: Preliminary analysis of RAG and full-context performance. (Right) The average accuracy of all performance metrics across evidence distances. (Left) Inference latency over dialogue turns. The dashed lines indicate the overall average results.

Figure 4: Overview of our ProStream with four components discussed in turn from $\lx@sectionsign$ 5.1 to $\lx@sectionsign$ 5.4 .

Method	The Big Bang Theory							Friends							The Office
Method	Time $\downarrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	Gem $\uparrow$	Time $\downarrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	Gem $\uparrow$	Time $\downarrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	Gem $\uparrow$
RAG	0.957	0.068	0.247	0.781	0.355	0.508	0.331	0.923	0.061	0.231	0.768	0.345	0.502	0.309	0.918	0.074	0.271	0.778	0.322	0.524	0.385
RQ-RAG	5.149	0.051	0.201	0.770	0.275	0.460	0.329	5.098	0.043	0.184	0.758	0.372	0.439	0.313	1.058	0.072	0.270	0.780	0.466	0.535	0.376
RAPTOR	0.519	0.041	0.165	0.761	0.302	0.375	0.214	0.513	0.035	0.157	0.746	0.396	0.367	0.214	0.443	0.032	0.142	0.745	0.324	0.334	0.201
Full Context	7.333	0.065	0.249	0.737	0.250	0.710	0.521	4.870	0.057	0.241	0.732	0.360	0.706	0.505	4.776	0.066	0.266	0.742	0.292	0.731	0.574
Summary	2.001	0.049	0.200	0.783	0.296	0.407	0.278	2.352	0.042	0.180	0.762	0.334	0.385	0.246	1.641	0.048	0.201	0.770	0.365	0.397	0.272

Table 1: Main comparison results on our STEM-Bench. Best results are bolded , second best are underlined . Metrics marked with

\uparrow

indicate higher is better, while

\downarrow

denotes lower is better. For ablations (bottom), w/o indicates without.

Figure 5: Average performance metrics (left) and latency (right) comparisons across different experimental settings. (a) Results under different LLM backbones, where the numbers above the bars denote the percentage improvement of ProStream over the Full-Context baseline. (b) Results under varying context complexity levels. The distribution curves visualize context entropy, showing that ProStream (green) produces a more polarized context complexity distribution than Full Context (red).

Figure 6: Qualitative comparison (Case Study) of the RAG method and our ProStream.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Variant	The Big Bang Theory							Friends							The Office
		Time $\downarrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	Gem $\uparrow$	Time $\downarrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	Gem $\uparrow$	Time $\downarrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	Gem $\uparrow$
BGE	RAG	0.814	0.087	0.301	0.796	0.583	0.572	0.413	0.814	0.081	0.296	0.785	0.621	0.576	0.399	0.801	0.094	0.339	0.796	0.647	0.611	0.413
BGE	our	0.393	0.116	0.383	0.816	0.460	0.623	0.498	0.442	0.097	0.354	0.802	0.464	0.597	0.469	0.516	0.114	0.407	0.819	0.484	0.622	0.498
UAE	RAG	0.774	0.086	0.296	0.794	0.583	0.572	0.410	0.788	0.082	0.296	0.785	0.619	0.573	0.408	0.756	0.093	0.337	0.796	0.650	0.612	0.410
UAE	our	0.825	0.116	0.386	0.818	0.482	0.615	0.501	0.829	0.098	0.357	0.802	0.482	0.592	0.471	0.688	0.117	0.413	0.821	0.503	0.624	0.501
MiniLM	RAG	1.225	0.064	0.240	0.781	0.389	0.500	0.331	1.162	0.060	0.237	0.774	0.424	0.497	0.309	1.067	0.072	0.269	0.781	0.466	0.533	0.331

Appendix

Table 2: Results of different embedding models on STEM-Bench. Metrics marked with

\uparrow

indicate higher is better, while

\downarrow

denotes lower is better.

(a) Sensitivity to $\tau_{\text{drift}}$
Setting	The Big Bang Theory					Friends					The Office
Setting	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$
0.5	0.15	0.43	0.85	0.51	0.64	0.12	0.39	0.82	0.52	0.62	0.14	0.46	0.84	0.50	0.69
0.6	0.16	0.44	0.85	0.51	0.66	0.12	0.40	0.82	0.52	0.63	0.14	0.46	0.83	0.50	0.68
0.7	0.12	0.39	0.82	0.49	0.61	0.10	0.36	0.80	0.49	0.59	0.11	0.41	0.82	0.49	0.63
0.8	0.16	0.45	0.85	0.49	0.68	0.11	0.39	0.82	0.52	0.62	0.14	0.46	0.83	0.51	0.67

Appendix

Table 3: Threshold sensitivity analysis of ProStream. We vary the drift threshold

\tau_{\text{drift}}

and the concept threshold

\theta_{\text{concept}}

, and report BLEU-4 (B-4), ROUGE-L (R-L), BERTScore (B-S), Evidence F1 (Evid.), and KEM. Performance remains stable across a broad range of threshold values.

Setting	The Big Bang Theory					Friends					The Office
Setting	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$
Full	0.16	0.44	0.85	0.51	0.66	0.13	0.40	0.82	0.52	0.62	0.14	0.46	0.83	0.50	0.68
Perturb $\mathcal{B}$	0.15	0.44	0.85	0.50	0.66	0.13	0.41	0.83	0.52	0.64	0.15	0.46	0.84	0.51	0.66
Perturb $\mathcal{M}_{\text{pend}}$	0.16	0.44	0.85	0.51	0.66	0.13	0.40	0.82	0.52	0.62	0.14	0.46	0.83	0.50	0.68
Perturb $\mathcal{M}_{\text{tree}}$	0.16	0.44	0.85	0.51	0.66	0.12	0.40	0.82	0.52	0.63	0.14	0.46	0.83	0.51	0.68

Appendix

Table 4: Results of stage-wise error propagation analysis. Metrics reported are BLEU-4 (B-4), ROUGE-L (R-L), BERTScore (B-S), Evidence F1 (Evid.), and KEM.

$\mathbf{\mathcal{T}_{\max}}$	The Big Bang Theory					Friends					The Office
$\mathbf{\mathcal{T}_{\max}}$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$	B-4 $\uparrow$	R-L $\uparrow$	B-S $\uparrow$	Evid. $\uparrow$	KEM $\uparrow$
50k	0.15	0.41	0.82	0.16	0.53	0.11	0.35	0.79	0.23	0.48	0.15	0.44	0.82	0.21	0.57
75k	0.15	0.40	0.82	0.16	0.53	0.11	0.36	0.78	0.23	0.50	0.15	0.45	0.82	0.21	0.57
100k	0.16	0.41	0.82	0.16	0.54	0.11	0.36	0.79	0.23	0.49	0.15	0.44	0.82	0.21	0.57
125k	0.15	0.41	0.82	0.16	0.54	0.11	0.37	0.79	0.23	0.50	0.15	0.45	0.82	0.21	0.57
150k	0.15	0.40	0.82	0.16	0.53	0.11	0.35	0.78	0.23	0.49	0.15	0.44	0.82	0.21	0.57

Appendix

Table 5: Sensitivity to memory capacity

\mathcal{T}_{\max}

. Metrics reported are BLEU-4 (B-4), ROUGE-L (R-L), BERTScore (B-S), Evidence F1 (Evid.), and KEM.

Method	Single-hop	Multi-hop	Temporal	Open-domain
Mistral-7B-Instruct-v0.2	9.1	15.1	9.3	8.6
Llama-2-70B-chat	20.8	18.2	15.9	18.8
Llama-3-70B-Instruct	17.0	17.0	12.0	13.0
gpt-3.5-turbo-4K	23.8	18.0	15.6	20.4
gpt-3.5-turbo-8K	38.5	25.1	22.7	25.9
Ours	25.5	31.3	38.8	20.4

Appendix

Table 6: Transfer evaluation on LoCoMo. Best results are bolded .

Method	Accuracy
Vanilla	10.0
RecurSum	10.0
ConditionMem	40.0
MemoryBank	23.33
Mem-PAL	46.67
Ours	50.0

Appendix

Table 7: Transfer evaluation on LongMemEval. Best result is bolded .

Figure 7: Two examples of error analysis.

Long-Term Conversational Memory

Long-term conversational memory helps AI assistants retain, update, and use information from past conversations across sessions.

See all 267 papers

ProV-Mem: Proactive Memory for Streaming Video Question Answering
Oct 15, 2026Bingbing Wang, Jing Li, Han Gao +3Video QA Memory-Augmented VLMs
Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · Research Institute for Generative AI, The Hong Kong Polytechnic University, Hong Kong, China · Tencent Inc., China +1
Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization
Aug 10, 2026Hyangsuk Min, Hwanjun SongConversational Memory Text Summarization
KAIST
Improving Multi-turn Dialogue Consistency with Self-Recall Thinking
May 14, 2026Renning Pang, Tian Lan, Leyuan Liu +3Memory-Augmented Language Models Long-Term Conversational Memory
University of Electronic Science and Technology of China

Oct 15, 2026J/K move · Enter open · S save

ProV-Mem: Proactive Memory for Streaming Video Question Answering

Bingbing Wang, Jing Li, Han Gao, Yishi Lin+2

Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · Research Institute for Generative AI, The Hong Kong Polytechnic University, Hong Kong, China · Tencent Inc., China+1

Video QA Memory-Augmented VLMs

Open paper AskPDF arXiv

Proactive streaming video question answering requires a model to continuously monitor an incoming stream and autonomously decide when to speak, what to say, and whether to emit a response. Existing streaming systems face a fundamental difficulty: retaining long visual histories is costly, whereas discarding them can lead to missed events and repeated responses. We propose ProV-Mem, a training-free proactive memory framework built around a frozen VLM backbone. ProV-Mem decomposes the proactive decision into three lightweight memory tiers: a temporal-trigger memory controls when to query the backbone and recovers from prolonged silence, an instructional memory guides what to say through a fixed prompting prior for concise and visually grounded generation, and an episodic memory determines whether to emit by filtering near-duplicate responses. Under an idealized conditional-sufficiency assumption, we derive a stagewise approximation bound relating the decomposed process to a full-state joint policy. We further introduce Cost-Aware Area Under the Curve (CAUC), which extends PAUC by explicitly discounting redundant, overly frequent, and unnecessarily long emissions. Experiments on the four scenarios of ProactiveVideoQA show that ProV-Mem achieves the best CAUC across all scenarios and the best PAUC on WEB, EGO, and TV, while maintaining low per-response inference latency.

Abstract

Long-Term Conversational Memory

Explore similar work

ProV-Mem: Proactive Memory for Streaming Video Question Answering

Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization

Improving Multi-turn Dialogue Consistency with Self-Recall Thinking