AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs

Manoj Madushanka Perera; Adnan Mahmood; Kasun Eranda Wijethilake; Quan Z. Sheng

cs.CLApr 14, 2026

AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs

Authors: Manoj Madushanka Perera, Adnan Mahmood, Kasun Eranda Wijethilake, Quan Z. Sheng

Organizations: School of Computing, Macquarie University, Sydney, NSW 2109, Australia

Abstract

Recent advancements in Large Language Models (LLMs) have improved their ability to process extended conversational contexts, yet fine-tuning and evaluating short- and long-term memories remain difficult due to the absence of datasets that encode both short- and long-term conversational history. Existing conversational datasets lack memory grounding, overlook topic continuity, or rely on costly human annotation. To address these gaps, we introduce AgenticAI-DialogGen, a modular agent-based framework that generates persona-grounded and topic-guided conversations without human supervision. The framework uses LLM agents to extract knowledge graphs, identify topics, build speaker personas, and simulate topic-guided conversations from unstructured conversations. A QA module generates memory-grounded Question Answer (QA) pairs drawn from short- and long-term conversational histories. We also generated a new dataset entitled, TopicGuidedChat (TGC), where long-term memory is encoded as speaker-specific knowledge graphs and short-term memory as newly generated topic-guided conversations. Evaluations depict that AgenticAI-DialogGen yields higher conversational quality and LLMs fine-tuned on TGC dataset achieve improved performance on memory-grounded QA tasks.

Figures & tables

Figure 1: A high-level overview of the AgenticAI-DialogGen framework.

Figure 2: A modular agent-based architecture of AgenticAI-DialogGen.

Dataset	MT	OD	PG	TC	KG	LT	ST	QA	AG
TGC (Ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
MSC Xu et al. (2022)	✓	✓	✓	x	x	✓	x	x	x
TopiOCQA Adlakha et al. (2022)	✓	✓	x	✓	x	x	x	✓	x
CoQA Reddy et al. (2019)	✓	x	x	x	x	x	x	✓	x
PersonaChat Zhang et al. (2018)	✓	✓	✓	x	x	x	x	x	x
QuAC Choi et al. (2018)	✓	x	x	x	x	x	x	✓	x

Table 1: Comparison of the TGC dataset with existing conversational datasets.

Dataset Records	Scale
# Persona Pairs	1,001
# Conversational Turns	88,500
Avg. Turns / Topic	30
# Topics	2,950
Avg. Topics / Persona Pair	3
# Unique Tokens	17,057

Table 2: Scale of the TGC dataset.

Type	Metric	MSC	TGC
Conversational Turns	Topic Consistency	64.12%	97.82%
	Factual Correctness	95.54%	94.37%
	Completeness	65.31%	88.22%
	Coherence	87.12%	94.67%
	Naturalness	98.76%	97.11%
Knowledge Graphs	Topic Consistency	–	90.36%

Table 3: Human evaluation of the dataset quality.

Metric	Submetric	MSC Dataset			PC Dataset			DD Dataset
		TGC w/ v	TGC w/o v	MSC	TGC w/ v	TGC w/o v	PC	TGC w/ v	TGC w/o v	DD
Conversational Flow (BERTScore)	Precision	30.13	24.80	12.20	28.11	22.92	11.64	27.26	21.80	9.33
	Recall	30.60	24.60	12.70	28.50	23.43	12.01	27.78	22.19	10.10
	F1	30.36	24.70	12.44	28.30	23.17	11.82	27.52	21.99	9.70
Topic Consistency	Self-BLEU	50.25	43.23	40.50	49.80	44.80	38.70	49.10	45.50	39.01
	Perplexity [ $\geq$ 1]	47.58	46.25	41.57	46.00	44.50	40.00	43.50	42.00	38.00

Table 4: Automatic evaluation of generated datasets across metrics.

Model	Evaluated Dataset	Precision	Recall	F1	Exact Match
Zero-shot Evaluation
GPT-4 OpenAI (2023)	TGC / KG	82.10	85.50	83.77	35.87
	TGC w/o KG	69.81	73.43	71.57	26.74
	MSC	60.23	63.94	62.03	20.95
Gemini 2.5 Flash DeepMind (2025)	TGC / KG	80.11	84.23	82.12	35.21
	TGC w/o KG	67.52	72.30	69.83	22.31

Table 5: Performance comparison of LLMs on memory QA evaluation (zero-shot and few-shot settings).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Overview of a sample TGC dataset for a single topic.

Long-Term Conversational Memory

Long-term conversational memory helps AI assistants retain, update, and use information from past conversations across sessions.

See all 267 papers

Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering
Sep 7, 2026Yifan Wang, Xinkui Lin, Yongxiu Xu +9Graph Retrieval-Augmented Generation Persistent Memory for Language Models
University of Electronic Science and Technology of China, Chengdu, China 4State Key Laboratory of Internet Architecture, Tsinghua University, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 3School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · 5Independent Researcher
HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents
Apr 20, 2026Shuqi Cao, Jingyi He, Fei TanPersistent Memory for Language Models Hierarchical Memory
East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations
Sep 9, 2026Hyojeong Yu, Hyukhun Koh, Minsung Kim +2Conversational Memory Persistent Memory for Language Models
Dept. of ECE, Seoul National University · IPAI, Seoul National University

Sep 7, 2026J/K move · Enter open · S save

Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering

Yifan Wang, Xinkui Lin, Yongxiu Xu, Shen Gao+8

University of Electronic Science and Technology of China, Chengdu, China 4State Key Laboratory of Internet Architecture, Tsinghua University, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 3School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · 5Independent Researcher

Graph Retrieval-Augmented Generation Persistent Memory for Language Models Long-Term Conversational Memory Conversational QA

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) enables large language models (LLMs) to answer questions by accessing external knowledge and has been widely adopted for long-term conversational memory question answering. However, existing methods suffer from two key challenges: (1) fragmented evidence scattered across temporally distant sessions, and (2) noisy content within retrieved sessions that triggers the lost-in-the-middle effect. To address these challenges, we propose MemLoc, a unified Retrieve-Localize-Generate framework for long-term conversational memory QA. For retrieval, MemLoc decomposes each session into multi-granularity memory units and performs query routing via an inner-memory graph with entropy-based granularity selection. It further models cross-session semantic and temporal dependencies through a cross-memory graph, enabling coarse-to-fine retrieval of top-K relevant memory candidates. For localization, we introduce a reasoning-based evidence locator trained with Self-reflective Hint Policy Optimization (SHPO), which performs progressive refinement by extracting query-relevant fragments within memory units to suppress noise and reranking across candidates to remove redundancy, producing a compact evidence set with lightweight location IDs. For generation, these IDs act as precise grounding signals that guide the LLM to the correct memory positions, mitigating the lost-in-the-middle effect while preserving original contextual integrity. Extensive experiments on four benchmarks demonstrate that MemLoc achieves state-of-the-art retrieval accuracy and response quality while maintaining efficiency. Our code is available at: https://github.com/Nikol-coder/MemLoc.

Abstract

Long-Term Conversational Memory

Explore similar work

Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations