Personalization Matters: Long-Horizon Conversation Agent with User-Centric Information in Online Shopping Interactions

Rena Gao; Yue Dai; Hao Guan; Shengxiang Gao; Wangyang Wu; Yixin Shen; Jey Han Lau

cs.MAOct 8, 2026

Personalization Matters: Long-Horizon Conversation Agent with User-Centric Information in Online Shopping Interactions

Authors: Rena Gao, Yue Dai, Hao Guan, Shengxiang Gao, Wangyang Wu, Yixin Shen, Jey Han Lau

Organizations: ByteDance · The University of Melbourne · University of Technology Sydney · AXON

Abstract

Personalized conversational shopping requires maintaining preference consistency over multi-turn interactions, where users reveal constraints gradually. Existing approaches often rely on static profiles and do not explicitly control long-horizon interaction behavior. We propose a multi-agent, multimodal Retrieval-Augmented Generation (RAG) framework that decomposes dialogue state tracking, recommendation retrieval, preference-aware reasoning, and response generation, while integrating product metadata, product reviews, image-derived descriptions, and user historical reviews. To evaluate interaction-level quality, we adopt a trajectory-level protocol with four dimensions: Global Preference Consistency, Cumulative Information Synthesis, Interaction Trajectory, and Tone Consistency. On an Amazon Reviews 2023 benchmark, retrieval-enabled variants outperform a no-RAG baseline on automatic trajectory metrics (average 4.82 vs. 3.74). In a small real-user study ( $n{=}5$ ), the Full variant achieves the highest mean overall rating (4.60 vs. 2.20 for Baseline), providing exploratory evidence that role decomposition plus user-centric retrieval improves perceived personalization.\footnote{Code and dataset are available at: https://github.com/RenaGao/Multimodel_RAG_Indexing

Figures & tables

Figure 1: A failure case without effective personalization during online shopping interactions: the response is locally reasonable but does not reflect previously stated style preferences, revealing a gap between turn-level relevance and long-horizon consistency.

Figure 2: Overall framework of the proposed multi-agent personalized conversational shopping system. Each turn is decomposed into dialogue context tracking, retrieval-based recommendation, preference-aware reasoning, and response generation, supported by external dialogue memory and the Amazon Reviews 2023 corpus.

Statistic	Value
Raw Amazon categories	34
Top active users retained per category	100
Candidate user pool	3,300
Target users / dialogue instances	327
Dialogues per variant	327
Dialogue runs ( $327\times 5$ variants)	1,635

Table 1: Scale summary of the processed dialogue dataset, detailed per-category item/review statistics are reported in Appendix Table 6 .

Variant	$\mathcal{K}^{d}_{t}$	$\mathcal{K}^{p}_{t}$	$\mathcal{K}^{r}_{t}$	Prod. emb.	Pipeline
Baseline	✗	✗	✗	—	Multi-agent
TextRAG	✓	✓	✓	$e^{m},e^{r}$	Multi-agent
NoRev	✓	✓	✗	$e^{m},e^{r},e^{d}$	Multi-agent
Single	✓	✓	✓	$e^{m},e^{r},e^{d}$	Single-agent
Full	✓	✓	✓	$e^{m},e^{r},e^{d}$	Multi-agent

Table 2: Assistant-side variants.

\mathcal{K}^{d}_{t}

\mathcal{K}^{p}_{t}

, and

\mathcal{K}^{r}_{t}

denote dialogue, product, and user-review retrieval. Prod. emb. lists embeddings used in Eq. ( 7 ): metadata (

e^{m}

), review (

e^{r}

), and description (

e^{d}

Metric	Baseline	TextRAG	NoRev 1	Single 2	Full 3
GPC	3.77	4.93	4.88	4.91	4.93
CIS	3.17	4.88	4.79	4.78	4.89
IT	3.02	4.47	4.39	4.19	4.45
TC	5.00	5.00	5.00	5.00	5.00
Avg.	3.74	4.82	4.76	4.72	4.82

Table 3: Automatic evaluation results, averaged over 327 dialogues per variant. 1 Multimodal (NoRev). 2 Multimodal (Single-Agent). 3 Multimodal (Full).

Metric	Baseline	TextRAG	NoRev 1	Single 2	Full 3
Acc.	10.31%	42.73%	41.90%	48.20%	43.73%

Table 4: Retrieval accuracy. 1 Multimodal (NoRev). 2 Multimodal (Single-Agent). 3 Multimodal (Full).

Dimension	Baseline	TextRAG	NoRev 1	Single 2	Full 3
Overall	2.20	3.40	3.80	3.80	4.60
GPC	1.20	3.60	4.00	4.00	4.60
CIS	1.60	3.60	4.20	4.00	4.80
IT	2.00	3.60	3.80	3.60	4.60
TC	4.20	4.60	4.60	4.80	5.00

Table 5: User-study ratings (mean, 1–5 scale,

n=5

). 1 Multimodal (NoRev). 2 Multimodal (Single-Agent). 3 Multimodal (Full).

Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Items	Reviews	Avg.	Dlg.
Electronics	27,292	9,907,789	363.03	10
Home_and_Kitchen	51,421	8,780,109	170.75	10
Kindle_Store	113,360	7,172,600	63.27	10
Movies_and_TV	67,650	6,112,901	90.36	8
Health_and_Household	34,382	5,492,657	159.75	10
Clothing_Shoes_and_Jewelry	63,329	5,360,811	84.65	10

Appendix

Table 6: Product and review statistics, with trajectory counts (Dlg.) from the released query set. Counts total 327: Movies and TV has 8, Subscription Boxes has 9, and other categories have 10.

Method	Role calls / turn	Auto. Avg.	User Overall
Baseline	1	3.74	2.20
TextRAG	4	4.82	3.40
NoRev	4	4.76	3.80
Single	1	4.72	3.80
Full	4	4.82	4.60

Appendix

Table 7: Assistant-role calls per turn and performance. Full uses Dialogue, Recommend, Reason, and Reply stages; Single combines these roles in one call. The Baseline count is retained from the original report and awaits configuration verification.

Comparison / metric	Adjusted $p$	$r$
RAG vs. Baseline ∗	$<0.001$	$\geq 0.90$
Full–Single: IT	$<0.001$	0.433
Full–Single: CIS	0.040	0.508
Full–Single: Overall	0.002	0.351

Appendix

Table 8: Paired comparisons. ∗ Bounds across TextRAG, NoRev, Single, and Full versus Baseline on GPC, CIS, and IT. Overall is the mean of the four trajectory scores.

Variant	SR@all	First-hit turn
TextRAG	0.352	2.26
NoRev	0.309	2.29
Single	0.352	2.50
Full	0.361	2.49

Appendix

Table 9: Target-ASIN retrieval results. First-hit turns are conditional on a hit. These measures track target retrieval rather than user acceptance.

Metric	GPC	CIS	IT	Overall
Spearman $\rho$	0.508	0.533	0.471	0.538

Appendix

Table 10: Agreement between GPT-4o and Qwen3-235B scores on the stratified subset.

Memory	Overall	IT	CIS
Full	4.796	4.387	4.882
None	4.610	3.796	—
Summary only	4.801	—	4.860
Top-1 turn only	4.634	3.935	—

Appendix

Table 11: Memory ablation results. Dashes indicate values absent from the reported results.

Work	Training / inference setup	Evidence and task resources
ChatCRS Li et al. (2025a)	LLM generation with knowledge retrieval and a LoRA-fine-tuned goal planner.	External knowledge base and annotated dialogue goals; multi-goal CRS datasets.
RevCore Lu et al. (2021)	Trained recommendation and response-generation components.	Item-linked reviews, entities, and conversational recommendation data.
MACRec Wang et al. (2024a)	Configurable LLM agents with task-dependent tools.	User/item information and task-dependent tools; multiple recommendation scenarios.
RA-Rec Kemper et al. (2024)	Prompt-driven state tracking and generation with a pretrained dense retriever.	Domain-specific state fields, item reviews and metadata; demonstrated on Yelp restaurants.
PEARL Kim et al. (2024)	Review-grounded dialogue dataset with downstream model experiments.	Review-derived personas and knowledge for synthetic movie recommendation dialogues.
MUSE Wang et al. (2025)	Multimodal conversational dataset with benchmark experiments.	Scenario-grounded profiles and clothing-domain multimodal conversations.

Appendix

Table 12: Training and data resources used in related conversational recommendation work.

Criterion	Description and Scoring Rubric (1–5)
1. Global Personalization Consistency (GPC)	Did the agent maintain the user’s persona constraints throughout the entire dialogue?
	1 (Inconsistent/Amnesic): Agent started well but forgot constraints (e.g., budget, size) in later turns, or hallucinated user details.
	3 (Fragmented): Agent remembered explicit constraints (e.g., Size M) but forgot implicit ones (e.g., Style preferences) as the chat went on.
	5 (Rock Solid): Every single response, from start to finish, aligned with the User Profile. The agent never “reset” or forgot context.
2. Cumulative Information Synthesis (CIS)	Looking at all agent responses combined, did it build a complete, factual picture?
	1 (Fragmented/Repetitive): Agent repeated the same facts multiple times or gave contradictory information across different turns.

Appendix

Table 13: Evaluation Criteria for AI Shopping Assistant. Each dimension (1–5 scale) assesses cumulative dialogue quality across the entire interaction, focusing on personalization, synthesis, trajectory, and tone stability.

Interactivity Macro-level Features	Definition
Topic Management	the strategies and techniques used to control and navigate the flow of topics.
Tone Choice Appropriateness	the suitability of the tone used in communication, ensuring it aligns with the context, audience, and purpose to convey the intended message.
Conversation Opening	the initial interaction or exchange that begins a dialogue, often setting the tone and context for the dialogue.
Conversation Closing	the process of ending a dialogue or interaction, which involves signaling the conclusion of the discussion, summarizing key points, and often expressing a farewell.

Appendix

Table 14: Definitions of macro-level interactivity features, where higher scores indicate more natural, authentic interaction and active engagement in the dialogue (adapted from Gao et al. (2025b) ).

Recommender Agent Response
User: Can you help me find a good moisturizer for dry skin?
Recommender Agent: … To provide the best recommendation, a few details are needed:
(1) Skin Concerns: Are you dealing with flakiness, redness, irritation, or general dryness?
(2) Skin Type: Do you have sensitive skin, or are you okay with fragrance or other ingredients?
(3) Budget: Are you looking for something affordable, mid-range, or premium?
(4) Texture Preference: Do you prefer a lightweight gel, rich cream, or a lotion-like consistency?

Appendix

Table 15: Recommender Agent output example generated by BaseAgent. The agent elicits user preferences and provides structured, evidence-based product recommendations.

Reasoner Agent Evaluation on Recommend Agent Output
Reasoner Agent: … 1. Personalized Inquiry First: The agent begins by asking questions about skin concerns, type, budget, texture preference, and allergies. This ensures the final recommendation is customized rather than generic, increasing the likelihood of satisfaction and effectiveness.
2. Categorization by Skin Needs: The agent groups moisturizers by skin profiles (e.g., sensitive, very dry, lightweight, intense hydration), allowing users to quickly identify the most relevant category and reducing decision fatigue.
3. Inclusion of Dermatologist-Backed Brands: Products such as CeraVe, La Roche-Posay, and Vanicream are widely recognized by dermatologists for safety, efficacy, and ingredient transparency, which builds user trust.
4. Clear Ingredient Guidance: Highlighting key ingredients (ceramides, hyaluronic acid, glycerin, etc.) helps users understand why certain products work and supports informed decisions.
5. Balance of Price and Performance: Offers both affordable and premium options (e.g., Neutrogena vs. Kiehl’s), ensuring accessibility without compromising quality.
6. Texture and Use-Case Consideration: Distinguishes lightweight gels (for daytime) from rich creams (for nighttime or severe dryness), reflecting practical understanding of skincare routines.

Appendix

Table 16: Reasoner Agent evaluation of the Recommender Agent’s output, highlighting personalization, structure, credibility, and educational value.

Figure 9: One turn question and response from different methods

Figure 21: MultimodalRag Single Agent User Interaction 1

Figure 22: MultimodalRag Single Agent User Interaction 2

Figure 23: MultimodalRag Single Agent User Interaction 3

Figure 24: MultimodalRag Single Agent User Interaction 4

Figure 25: MultimodalRag Single Agent User Interaction 5

Figure 26: MultimodalRag Single Agent User Interaction 6

Figure 27: MultimodalRag Single Agent User Interaction 7

Figure 28: MultimodalRag Single Agent User Interaction 8

Figure 29: MultimodalRag no Review User Interaction 1

Figure 30: MultimodalRag no Review User Interaction 2

Figure 31: MultimodalRag no Review User Interaction 3

Figure 32: MultimodalRag no Review User Interaction 4

Figure 33: MultimodalRag no Review User Interaction 5

Figure 34: MultimodalRag no Review User Interaction 6

Figure 35: MultimodalRag no Review User Interaction 7

Figure 36: MultimodalRag Full User Interaction 1

Figure 37: MultimodalRag Full User Interaction 2

Figure 38: MultimodalRag Full User Interaction 3

Figure 39: MultimodalRag Full User Interaction 4

Figure 40: MultimodalRag Full User Interaction 5

Figure 41: MultimodalRag Full User Interaction 6

Figure 42: MultimodalRag Full User Interaction 7

Explore similar work

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Jun 16, 2026Zeyao Du, Tong Li, Haibo ZhangTool-Use Evaluation Long-Horizon Agent Evaluation
Shopee
RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce
Sep 30, 2026Xinwei Yang, Kelong Mao, Yudong Guo +4LLM Agent Evaluation AI Agent Benchmarks
College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore
Learning User-Aware Recall: Personalized Retrieval in Long-Term Conversational Memory
May 28, 2026ZhiShu Jiang, Haibo Liu, Xin Shen +6Query Reformulation Information Retrieval
Baidu Inc. · The University of Queensland

Jun 16, 2026, cs.AIJ/K move · Enter open · S save

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

Zeyao Du, Tong Li, Haibo Zhang

Shopee

Tool-Use Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks Agentic Commerce Preference Elicitation Web Agents

Open paper AskPDF arXiv

As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.

Abstract

Explore similar work

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

Learning User-Aware Recall: Personalized Retrieval in Long-Term Conversational Memory