MACS: A Hybrid Multi-Agent Framework for Reliable Conversational E-Commerce Recommendation

Juli Huang; Hannah Clay; Sajjad Beygi; Thomas Sarda; Negin Golrezaei; Amin Saberi

cs.IRAug 14, 2026

MACS: A Hybrid Multi-Agent Framework for Reliable Conversational E-Commerce Recommendation

Authors: Juli Huang, Hannah Clay, Sajjad Beygi, Thomas Sarda, Negin Golrezaei, Amin Saberi

Organizations: Stanford University Department of Computer Science Stanford, CA, USA · University of Southern California Los Angeles, CA, USA · Massachusetts Institute of Technology MIT Sloan School of Management Cambridge, MA, USA · Stanford University Department of Management Science & Engineering Stanford, CA, USA

Abstract

Conversational recommendation for e-commerce is increasingly mediated by large language models (LLMs), yet many real-world deployments operate under a stricter requirement: recommendations must be drawn only from a merchant's fixed catalog, without web search or unsupported product claims. In this setting, the main challenge is reliability under hard constraints: the system must satisfy user requirements, remain grounded in available inventory, and preserve preferences across multiple conversational turns. We present MACS (Multi-Agent Commerce System), a hybrid multi-agent framework for reliable conversational recommendation in fixed-catalog settings. MACS uses LLMs for language-facing tasks such as interpreting user requests, eliciting preferences, and generating responses, while correctness-critical operations, including product retrieval, hard-constraint filtering, brand exclusion, and progressive relaxation, are executed deterministically by the merchant agent. A session-persistent preference layer tracks constraints across turns, enabling consistent handling of budget overwrites and exclusion reversals. On a 140-query single-turn benchmark, MACS achieves the highest pass rate (87.1%) and perfect brand compliance (1.000). On a 10-scenario multi-turn benchmark, MACS achieves the strongest macro Pass@5 (72% vs. 56% GPT+Catalog / 52% Gemini+Catalog) with zero constraint drift. The advantage is sharpest on exclusion reversal (100% vs. 20% / 0%) and constraint accumulation (100% vs. 60% / 40%). Mean judged response quality is similar across systems (0.751 vs. 0.736). These results suggest that hybrid architectures combining deterministic constraint enforcement with session-persistent preference tracking provide stronger reliability-oriented performance than catalog-bound prompt-only baselines in the fixed-catalog merchant setting.

Figures & tables

Panel of Figure 1 — Figure 1. MACS interaction flows. The AI shopping agent handles natural-language interaction and protocol generation; the merchant agent enforces hard constraints, validates inventory, and returns structured responses. Flow 1 (Search): the user’s query is parsed, constraint-filtered via the knowledge graph and SQL store, and results are returned. Flow 2 (Add to Cart): the merchant agent validates inventory before confirming. All constraint enforcement occurs exclusively in the merchant agent. Two interaction flow diagrams. Flow 1 (Search): User to AI Shopping Agent (parse request, generate UCP) to Merchant Agent (parse UCP, use KG, prices) to Create UCP response to AI Shopping Agent (present to user) to User (sees results). Flow 2 (Add to Cart): User to AI Shopping Agent to Merchant Agent (add to cart, validate inventory) to Create UCP response to AI Shopping Agent to User (cart updated).

Figure 2. MACS comparison and checkout view. Pros/cons are derived from structured catalog attributes, and delivery options and pricing are retrieved directly from the merchant agent’s relational store, not generated by the LLM. Side-by-side product comparison showing Lenovo IdeaPad vs.\ ThinkBook with pros/cons bullet points, delivery options (Standard, Express, Overnight), and itemized pricing breakdown.

Use case	$w_{\text{price}}$	$w_{\text{rating}}$	$w_{\text{spec}}$	$\phi(p)$ components
Gaming	0.15	0.20	0.60	GPU tier (0–0.40) + refresh rate (0–0.15) + RAM $\geq$ 16 GB (+0.03)
ML	0.15	0.20	0.60	RAM tier (0–0.40) + GPU (0–0.20)
Video edit.	0.15	0.20	0.60	GPU tier (0–0.30) + RAM tier (0–0.20)
Student	0.45	0.30	0.20	Battery tier (0–0.30) + RAM tier (0–0.15)
Programming	0.20	0.30	0.45	RAM tier only (0–0.20)
Default	0.35	0.35	0.20	RAM tier only (0–0.20)

Table 1. Best-value scoring weights and

\phi(p)

spec-score components per use case (

c_{\text{vol}}=0.05

for all).

Phase	Median (ms)	Notes
CPU phases (rewrite, filter)	$<$ 1	Pure Python, no I/O
Domain detection — fast path	$<$ 1	Keyword dictionary
Domain detection — LLM	$\sim$ 1,630	gpt-4o-mini; ambiguous only
Criteria extraction (LLM)	1,927	Primary bottleneck
Question generation (LLM)	1,645	Preference elicitation
Post-rec intent detection	556	gpt-4o-mini classification

Table 2. Measured median latencies by pipeline phase (

n=10

runs per phase, direct-call benchmark; median reported to reduce sensitivity to occasional LLM cold-start outliers). LLM calls dominate total latency, and knowledge graph traversal and cache hits are negligible.

System	Avg $\uparrow$	Pass% $\uparrow$	Brand $\uparrow$	Filter $\uparrow$	Quality $\uparrow$
MACS	0.681	87.1%	1.000	0.970	0.394
Gemini+Catalog	0.662	68.6%	0.833 †	N/A	0.427
GPT+Catalog	0.639	72.1%	1.000 †	N/A	0.392

Table 3. Three-way single-turn comparison (

n=140

unique queries, pass threshold

\geq 0.5

. MACS Brand/Filter: deterministic SQL-layer compliance. Baseline Brand † : regex negation-window check on free-text responses (6 exclusion queries; 1 Gemini violation on Q26 “no HP, no Acer”). Filter N/A: cannot verify price compliance from free text. Quality: LLM-judge narrative score (GPT-4o-mini, temperature 0).

Strong groups
Group	$N$	Avg $\uparrow$	Pass% $\uparrow$
Multi-intent rigidity	5	0.775	100.0%
One-liner purchase intent	6	0.833	100.0%
Refinement	3	0.756	100.0%
Follow-up service QA	4	0.810	100.0%
Multi-constraint	17	0.791	94.1%

Table 4. Per-group single-turn results on the expanded 225-query benchmark (MACS only). Per-group Pass% uses threshold

\geq 0.65

(stricter, to identify reliably-passing groups); the " All " row uses

\geq 0.5

for comparability with Table 3 . All queries evaluated independently (no prior session context).

System	MT Avg $\pm$ SD	MT Pass@5	Drift
MACS	0.751 $\pm$ 0.095	72%	0.000
GPT+Catalog	0.736 $\pm$ 0.090	56%	0.000
Gemini+Catalog	0.717 $\pm$ 0.118	52%	0.014

Table 5. Multi-turn results (MT, 10 scenarios, Pass@5, threshold

\geq 0.65

). MT scoring:

0.55\times

constraint (deterministic)

+

0.45\times

judge (GPT-4o-mini, temp. 0). All systems catalog-bound; same token budget; no web search. Pass@5 = macro fraction of 5 runs passing threshold. Drift: fraction of run-scenario pairs with a constraint violation. MACS uniquely achieves 100% Pass@5 on the two hardest state-management scenarios (S1: Constraint Accumulation, S8: Exclusion Reversal); aggregate mean score differences lie within one SD.

Category	MACS	GPT+Catalog	Gemini+Catalog
Brand Exclusion	0.640 / 20%	0.667 / 40%	0.685 / 40%
Budget Overwrite	✓0.793 / 100%	✓0.739 / 80%	✓0.712 / 80%
Budget Refinement	✓0.748 / 60%	✓0.802 / 80%	✓0.811 / 80%
Clarification	✓0.793 / 100%	✓0.802 / 100%	✓0.730 / 100%
Comparison	0.658 / 40%	0.631 / 0%	0.586 / 0%
Constraint Accum.	✓0.784 / 100%	✓0.739 / 60%	0.640 / 40%

Table 6. Multi-turn category breakdown (Pass@5 runs; avg score / Pass@5; ✓ = Pass@5

\geq 60

%). Categories map 1:1 to the 10 scenarios.

System	Grounding $\uparrow$	Disclosure $\uparrow$	Drift $\downarrow$
MACS	1.000	0.925	0.000
GPT+Catalog	1.000	0.588	0.000
Gemini+Catalog	1.000	0.550	0.014

Table 7. Reliability, grounding, and disclosure evaluation (

n=8

catalog-impossible queries for Disclosure;

n=10

scenarios

\times

K=5 runs for Drift). Grounding: fraction of returned products verifiable in the catalog at stated price. Disclosure: fraction of impossible queries acknowledged explicitly. Drift: fraction of run-scenario pairs with a constraint violation.

\uparrow

higher is better;

\downarrow

lower is better.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ID	Name	Turns	Key challenge
S1	Constraint Accumulation	4	Preserve all accumulated constraints
S2	Mind-Change Pivot	3	Use-case pivot (gaming $\to$ editing)
S3	Brand Excl. Persist.	4	Maintain brand exclusion across turns
S4	Vague $\to$ Specific	4	Elicit & resolve underspecified query
S5	Comparison + Follow-up	3	Context-free comparison
S6	Price Refinement	3	Budget update (narrowing)

Appendix

Table 8. Multi-turn descriptions (10 scenarios, 3–5 scripted turns each). “Key challenge” identifies the conversational state-management property under test. All scenarios are recommendation-focused except S4 and S5.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 516 papers

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce
Sep 30, 2026Xinwei Yang, Kelong Mao, Yudong Guo +4LLM Agent Evaluation AI Agent Benchmarks
College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore
MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
Jul 31, 2026Qiming Shi, Yulong Tao, Linbo Jin +10LLM Agent Evaluation Long-Horizon Agent Evaluation
2State Key Lab of CAD&CG, Zhejiang University · 1Alibaba Group · School of Software and Microelectronics, Peking University +2
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Jun 16, 2026Zeyao Du, Tong Li, Haibo ZhangWeb Agents Long-Horizon Agent Evaluation
Shopee

Sep 30, 2026J/K move · Enter open · S save

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

Xinwei Yang, Kelong Mao, Yudong Guo, Sulong Xu+3

College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore

LLM Agent Evaluation AI Agent Benchmarks Recommender Systems Conversational Agents Agentic Commerce

Open paper AskPDF arXiv

Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-level decision support: users reveal and revise constraints, coordinate multiple goals, and expect product-grounded recommendations over a full conversation. Existing benchmarks are mostly outcome-oriented or execution-oriented, leaving this evolving decision process under-evaluated. We introduce REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation. Our analysis shows that current systems produce locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially under ambiguous intent, bundle, and multi-intent scenarios. We further propose REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards. Experiments show that REALSHOP_AGENT consistently outperforms strong baselines on REALWORLDSHOP.

Abstract

Recommender Systems

Explore similar work

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent