MACS: A Hybrid Multi-Agent Framework for Reliable Conversational E-Commerce Recommendation
Authors: Juli Huang, Hannah Clay, Sajjad Beygi, Thomas Sarda, Negin Golrezaei, Amin Saberi
Organizations: Stanford University Department of Computer Science Stanford, CA, USA · University of Southern California Los Angeles, CA, USA · Massachusetts Institute of Technology MIT Sloan School of Management Cambridge, MA, USA · Stanford University Department of Management Science & Engineering Stanford, CA, USA
Conversational recommendation for e-commerce is increasingly mediated by large language models (LLMs), yet many real-world deployments operate under a stricter requirement: recommendations must be drawn only from a merchant's fixed catalog, without web search or unsupported product claims. In this setting, the main challenge is reliability under hard constraints: the system must satisfy user requirements, remain grounded in available inventory, and preserve preferences across multiple conversational turns. We present MACS (Multi-Agent Commerce System), a hybrid multi-agent framework for reliable conversational recommendation in fixed-catalog settings. MACS uses LLMs for language-facing tasks such as interpreting user requests, eliciting preferences, and generating responses, while correctness-critical operations, including product retrieval, hard-constraint filtering, brand exclusion, and progressive relaxation, are executed deterministically by the merchant agent. A session-persistent preference layer tracks constraints across turns, enabling consistent handling of budget overwrites and exclusion reversals. On a 140-query single-turn benchmark, MACS achieves the highest pass rate (87.1%) and perfect brand compliance (1.000). On a 10-scenario multi-turn benchmark, MACS achieves the strongest macro Pass@5 (72% vs. 56% GPT+Catalog / 52% Gemini+Catalog) with zero constraint drift. The advantage is sharpest on exclusion reversal (100% vs. 20% / 0%) and constraint accumulation (100% vs. 60% / 40%). Mean judged response quality is similar across systems (0.751 vs. 0.736). These results suggest that hybrid architectures combining deterministic constraint enforcement with session-persistent preference tracking provide stronger reliability-oriented performance than catalog-bound prompt-only baselines in the fixed-catalog merchant setting.
Figures & tables
Figure 1. MACS interaction flows. The AI shopping agent handles natural-language interaction and protocol generation; the merchant agent enforces hard constraints, validates inventory, and returns structured responses. Flow 1 (Search): the user’s query is parsed, constraint-filtered via the knowledge graph and SQL store, and results are returned. Flow 2 (Add to Cart): the merchant agent validates inventory before confirming. All constraint enforcement occurs exclusively in the merchant agent. Two interaction flow diagrams. Flow 1 (Search): User to AI Shopping Agent (parse request, generate UCP) to Merchant Agent (parse UCP, use KG, prices) to Create UCP response to AI Shopping Agent (present to user) to User (sees results). Flow 2 (Add to Cart): User to AI Shopping Agent to Merchant Agent (add to cart, validate inventory) to Create UCP response to AI Shopping Agent to User (cart updated).
Figure 2. MACS comparison and checkout view. Pros/cons are derived from structured catalog attributes, and delivery options and pricing are retrieved directly from the merchant agent’s relational store, not generated by the LLM. Side-by-side product comparison showing Lenovo IdeaPad vs.\ ThinkBook with pros/cons bullet points, delivery options (Standard, Express, Overnight), and itemized pricing breakdown.
Table 1. Best-value scoring weights and ϕ(p) spec-score components per use case ( cvol=0.05 for all).
Phase
Median (ms)
Notes
CPU phases (rewrite, filter)
< 1
Pure Python, no I/O
Domain detection — fast path
< 1
Keyword dictionary
Domain detection — LLM
∼ 1,630
gpt-4o-mini; ambiguous only
Criteria extraction (LLM)
1,927
Primary bottleneck
Question generation (LLM)
1,645
Preference elicitation
Post-rec intent detection
556
gpt-4o-mini classification
Table 2. Measured median latencies by pipeline phase ( n=10 runs per phase, direct-call benchmark; median reported to reduce sensitivity to occasional LLM cold-start outliers). LLM calls dominate total latency, and knowledge graph traversal and cache hits are negligible.
Table 4. Per-group single-turn results on the expanded 225-query benchmark (MACS only). Per-group Pass% uses threshold ≥0.65 (stricter, to identify reliably-passing groups); the " All " row uses ≥0.5 for comparability with Table 3 . All queries evaluated independently (no prior session context).
System
MT Avg ± SD
MT Pass@5
Drift
MACS
0.751 ± 0.095
72%
0.000
GPT+Catalog
0.736 ± 0.090
56%
0.000
Gemini+Catalog
0.717 ± 0.118
52%
0.014
Table 5. Multi-turn results (MT, 10 scenarios, Pass@5, threshold ≥0.65 ). MT scoring: 0.55× constraint (deterministic) +0.45× judge (GPT-4o-mini, temp. 0). All systems catalog-bound; same token budget; no web search. Pass@5 = macro fraction of 5 runs passing threshold. Drift: fraction of run-scenario pairs with a constraint violation. MACS uniquely achieves 100% Pass@5 on the two hardest state-management scenarios (S1: Constraint Accumulation, S8: Exclusion Reversal); aggregate mean score differences lie within one SD.
Table 7. Reliability, grounding, and disclosure evaluation ( n=8 catalog-impossible queries for Disclosure; n=10 scenarios × K=5 runs for Drift). Grounding: fraction of returned products verifiable in the catalog at stated price. Disclosure: fraction of impossible queries acknowledged explicitly. Drift: fraction of run-scenario pairs with a constraint violation. ↑ higher is better; ↓ lower is better.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
ID
Name
Turns
Key challenge
S1
Constraint Accumulation
4
Preserve all accumulated constraints
S2
Mind-Change Pivot
3
Use-case pivot (gaming → editing)
S3
Brand Excl. Persist.
4
Maintain brand exclusion across turns
S4
Vague → Specific
4
Elicit & resolve underspecified query
S5
Comparison + Follow-up
3
Context-free comparison
S6
Price Refinement
3
Budget update (narrowing)
Appendix
Table 8. Multi-turn descriptions (10 scenarios, 3–5 scripted turns each). “Key challenge” identifies the conversational state-management property under test. All scenarios are recommendation-focused except S4 and S5.
Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-level decision support: users reveal and revise constraints, coordinate multiple goals, and expect product-grounded recommendations over a full conversation. Existing benchmarks are mostly outcome-oriented or execution-oriented, leaving this evolving decision process under-evaluated. We introduce REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation. Our analysis shows that current systems produce locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially under ambiguous intent, bundle, and multi-intent scenarios. We further propose REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards. Experiments show that REALSHOP_AGENT consistently outperforms strong baselines on REALWORLDSHOP.
Xinwei Yang, Kelong Mao, Yudong Guo +4
College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore
Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3% of the mean final net assets achieved by human participants.
Qiming Shi, Yulong Tao, Linbo Jin +10
2State Key Lab of CAD&CG, Zhejiang University · 1Alibaba Group · School of Software and Microelectronics, Peking University +2
As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.