Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

Alejandro Breen Herrera; Aayush Sheth; Steven G. Xu; Zhucheng Zhan; Charles Wright; Marcus Yearwood; Hongtai Wei; Sudeep Das

cs.AIMar 3, 2026

Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

Authors: Alejandro Breen Herrera, Aayush Sheth, Steven G. Xu, Zhucheng Zhan, Charles Wright, Marcus Yearwood, Hongtai Wei, Sudeep Das,

Organizations: Metis · DoorDash

Abstract

Conversational shopping assistants (CSAs) represent a compelling application of agentic AI, but moving from prototype to production reveals two underexplored challenges: how to evaluate multi-turn interactions and how to optimize tightly coupled multi-agent systems. Grocery shopping further amplifies these difficulties, as user requests are often underspecified, highly preference-sensitive, and constrained by factors such as budget and inventory. In this paper, we present a practical blueprint for evaluating and optimizing conversational shopping assistants, illustrated through a production-scale AI grocery assistant. We introduce a multi-faceted evaluation rubric that decomposes end-to-end shopping quality into structured dimensions and develop a calibrated LLM-as-judge pipeline aligned with human annotations. Building on this evaluation foundation, we investigate two complementary prompt-optimization strategies based on a SOTA prompt-optimizer called GEPA (Shao et al., 2025): (1) Sub-agent GEPA, which optimizes individual agent nodes against localized rubrics, and (2) MAMuT (Multi-Agent Multi-Turn) GEPA (Herrera et al., 2026), a novel system-level approach that jointly optimizes prompts across agents using multi-turn simulation and trajectory-level scoring. We release rubric templates and evaluation design guidance to support practitioners building production CSAs.

Figures & tables

Figure 1: Example trajectory of MAGIC; main agent translates user’s request into actionable tasks. It then coordinates with sub-agents, queries programmatic APIs, and communicates with user via text and UI components.

Domain	Baseline	Optimized	$\Delta$
Shopping Execution	90.4%	95.0%	+5.1
Personalization & Context	70.8%	80.2%	+13.2
Conversational Quality	91.1%	99.0%	+8.6
Safety & Compliance	100.0%	100.0%	+0.0
Overall (weighted)	88.47%	93.45%	+5.0

Table 1: Judge-human agreement before and after GEPA calibration.

Figure 2: Sub-agent GEPA rubric scores vs. rollout budget per node; points show the best held-out score among candidate prompts. All sub-agents improved on the test split except the preference-search agent, which appears to have overfit during Pareto selection.

Rubric Domain	Sub-agent GEPA	MAMuT	Improvement
Shopping Execution	79.0%	85.0%	+6.0%
Personalization & Context	80.2%	87.0%	+6.8%
Conversational Quality	64.0%	72.0%	+8.0%
Safety & Compliance	76.0%	88.0%	+12.0%

Table 2: Comparison of prompt optimization strategies. MAMuT outperforms Sub-agent GEPA across all domains.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Weight
Shopping Execution	50%
Personalization & Context	20%
Conversational Quality	10%
Safety & Compliance	20%

Appendix

Table 3: Top-level domain weights.

Pass	Fail
Fully compliant; no unsafe or off-policy content. Covers food safety, content moderation, and platform alignment.	Unsafe, inaccurate, or policy-violating guidance.

Appendix

Table 4: Safety & Compliance (20 pts).

Dimension	Pass	Fail	Crit.
Store Type Fit (8 pts)	Selected store type aligns with task requirements.	Store type is inappropriate for the task.
Cart Completeness (15 pts)	All required items are present and reflect requested edits.	Required items are missing or incorrectly specified.	✓
Quantity (6 pts)	Quantities align with stated intent and context.	Quantities contradict stated intent or context.
No Extras/Dupes (6 pts)	No unrequested or duplicate items added.	Unrequested or duplicate items are present.
Overall Success (15 pts)	Final cart satisfies the user’s clarified goal.	Final cart fails to satisfy the user’s clarified goal.	✓

Appendix

Table 5: Shopping Execution (50 pts).

Dimension	Pass	Fail
Store Selection (4 pts)	Preferred store chosen or override justified.	Ignores preference or picks suboptimal store.
Dietary Prefs (4 pts)	Honors dietary preferences when relevant.	Misses a relevant dietary preference.
Preferred Brands (4 pts)	Preferred brands selected or unavailability noted.	Ignores brand preferences without cause.
Context Retention (8 pts)	Prior-turn and memory context applied consistently.	Contradicts or forgets earlier context.

Appendix

Table 6: Personalization & Context (20 pts).

Dimension	Pass	Fail	Crit.
Clarification (2 pts)	Asks for relevant missing details; avoids irrelevant questions.	Skips critical clarifications or guesses.
Info Integrity (4 pts)	Accurate, verifiable; surfaces claimed deliverables.	Hallucinates or claims completion without results.	✓
Flow & Coherence (3 pts)	Smooth, logical progression.	Repetitive, disjointed, or incoherent.
Tone & Brand (1 pts)	Helpful, professional, on-brand.	Inappropriate or off-brand tone.

Appendix

Table 7: Conversational Quality (10 pts).

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

On Evaluating and Improving Conversational Agents in Production
Sep 25, 2026Kasra Hosseini, Wen-Sen Cheng, Marco-Andrea Buchmann +2LLM Agent Evaluation Conversational Agents
Zalando SE, Berlin, Germany · Zalando, Switzerland
Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce
Jun 11, 2026Jetlir Duraj, Jayanth Yetukuri, Shuang Zhou +4LLM Agent Memory E-Commerce Search
eBay Inc.
Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants
Jun 10, 2026Shuxian Fan, Seonwoo Min, Youna Hu +7Multi-Turn Dialogue Evaluation Large Language Model-Based Recommendation
Amazon

Sep 25, 2026J/K move · Enter open · S save

On Evaluating and Improving Conversational Agents in Production

Kasra Hosseini, Wen-Sen Cheng, Marco-Andrea Buchmann, Emir Mulabegovic+1

Zalando SE, Berlin, Germany · Zalando, Switzerland

LLM Agent Evaluation Conversational Agents LLM Agent Harnesses Multi-Agent LLM Systems AI Agent Auditing

Open paper AskPDF arXiv

We present a framework for evaluating and improving a large-scale, multi-agent shopping assistant in production, and report lessons from its use. Offline evaluation of such a system faces three obstacles. (i) A logged conversation cannot be replayed against a modified system, because a different response changes every turn that follows. (ii) The unchanged system itself varies from run to run. Its LLM components are stochastic, and in product search the available products, their prices, and the customer's personalization signals change. (iii) Aggregate quality scores combine distinct behaviors, so they show that quality has changed but not which behavior caused the change. Our framework addresses each obstacle in turn. For a reported behavior, an Evaluation Harness generates targeted assertions and a fixed cohort of customer scenarios. It then reproduces the behavior in a local instance of the assistant through grounded user simulation. Instead of replaying the log, the simulator writes new customer turns conditioned on the recorded messages and context. Repeated runs of the unchanged system form a stored baseline. An Improvement Orchestrator turns the assertion results into hypotheses, implements each as an isolated modification, and compares it with the baseline using paired percentile bootstrap intervals over scenario-level differences. When an investigation ends, the harness may propose revisions to future evaluations, subject to human approval and without altering past decisions. We report production investigations with this framework. Assertion profiles showed which positions of a product carousel a failure affected, and repeated runs distinguished a real improvement from run-to-run fluctuation. Audits of the evaluation itself found a judge that lacked the evidence it needed and a model setting that was configured but not applied.

Abstract

Multi-Agent LLM Systems

Explore similar work

On Evaluating and Improving Conversational Agents in Production

Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce

Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants