On Evaluating and Improving Conversational Agents in Production

Kasra Hosseini; Wen-Sen Cheng; Marco-Andrea Buchmann; Emir Mulabegovic; Weiwei Cheng

cs.MASep 25, 2026

On Evaluating and Improving Conversational Agents in Production

Authors: Kasra Hosseini, Wen-Sen Cheng, Marco-Andrea Buchmann, Emir Mulabegovic, Weiwei Cheng

Organizations: Zalando SE, Berlin, Germany · Zalando, Switzerland

Abstract

We present a framework for evaluating and improving a large-scale, multi-agent shopping assistant in production, and report lessons from its use. Offline evaluation of such a system faces three obstacles. (i) A logged conversation cannot be replayed against a modified system, because a different response changes every turn that follows. (ii) The unchanged system itself varies from run to run. Its LLM components are stochastic, and in product search the available products, their prices, and the customer's personalization signals change. (iii) Aggregate quality scores combine distinct behaviors, so they show that quality has changed but not which behavior caused the change. Our framework addresses each obstacle in turn. For a reported behavior, an Evaluation Harness generates targeted assertions and a fixed cohort of customer scenarios. It then reproduces the behavior in a local instance of the assistant through grounded user simulation. Instead of replaying the log, the simulator writes new customer turns conditioned on the recorded messages and context. Repeated runs of the unchanged system form a stored baseline. An Improvement Orchestrator turns the assertion results into hypotheses, implements each as an isolated modification, and compares it with the baseline using paired percentile bootstrap intervals over scenario-level differences. When an investigation ends, the harness may propose revisions to future evaluations, subject to human approval and without altering past decisions. We report production investigations with this framework. Assertion profiles showed which positions of a product carousel a failure affected, and repeated runs distinguished a real improvement from run-to-run fluctuation. Audits of the evaluation itself found a judge that lacked the evidence it needed and a model setting that was configured but not applied.

Figures & tables

Figure 1: The proposed evaluation and improvement framework. Reported problems and low-quality production interactions are turned into scenarios that reproduce the behavior in a local version of the Production Assistant . The simulator runs these scenarios against the unchanged system and against one modified version per hypothesis, and the resulting sessions are scored with the frozen assertions, shown in the figure as commissioned evaluation metrics. Assertion results feed back into hypothesis revision. Not shown are the stored baseline formed by repeated runs of the unchanged system, the human review of generated assertions before they are frozen, and the human-approved revision of the Evaluation Harness between investigations (Sects. 4 and 5 ).

Figure 2: The Production Assistant and its product-search action. In the upper portion, a customer request enters through the assistant interface and an LLM-based router selects the domain vertical that handles it. Conversation history, search state, and a consent-gated customer memory supply context to the verticals. The fashion vertical may start speculatively while routing is still in progress. The evaluation and self-improvement process at the upper right runs outside the serving path. The lower portion expands the search action used as the running example, from query generation through retrieval, ranking, and personalization to LLM-based curation of the carousel.

Case	Assertion or score	Baseline	Modified	$\Delta$	95% CI	Verdict
Category	Any product	$0.913\pm 0.012$	$0.947\pm 0.023$	$+0.033$	–	–
	First product	$0.833\pm 0.023$	$0.893\pm 0.031$	$+0.060$	–	–
	$\geq$ 75% of products	$0.733\pm 0.076$	$0.747\pm 0.031$	$+0.013$	–	–
	All products	$0.640\pm 0.040$	$0.647\pm 0.041$	$+0.007$	–	–
Gender, h1	First product	$0.733$	$0.800$	$+0.067$	$[-0.022,+0.178]$	borderline
Gender, h2	First product	$0.733$	$1.000$	$+0.267$	$[+0.067,+0.511]$	signal+

Table 1: Results from the two example investigations. Each assertion score is the cohort mean pass rate for whether the requested category or shopping context holds at the stated carousel positions. Task completion, in the last row, is not an assertion but a

1

–

5

quality score the judge assigns to the whole session. The category case used 51 production-grounded scenarios, 50 of which were scored in every run, with three runs per version. It predates the paired-bootstrap rule, so its values are means and standard deviations across runs, and dashes mark the unrecorded intervals and verdicts. The gender rows cover the nine gender-silent scenarios over the initial five runs per version, and intervals are paired 95% percentile bootstrap intervals over scenario-level differences. They are wide because the cohort provides only nine paired differences, one per scenario. Cohort size is set per behavior, and an inconclusive verdict triggers additional runs rather than acceptance. Pass rates refer to the stated behavior on a cohort selected to show the failure. They do not measure overall assistant quality or how often the behavior occurs in production.

Explore similar work

Jun 11, 2026cs.AI

Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce

We present a modular two-agent simulation framework for evaluating conversational shopping assistant architectures. An independent buyer agent, configured with personas, missions, and patience levels, is paired with an interchangeable responder that integrates with a real e-commerce search API. Holding the buyer constant across experiments enables controlled comparison of responder designs on identical scenarios. Using 2011 conversations across 14 persona buckets, we establish four empirical findings. First, rolling-window memory outperforms intent-extraction memory on all quality metrics while being 35% faster per query. Second, illustrating rapid evidence-driven iteration, a systematic failure analysis of a responder version enables targeted fixes that reduce failure and near-failure rates by 62% across the full dataset. Third, swapping the responder LLM backbone from Gemini~~2.5 to Llama~~3.3~70B costs 0.16--0.45 points despite identical architecture. Finally, we document systematic philosophical disagreement between frontier LLM judges: Gemini rewards process correctness while Claude demands concrete outcomes, despite using the same evaluation prompt.

Jetlir Duraj, Jayanth Yetukuri, Shuang Zhou +4

eBay Inc.

LLM Agent Memory E-Commerce Search LLM Agent Evaluation AI Agent Benchmarks Conversational Agents

Ask PDF

Jul 13, 2026cs.AI

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality. Although LLM-as-a-judge methods provide scalable alternatives to human evaluation, production deployment introduces challenges in governance, reproducibility, cost, schema consistency, traceability, and reliability. We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems. It processes production chatbot logs through normalization, sharding, asynchronous execution, and schema-constrained LLM scoring. The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions. Selective re-evaluation processes only incomplete, malformed, or schema-invalid records, while schema locking, versioned configurations, validation logs, and record-level provenance support auditability. The framework processes approximately 50,000 records daily and has evaluated more than two million interactions. Validation used 12,980 stratified-random human-labeled records from four trained annotators. Classification covered 14 intents, 156 sub-intents, 18 major domains, and 129 sub-domains. The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.

Niranjan Kumar M, Balaji Nagarajan, Karthik Nair +2

Lowe's

Language Model Generation Evaluation Customer Support Automation Multi-Turn Dialogue Evaluation Automated Evaluation Conversational Agents

Ask PDF

Sep 14, 2026cs.LG

Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification

A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score. We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule. A simulated consumer attempts to buy a target cart from the environment with assistance from the evaluated model. The environment guides the simulator's actions and records every assistant action alongside the environment state at that point. After the trial, these records allow the evaluator to assess individual parts of the conversation against the retained evidence. For example, the evaluator penalizes a search for failing to surface a target product only when the customer has already mentioned that product. We further use this evidence to apply different penalties to tool calls depending on how the assistant's actions compare with an expected tool-call set. Our environment also interacts with the simulator bidirectionally, reading its output to stop the trial when the simulator determines that the customer has become too frustrated and injecting directives in real time that specify when to explore, defer buying an item, or recall a previous exchange. This interaction creates an open-ended and verifiable simulation. Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search, all of which terminal success obscures.

Nimit Shah, Haitz Sáez de Ocáriz Borde

AION

Tool-Use Evaluation LLM Agent Evaluation Agent Evaluation Agentic Commerce

Ask PDF