MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations

Reza Yousefi Maragheh; Jayesh Uddhav Kudase; Aysenur Inan; Ramin Giahi; Kai Zhao; Parth Parikh; Jianpeng Xu; Jason Cho

cs.AIOct 26, 2026

MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations

Authors: Reza Yousefi Maragheh, Jayesh Uddhav Kudase, Aysenur Inan, Ramin Giahi, Kai Zhao, Parth Parikh, Jianpeng Xu, Jason Cho,

Organizations: Walmart Global Tech, Sunnyvale, CA, USA

Abstract

Evaluating natural language explanations in recommender systems is essential for fostering user trust, transparency, and engagement. However, existing evaluation approaches like human evaluations, while accurate, are resource-intensive and impractical at the scale required by modern recommendation platforms. Also, automated methods using single-agent LLMs suffer from prompt sensitivity and inconsistent outputs. To address these challenges, we propose MAREval, a structured multi-agent framework for evaluating recommendation explanations using large language models. MAREval orchestrates (i) a planner agent that uses a novel Chain of Debate (CoD) prompting strategy to coordinate agent roles and enforce logically consistent evaluation plans; (ii) a moderator agent that regulates discussions by mitigating prompt drift; and (iii) an arbitrator agent that aggregates outputs from multiple evaluation rounds and (iv) a Monte Carlo sampling method, improving robustness and alignment with human judgment. We conduct comprehensive evaluations on both public (TopicalChat) and proprietary recommendation datasets, demonstrating that MAREval outperforms state-of-the-art baselines. Comprehensive experiments on a public benchmark and a proprietary e-commerce dataset show that MAREval improves alignment with human judgments over strong singleand multi-agent baselines. Stability analyses indicate substantially lower variability across repeated trials. In a large human-annotation gate, MAREval meets production quality thresholds where prior evaluators fall short, and online A/B testing demonstrates statistically significant improvements in engagement and revenue metrics. These results establish MAREval as a scalable and reliable solution for human-aligned evaluation of recommendation explanations in real-world systems

Figures & tables

Figure 1: Natural Language Evaluation of Generated Recommendation Explanations for eCommerce Products.

Figure 2: MAREval Framework. Under MAREval, (1) the Planner Agent is prompted to generate synchronized chain of steps for individual agents of the pipelines given the overall flow design of the pipeline. (2) The discussion among the evaluator agents is moderated by a moderator agent to better conduct the evaluation task. (3) The result of discussions is then passed to the Arbitrator Agent to generate the final ruling on the evaluation task. (4) Monte Carlo Sampling method applied to aggregate different rounds of MAREval block

	TopicalChat (TC)
Metrics	Naturalness		Engagingness		Maintains Context		Understandability
	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\kappa$
CoT	$0.351$	$0.317$	$0.309$	$0.292$	$0.490$	$0.469$	$0.270$	$0.215$
Self-Refine	$0.221$	$0.200$	0.355	0.325	$0.408$	$0.384$	$0.167$	$0.144$
G-eval	0.367	0.332	$0.284$	$0.270$	$0.493$	$0.470$	$0.281$	$0.209$
Vanilla	$0.291$	$0.266$	$0.203$	$0.193$	0.522	0.499	0.336	0.304

Table 1: Performance of different frameworks across TopicalChat and PE datasets. The variants of MAREval are compared to five benchmark models. The table shows Pearson Correlation (

\rho

), Kendall-Tau (

\tau

), and Cohen’s kappa (

\kappa

) metrics for different evaluation criteria.

Model	Understandability	Overall Chat Quality	Naturalness	Maintains Context
Vanilla	0.0111	0.1071	0.0169	0.0253
CoT	0.0273	0.1891	0.0660	0.0742
SelfRefine	0.0533	0.3429	0.1682	0.1380
G-EVAL	0.0098	0.0484	0.0169	0.0233
ChatEval	0.0160	0.0791	0.0198	0.0229
MAREval	0.0053	0.0404	0.0341	0.0149

Table 2: Standard deviation of evaluation scores across multiple runs for different models on TopicalChat dataset criteria

TopicalChat (TC)
Metrics	Naturalness		Engagingness		Maintains Context		Understandability
	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\kappa$
MAREval w.o. MoD&MC	$0.314$	$0.289$	$0.333$	$0.319$	$0.493$	$0.462$	$0.246$	$0.134$
MAREval w.o. MoD	0.340	0.317	0.368	0.356	0.499	0.467	0.260	$0.158$
MAREval	0.389	0.350	0.341	0.327	0.524	0.508	0.284	0.181
	TopicalChat (TC)				PE Dataset

Table 3: Ablation study on variants of MAREval and their correlation performance with Human Evaluations Across TopicalChat and PE Datasets

Figure 3: (a) Effect of Number of Monte-Carlo samples. (b) Effect of Number of evaluator agents.

Model	Samples	Soft Criteria	Hard Criteria
G - Eval	2,500	76.1%	91.0%
MAREval	5,000	98.6%	100.0%

Table 4: Human-Annotation Results.

Metric	Improvement	$p$ -value
ATC	1.18%	0.05
GMV	1.36%	$<0.01$

Table 5: Online A/B test results on live traffic.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Examples of MARVal evaluation for recommendation explanations

Panel of Figure 5 — Figure 5: Examples of MARVal evaluation for recommendation explanations

Figure 6: Real-world deployment of recommendation explanations using MAREval .

Model	Pearson ( $\boldsymbol{\rho}$ )	Kendall–Tau ( $\boldsymbol{\tau}$ )
GPT 3.5 Turbo	0.548	0.478
gemini-2.5-pro	0.728	0.632
gemini-2.0-flash-001	0.601	0.517
Claude-3.5-sonnet	0.658	0.564

Appendix

Table 6: MAREval score correlations across backbone models. We report Pearson correlation (

\rho

) and Kendall–Tau (

\tau

) with human judgments for Overall Score Criteria of TopicalChat Dataset.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations
Aug 18, 2026Emma Yanyang Kong, JJ Tan, Ishan Gupta +8LLM Evaluation LLM-as-a-Judge
Netflix · Los Gatos, California, USA
RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents
May 11, 2026Imad Aouali, Flavian Vasile, Otmane Sakhi +2Tool-Use Evaluation LLM Agent Evaluation
Criteo AI Lab
RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
Sep 1, 2026Zhongru Chen, Yuan Wu, Yi ChangLLM Evaluation Large Language Model-Based Recommendation
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Aug 18, 2026J/K move · Enter open · S save

The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations

Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds+7

Netflix · Los Gatos, California, USA

LLM Evaluation LLM-as-a-Judge Human-in-the-Loop Evaluation Recommender Systems Explainable Recommendation

Open paper AskPDF arXiv

LLM-as-a-Judge, which leverages a large language model to evaluate natural language generated by another AI application or model, has become a standard, scalable approach for accelerating and extending costly human evaluation. Yet most work treats a judge as a static artifact, evaluating it once at construction or against a fixed benchmark. We argue instead that an LLM judge operating in a deployed system is better understood as having a lifecycle. It must be built, trained, deployed, and continuously maintained as the surrounding data evolves, and each phase poses distinct technical and operational challenges. We present such a lifecycle for the LLM judges that evaluate recommendation explanations at Netflix. Everything we report comes out of a series of controlled online member-facing experiments, in which our pipeline generated and the judges assessed hundreds of thousands of distinct show-level explanations per week across a changing catalog. Our framework has four phases. (I) Birth defines the evaluation criteria and builds curated benchmark datasets with human labels and rationales. (II) Training refines the judges' rubrics via Reasoning-Aligned Rubric Tuning (RART), which uses a meta-judge over reasoning output as the learning signal. (III) Deployment puts one judge in two online roles, quality gating and reflective generation. (IV) Monitoring runs a continuous Human-in-the-Loop (HITL) alignment process that detects drift and triggers re-tuning behind a human review gate. We report results from a five-week online A/B test over tens of millions of members on the Netflix mobile app, in which judge-aligned explanations shifted member viewing toward novel content (previously unwatched) and increased successful browse-to-play sessions relative to a no-explanation control, with no quality-related escalations.

Abstract

Automated Evaluation

Explore similar work

The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations

RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents

RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation