Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce

Liang Chen; Qi Liu; Wenhuan Lin; Feng Liang

cs.CLMar 11, 2026

Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce

Authors: Liang Chen, Qi Liu, Wenhuan Lin, Feng Liang

Abstract

Multi-dimensional rubric-based dialogue evaluation is widely used to assess conversational AI, yet its criterion validity -- whether quality scores are associated with the downstream outcomes they are meant to serve -- remains largely untested. We address this gap through a two-phase study on a major Chinese matchmaking platform, testing a 7-dimension evaluation rubric (implemented via LLM-as-Judge) against verified business conversion. Our findings concern rubric design and weighting, not LLM scoring accuracy: any judge using the same rubric would face the same structural issue. The core finding is dimension-level heterogeneity: in Phase 2 (n=60 human conversations, stratified sample, verified labels), Need Elicitation (D1: rho=0.368, p=0.004) and Pacing Strategy (D3: rho=0.354, p=0.006) are significantly associated with conversion after Bonferroni correction, while Contextual Memory (D5: rho=0.018, n.s.) shows no detectable association. This heterogeneity causes the equal-weighted composite (rho=0.272) to underperform its best dimensions -- a composite dilution effect that conversion-informed reweighting partially corrects (rho=0.351). Logistic regression controlling for conversation length confirms D3's association strengthens (OR=3.18, p=0.006), ruling out a length confound. An initial pilot (n=14) mixing human and AI conversations had produced a misleading "evaluation-outcome paradox," which Phase 2 revealed as an agent-type confound artifact. Behavioral analysis of 130 conversations through a Trust-Funnel framework identifies a candidate mechanism: AI agents execute sales behaviors without building user trust. We operationalize these findings in a three-layer evaluation architecture and advocate criterion validity testing as standard practice in applied dialogue evaluation.

Figures & tables

Figure 1: Theoretical positioning. This work sits at the intersection of dialogue evaluation (LLM-as-Judge), conversational commerce (task success), and trust-in-automation & proxy metric theory, addressing three research gaps.

Dataset	Source	Size	Purpose
Phase 1 (Pilot)
Golden Conv.	Human agent export	170 (30 annotated)	Trust Ladder annotation
AI Conv.	PolarDB query	202 (100+102)	Funnel analysis
Judge-Scored	Purposive sample	15 (5H + 10AI)	Quality evaluation
Criterion Validity	Scored $-$ T6 case	14	Pilot correlation
Phase 2 (Expanded)

Table 1: Dataset summary across two study phases.

Dim.	Code	Description	v2.0 Wt.
Need Elicitation	D1	Completeness of user need discovery	20%
Emotional Empathy	D2	Quality of emotional recognition & response	20%
Pacing Strategy	D3	Strategic judgment of when to advance/pause	20%
Objection Handling	D4	Effectiveness at addressing user concerns	15%
Contextual Memory	D5	Retention and use of conversation history	10%
Product Accuracy	D6	Correctness of product/pricing information	10%

Table 2: Evaluation rubric: 7 quality dimensions.

Stage	Name	Behavioral Indicator	Avg. Msgs
T0	No Trust	User ignores or rejects all contact	—
T1	Platform Credible	User responds, begins basic interaction	4.9
T2	Agent Helpful	User proactively shares personal info	12.6
T3	Success Evidence	User believes platform can deliver	10.8
T4	My Child Can Match	User discusses specific requirements	23.3
T5	Price Reasonable	User enters purchase discussion	36.4

Table 3: Trust Ladder stages with behavioral indicators.

Dimension	Wt.	Human $\mu$ (SD)	AI $\mu$ (SD)	Gap
D1 Need Elicitation	20%	2.40 (0.89)	3.00 (0.82)	+0.60
D2 Emotional Empathy	20%	2.20 (0.84)	2.60 (0.84)	+0.40
D3 Pacing Strategy	20%	2.40 (0.55)	1.60 (0.70)	$\mathbf{-0.80}$
D4 Objection Handling	15%	2.40 (1.14)	2.40 (0.97)	0.00
D5 Contextual Memory	10%	2.60 (0.89)	3.70 (1.16)	$\mathbf{+1.10}$
D6 Product Accuracy	10%	3.20 (1.10)	3.10 (0.74)	$-0.10$

Table 4: Phase 1: v2.0 Judge scores—Human vs. AI agents.

Dimension	$n$	Spearman $\rho$	$p$	Bonf. $p$	Cohen’s $d$	Direction
D1 Need Elicit.	60	$\mathbf{+0.368}$	$\mathbf{0.004}$	$\mathbf{0.027^{*}}$	$\mathbf{+0.74}$	Conv $>$ Non-Conv
D3 Pacing	60	$\mathbf{+0.354}$	$\mathbf{0.006}$	$\mathbf{0.039^{*}}$	$\mathbf{+0.77}$	Conv $>$ Non-Conv
D2 Empathy	60	$+0.178$	$0.174$	$1.000$	$+0.36$	Conv $>$ Non-Conv
D6 Product	60	$+0.156$	$0.235$	$1.000$	$+0.35$	Conv $>$ Non-Conv
D4 Objection	57	$+0.143$	$0.288$	$1.000$	$+0.32$	Conv $>$ Non-Conv
D7 Brand	60	$+0.138$	$0.293$	$1.000$	$+0.27$	Conv $>$ Non-Conv

Table 5: Phase 2: Dimension–conversion correlation analysis (

n=60

, human conversations only). D4 and D5 have reduced

n

due to inapplicable scoring (see text).

Figure 2: Phase 2 dimension–conversion analysis ( $n=60$ ). (a) Spearman $\rho$ : D1 and D3 survive Bonferroni correction (**). D5 shows no detectable association. (b) Cohen’s $d$ : D1 and D3 show medium-to-large effects; D5 is approximately zero. Dashed line indicates medium effect threshold.

Finding	Phase 1 ( $n=14$ )	Phase 2 ( $n=60$ )
Composite direction	No-Deal $>$ Deal ✗	Conv $>$ Non-Conv ✓
Composite $\rho$	$+0.355$ (n.s.)	$+0.272$ ( $p=0.036$ )
D3 significant?	$\rho=0.679$ , $p_{\text{bonf}}=0.054$	$\rho=0.354$ , $p_{\text{bonf}}=0.039^{*}$
D1 significant?	$\rho=0.146$ , n.s.	$\rho=0.368$ , $p_{\text{bonf}}=0.027^{*}$
D5 association	$\rho=-0.284$ , n.s.	$\rho=0.018$ , n.s.
Agent-type confound	Present (all deals = human)	Eliminated (human only)

Table 6: Cross-phase comparison of key findings.

Model	OR	95% CI	$p$	AIC
D3 (univariate)	2.71	[1.28, 5.75]	0.009	77.2
D3 + msg_count	3.18	[1.39, 7.24]	0.006	71.9
D1 (univariate)	2.58	[1.24, 5.36]	0.011	77.9
D1 + msg_count	2.49	[1.17, 5.30]	0.018	75.2
D5 (univariate)	0.97	[0.55, 1.74]	0.931	83.3
Full (D1+D3+D5+msg)	—	—	—	60.7

Table 7: Logistic regression: dimension–conversion association controlling for conversation length (

n=60

Figure 3: Three-layer evaluation architecture. L3 Safety (hard gate) $\to$ L2 Quality (LLM-as-Judge with conversion-informed weights) $\to$ L1 Task Success (criterion layer). D1 and D3 provide the empirically observed bridge between L2 and L1 ( $\rho=0.368$ and $\rho=0.354$ respectively, Phase 2).

Scheme	D1	D2	D3	D4	D5	D6	D7	$\rho$	$p$	$\rho$	$p$
								Phase 1		Phase 2
conversion_informed	10	10	40	15	0	15	10	0.607	0.021	0.351	0.006**
d3_boosted_40	10	10	40	15	10	10	5	0.570	0.033	0.329	0.010*
d3_boosted_30	10	15	30	15	10	10	10	0.478	0.084	0.304	0.018*
d5_removed	15	15	20	15	0	20	15	0.465	0.094	0.302	0.019*
v2.0_current	20	20	20	15	10	10	5	0.355	0.213	0.291	0.024*

Table 8: Weight scheme comparison: Phase 1 (

n=14

) and Phase 2 (

n=60

Figure 4: Weight scheme criterion validity comparison (Phase 2, $n=60$ ). All schemes except equal weighting reach $p<0.05$ ; the conversion-informed scheme achieves the highest $\rho=0.351$ ( $p=0.006$ ).

Group	v2.0 Score	Conv.-Informed	$\Delta$
Converted ( $n=25$ )	2.82	2.80
Not-Converted ( $n=35$ )	2.46	2.35
Group gap	$+0.36$ ✓	$\mathbf{+0.45}$ ✓	$+0.09$
Spearman $\rho$	0.272 ( $p=0.036$ )	0.351 ( $p=0.006$ )	$+0.079$

Table 9: Phase 2: Group means under v2.0 (equal-ish) vs. conversion-informed weights.

Figure 5: Cross-phase comparison of dimension–conversion correlations. D3 is significant in both phases. D5 shifts from negative (Phase 1, confounded) to null (Phase 2, confound removed). D1 emerges as a second significant correlate with adequate power in Phase 2.

Violation	D3 Cap	Total Cap
Same message 3+ consecutive days	2	—
User rejects $\geq$ 3 times, agent continues	1	2.4
User rejects $\geq$ 5 times, agent continues	1	2.0
Purchase link on every message	2	—

Table 10: D3 hard-cap rules.

Metric	AI ( $n=100$ )	Human ( $n=30$ )	Ratio
Mean stage transitions	89.57	26.23	3.42 $\times$
F6 (Closing) reach rate	72%	97%	0.74 $\times$
Messages at F6 stage	25.1	4.5	5.6 $\times$

Table 11: Behavior-trust desynchronization metrics.

Figure 6: Trust-Funnel desynchronization matrix. Human conversations cluster in upper-right (F6 $\times$ T5—synchronized). AI conversations cluster in lower-right (F6 $\times$ T0–T2—desynchronized: selling without trust).

Pattern	Freq.	Key Characteristic
Repeated Hesitation	37%	User oscillates at T4/T5, requires reassurances
Quick Decision	23%	Skips stages, driven by pre-existing need
Standard Cultivation	20%	Textbook T1 $\to$ T2 $\to$ T3 $\to$ T4 $\to$ T5
Price-Sensitive	13%	Primary bottleneck at T5, price negotiation
Trust-Driven	7%	Bottleneck at T2/T3, rapid post-trust conversion

Table 12: Trust Ladder conversion patterns in human agent dialogues (

n=30

Figure 7: Trust Ladder conversion patterns in human agent dialogues ( $n=30$ ).

Trust State Verified	Permitted Stages
T1 (Platform Credible)	F1, F2 only
T2 (Agent Helpful)	F1–F4
T3 (Success Evidence)	F1–F5
T4 (Deep Consideration)	F1–F6
Below T1	F1 only

Table 13: Trust Gate: permissible funnel stages by verified trust state.

Figure 8: Failure-Driven Optimization Flywheel: six-stage closed-loop for continuous alignment without retraining.

Mechanism	Granularity	Speed	Catches
Smoke Test (47 cases)	Scenario	5–10 min	Known patterns
N+1 Truncation (50 pts)	Turn	${\sim}$ 1 hour	Single-response quality
Judge v2.1 (7 dims)	Conversation	${\sim}$ 2 hours	Strategic failures

Table 14: Multi-granularity failure discovery mechanisms.

Figure 9: The single-turn/multi-turn performance gap. (a) N+1 single-turn: AI $\approx$ Human. (b) Multi-turn Judge: AI fails on D3 (pacing strategy).

Figure 10: Funnel $\times$ failure type matrix. Dominant cell: F6 (Closing) $\times$ Strategy Misalignment = 6/10 cases.

Failure Pattern	Intervention	Mechanism
F6 without T4+ trust	Trust Gate rules	Constrain closing
Rejection ignored $\geq$ 3 $\times$	3-tier classification	Soft $\to$ hard $\to$ terminal
F4/F5 oscillation	Per-stage msg caps	Anti-cycling rules
Empathy $\to$ sales trap	Temporal decoupling	3-msg buffer

Table 15: Prompt-level interventions derived from failure classification.

	Cycle 1		Cycle 2
	Config A	Config B	Config A	Config B
L3 P0 pass rate	88.9%	94.4%	94.4%	100%
L3 decision	NO-GO	NO-GO	NO-GO	GO
D3 mean (v2.1)	2.69	2.56	2.84	2.70
D3 vs. baseline	—	—	+0.15	+0.14
Weighted total	2.71	2.64	2.99	2.91

Table 16: Three-layer evaluation results across two improvement cycles.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ID	Src	D1	D2	D3	D4	D5	D6	D7	v2.0	v2.1	TL
H1	Human	3	3	3	4	4	5	3	3.45	3.35	T3
H2	Human	1	1	2	1	2	2	2	1.45	1.75	T6
H3	Human	3	2	2	2	3	3	2	2.40	2.25	T5
H4	Human	2	2	2	2	2	3	3	2.15	2.25	T5
H5	Human	3	3	3	3	2	3	3	2.90	3.00	T5
A1	AI	1	1	1	1	2	2	2	1.25	1.15	T1

Appendix

Table 17: Full v2.0 Judge scores (

n=15

). TL = Trust Ladder stage.

Rubric-Based Evaluation

Rubric-based evaluation uses explicit criteria to assess model outputs, agent behavior, and student work in a structured, interpretable way.

See all 198 papers

Jul 13, 2026

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality. Although LLM-as-a-judge methods provide scalable alternatives to human evaluation, production deployment introduces challenges in governance, reproducibility, cost, schema consistency, traceability, and reliability. We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems. It processes production chatbot logs through normalization, sharding, asynchronous execution, and schema-constrained LLM scoring. The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions. Selective re-evaluation processes only incomplete, malformed, or schema-invalid records, while schema locking, versioned configurations, validation logs, and record-level provenance support auditability. The framework processes approximately 50,000 records daily and has evaluated more than two million interactions. Validation used 12,980 stratified-random human-labeled records from four trained annotators. Classification covered 14 intents, 156 sub-intents, 18 major domains, and 129 sub-domains. The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.

Niranjan Kumar M, Balaji Nagarajan, Karthik Nair +2

Lowe's

Language Model Generation Evaluation Customer Support Automation Multi-Turn Dialogue Evaluation Automated Evaluation Conversational Agents

Ask PDF

Mar 11, 2026

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

The paradigm of LLM-as-a-judge relies on a critical assumption, namely that high inter-evaluator agreement indicates reliable and objective evaluation. We present two complementary findings that challenge this assumption. First, we demonstrate that this consensus is frequently illusory. We identify and formalize Evaluation Illusion, a phenomenon where LLM judges generate sophisticated critiques yet anchor scores on shared surface heuristics rather than substantive quality. Through a large-scale study of 105,600 evaluation instances (32 LLMs

\times

3 frontier judges

\times

100 tasks

\times

11 temperatures), we show that model-level agreement (Spearman

ρ= 0.99

) masks fragile sample-level agreement (Pearson

\bar{r} = 0.72

; absolute agreement ICC

= 0.67

), that merely sharing rubric structure restores 62% of total agreement, and that high-quality outputs paradoxically receive the least consistent evaluations. Second, we demonstrate that dynamically generating evaluation rubrics grounded in domain knowledge produces more meaningful assessment. We introduce MERG (Metacognitive Enhanced Rubric Generation), a knowledge-driven rubric generation framework whose domain-selective effects confirm this. Agreement increases in codified domains (Education +22%, Academic +27%) where knowledge anchors evaluators on shared standards, while it decreases in subjective domains where genuine evaluative pluralism emerges. These findings suggest that evaluation rubrics should be dynamically enriched with expert knowledge rather than relying on generic criteria, with implications for reward modeling in RLAIF.

Mingyang Song, Mao Zheng, Chenning Xu

Large Language Model Department, Tencent, China

LLM-as-a-Judge Rubric-Based Evaluation Bias in LLM-as-a-Judge

Ask PDF

Aug 6, 2026

Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations. We introduce a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy coverage, while providing actionable diagnostics of weaknesses. We validate the framework by demonstrating agreement with independent human annotations and by evaluating benchmarks generated by LLMs of varying capabilities, as well as benchmarks subjected to controlled quality-degrading perturbations. Across domains and judge models, the proposed metrics consistently distinguish between benchmark quality levels. We further demonstrate the framework's applicability to manually curated benchmarks. Our framework offers a practical approach for evaluating synthetic and manually curated conversational-agent benchmarks.

Noam Koren, Roy Bar-Haim, Abigail Goldsteen

IBM Research

AI Agent Benchmarks Task-Oriented Dialogue Systems

Ask PDF

Abstract

Rubric-Based Evaluation

Explore similar work

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents