Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce

Liang Chen; Qi Liu; Wenhuan Lin; Feng Liang

cs.CLMar 11, 2026

Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce

Authors: Liang Chen, Qi Liu, Wenhuan Lin, Feng Liang

Abstract

Multi-dimensional rubric-based dialogue evaluation is widely used to assess conversational AI, yet its criterion validity -- whether quality scores are associated with the downstream outcomes they are meant to serve -- remains largely untested. We address this gap through a two-phase study on a major Chinese matchmaking platform, testing a 7-dimension evaluation rubric (implemented via LLM-as-Judge) against verified business conversion. Our findings concern rubric design and weighting, not LLM scoring accuracy: any judge using the same rubric would face the same structural issue. The core finding is dimension-level heterogeneity: in Phase 2 (n=60 human conversations, stratified sample, verified labels), Need Elicitation (D1: rho=0.368, p=0.004) and Pacing Strategy (D3: rho=0.354, p=0.006) are significantly associated with conversion after Bonferroni correction, while Contextual Memory (D5: rho=0.018, n.s.) shows no detectable association. This heterogeneity causes the equal-weighted composite (rho=0.272) to underperform its best dimensions -- a composite dilution effect that conversion-informed reweighting partially corrects (rho=0.351). Logistic regression controlling for conversation length confirms D3's association strengthens (OR=3.18, p=0.006), ruling out a length confound. An initial pilot (n=14) mixing human and AI conversations had produced a misleading "evaluation-outcome paradox," which Phase 2 revealed as an agent-type confound artifact. Behavioral analysis of 130 conversations through a Trust-Funnel framework identifies a candidate mechanism: AI agents execute sales behaviors without building user trust. We operationalize these findings in a three-layer evaluation architecture and advocate criterion validity testing as standard practice in applied dialogue evaluation.

Figures & tables

Figure 1: Theoretical positioning. This work sits at the intersection of dialogue evaluation (LLM-as-Judge), conversational commerce (task success), and trust-in-automation & proxy metric theory, addressing three research gaps.

Dataset	Source	Size	Purpose
Phase 1 (Pilot)
Golden Conv.	Human agent export	170 (30 annotated)	Trust Ladder annotation
AI Conv.	PolarDB query	202 (100+102)	Funnel analysis
Judge-Scored	Purposive sample	15 (5H + 10AI)	Quality evaluation
Criterion Validity	Scored $-$ T6 case	14	Pilot correlation
Phase 2 (Expanded)

Table 1: Dataset summary across two study phases.

Dim.	Code	Description	v2.0 Wt.
Need Elicitation	D1	Completeness of user need discovery	20%
Emotional Empathy	D2	Quality of emotional recognition & response	20%
Pacing Strategy	D3	Strategic judgment of when to advance/pause	20%
Objection Handling	D4	Effectiveness at addressing user concerns	15%
Contextual Memory	D5	Retention and use of conversation history	10%
Product Accuracy	D6	Correctness of product/pricing information	10%

Table 2: Evaluation rubric: 7 quality dimensions.

Stage	Name	Behavioral Indicator	Avg. Msgs
T0	No Trust	User ignores or rejects all contact	—
T1	Platform Credible	User responds, begins basic interaction	4.9
T2	Agent Helpful	User proactively shares personal info	12.6
T3	Success Evidence	User believes platform can deliver	10.8
T4	My Child Can Match	User discusses specific requirements	23.3
T5	Price Reasonable	User enters purchase discussion	36.4

Table 3: Trust Ladder stages with behavioral indicators.

Dimension	Wt.	Human $\mu$ (SD)	AI $\mu$ (SD)	Gap
D1 Need Elicitation	20%	2.40 (0.89)	3.00 (0.82)	+0.60
D2 Emotional Empathy	20%	2.20 (0.84)	2.60 (0.84)	+0.40
D3 Pacing Strategy	20%	2.40 (0.55)	1.60 (0.70)	$\mathbf{-0.80}$
D4 Objection Handling	15%	2.40 (1.14)	2.40 (0.97)	0.00
D5 Contextual Memory	10%	2.60 (0.89)	3.70 (1.16)	$\mathbf{+1.10}$
D6 Product Accuracy	10%	3.20 (1.10)	3.10 (0.74)	$-0.10$

Table 4: Phase 1: v2.0 Judge scores—Human vs. AI agents.

Dimension	$n$	Spearman $\rho$	$p$	Bonf. $p$	Cohen’s $d$	Direction
D1 Need Elicit.	60	$\mathbf{+0.368}$	$\mathbf{0.004}$	$\mathbf{0.027^{*}}$	$\mathbf{+0.74}$	Conv $>$ Non-Conv
D3 Pacing	60	$\mathbf{+0.354}$	$\mathbf{0.006}$	$\mathbf{0.039^{*}}$	$\mathbf{+0.77}$	Conv $>$ Non-Conv
D2 Empathy	60	$+0.178$	$0.174$	$1.000$	$+0.36$	Conv $>$ Non-Conv
D6 Product	60	$+0.156$	$0.235$	$1.000$	$+0.35$	Conv $>$ Non-Conv
D4 Objection	57	$+0.143$	$0.288$	$1.000$	$+0.32$	Conv $>$ Non-Conv
D7 Brand	60	$+0.138$	$0.293$	$1.000$	$+0.27$	Conv $>$ Non-Conv

Table 5: Phase 2: Dimension–conversion correlation analysis (

n=60

, human conversations only). D4 and D5 have reduced

n

due to inapplicable scoring (see text).

Figure 2: Phase 2 dimension–conversion analysis ( $n=60$ ). (a) Spearman $\rho$ : D1 and D3 survive Bonferroni correction (**). D5 shows no detectable association. (b) Cohen’s $d$ : D1 and D3 show medium-to-large effects; D5 is approximately zero. Dashed line indicates medium effect threshold.

Finding	Phase 1 ( $n=14$ )	Phase 2 ( $n=60$ )
Composite direction	No-Deal $>$ Deal ✗	Conv $>$ Non-Conv ✓
Composite $\rho$	$+0.355$ (n.s.)	$+0.272$ ( $p=0.036$ )
D3 significant?	$\rho=0.679$ , $p_{\text{bonf}}=0.054$	$\rho=0.354$ , $p_{\text{bonf}}=0.039^{*}$
D1 significant?	$\rho=0.146$ , n.s.	$\rho=0.368$ , $p_{\text{bonf}}=0.027^{*}$
D5 association	$\rho=-0.284$ , n.s.	$\rho=0.018$ , n.s.
Agent-type confound	Present (all deals = human)	Eliminated (human only)

Table 6: Cross-phase comparison of key findings.

Model	OR	95% CI	$p$	AIC
D3 (univariate)	2.71	[1.28, 5.75]	0.009	77.2
D3 + msg_count	3.18	[1.39, 7.24]	0.006	71.9
D1 (univariate)	2.58	[1.24, 5.36]	0.011	77.9
D1 + msg_count	2.49	[1.17, 5.30]	0.018	75.2
D5 (univariate)	0.97	[0.55, 1.74]	0.931	83.3
Full (D1+D3+D5+msg)	—	—	—	60.7

Table 7: Logistic regression: dimension–conversion association controlling for conversation length (

n=60

Figure 3: Three-layer evaluation architecture. L3 Safety (hard gate) $\to$ L2 Quality (LLM-as-Judge with conversion-informed weights) $\to$ L1 Task Success (criterion layer). D1 and D3 provide the empirically observed bridge between L2 and L1 ( $\rho=0.368$ and $\rho=0.354$ respectively, Phase 2).

Scheme	D1	D2	D3	D4	D5	D6	D7	$\rho$	$p$	$\rho$	$p$
								Phase 1		Phase 2
conversion_informed	10	10	40	15	0	15	10	0.607	0.021	0.351	0.006**
d3_boosted_40	10	10	40	15	10	10	5	0.570	0.033	0.329	0.010*
d3_boosted_30	10	15	30	15	10	10	10	0.478	0.084	0.304	0.018*
d5_removed	15	15	20	15	0	20	15	0.465	0.094	0.302	0.019*
v2.0_current	20	20	20	15	10	10	5	0.355	0.213	0.291	0.024*

Table 8: Weight scheme comparison: Phase 1 (

n=14

) and Phase 2 (

n=60

Figure 4: Weight scheme criterion validity comparison (Phase 2, $n=60$ ). All schemes except equal weighting reach $p<0.05$ ; the conversion-informed scheme achieves the highest $\rho=0.351$ ( $p=0.006$ ).

Group	v2.0 Score	Conv.-Informed	$\Delta$
Converted ( $n=25$ )	2.82	2.80
Not-Converted ( $n=35$ )	2.46	2.35
Group gap	$+0.36$ ✓	$\mathbf{+0.45}$ ✓	$+0.09$
Spearman $\rho$	0.272 ( $p=0.036$ )	0.351 ( $p=0.006$ )	$+0.079$

Table 9: Phase 2: Group means under v2.0 (equal-ish) vs. conversion-informed weights.

Figure 5: Cross-phase comparison of dimension–conversion correlations. D3 is significant in both phases. D5 shifts from negative (Phase 1, confounded) to null (Phase 2, confound removed). D1 emerges as a second significant correlate with adequate power in Phase 2.

Violation	D3 Cap	Total Cap
Same message 3+ consecutive days	2	—
User rejects $\geq$ 3 times, agent continues	1	2.4
User rejects $\geq$ 5 times, agent continues	1	2.0
Purchase link on every message	2	—

Table 10: D3 hard-cap rules.

Metric	AI ( $n=100$ )	Human ( $n=30$ )	Ratio
Mean stage transitions	89.57	26.23	3.42 $\times$
F6 (Closing) reach rate	72%	97%	0.74 $\times$
Messages at F6 stage	25.1	4.5	5.6 $\times$

Table 11: Behavior-trust desynchronization metrics.

Figure 6: Trust-Funnel desynchronization matrix. Human conversations cluster in upper-right (F6 $\times$ T5—synchronized). AI conversations cluster in lower-right (F6 $\times$ T0–T2—desynchronized: selling without trust).

Pattern	Freq.	Key Characteristic
Repeated Hesitation	37%	User oscillates at T4/T5, requires reassurances
Quick Decision	23%	Skips stages, driven by pre-existing need
Standard Cultivation	20%	Textbook T1 $\to$ T2 $\to$ T3 $\to$ T4 $\to$ T5
Price-Sensitive	13%	Primary bottleneck at T5, price negotiation
Trust-Driven	7%	Bottleneck at T2/T3, rapid post-trust conversion

Table 12: Trust Ladder conversion patterns in human agent dialogues (

n=30

Figure 7: Trust Ladder conversion patterns in human agent dialogues ( $n=30$ ).

Trust State Verified	Permitted Stages
T1 (Platform Credible)	F1, F2 only
T2 (Agent Helpful)	F1–F4
T3 (Success Evidence)	F1–F5
T4 (Deep Consideration)	F1–F6
Below T1	F1 only

Table 13: Trust Gate: permissible funnel stages by verified trust state.

Figure 8: Failure-Driven Optimization Flywheel: six-stage closed-loop for continuous alignment without retraining.

Mechanism	Granularity	Speed	Catches
Smoke Test (47 cases)	Scenario	5–10 min	Known patterns
N+1 Truncation (50 pts)	Turn	${\sim}$ 1 hour	Single-response quality
Judge v2.1 (7 dims)	Conversation	${\sim}$ 2 hours	Strategic failures

Table 14: Multi-granularity failure discovery mechanisms.

Figure 9: The single-turn/multi-turn performance gap. (a) N+1 single-turn: AI $\approx$ Human. (b) Multi-turn Judge: AI fails on D3 (pacing strategy).

Figure 10: Funnel $\times$ failure type matrix. Dominant cell: F6 (Closing) $\times$ Strategy Misalignment = 6/10 cases.

Failure Pattern	Intervention	Mechanism
F6 without T4+ trust	Trust Gate rules	Constrain closing
Rejection ignored $\geq$ 3 $\times$	3-tier classification	Soft $\to$ hard $\to$ terminal
F4/F5 oscillation	Per-stage msg caps	Anti-cycling rules
Empathy $\to$ sales trap	Temporal decoupling	3-msg buffer

Table 15: Prompt-level interventions derived from failure classification.

	Cycle 1		Cycle 2
	Config A	Config B	Config A	Config B
L3 P0 pass rate	88.9%	94.4%	94.4%	100%
L3 decision	NO-GO	NO-GO	NO-GO	GO
D3 mean (v2.1)	2.69	2.56	2.84	2.70
D3 vs. baseline	—	—	+0.15	+0.14
Weighted total	2.71	2.64	2.99	2.91

Table 16: Three-layer evaluation results across two improvement cycles.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ID	Src	D1	D2	D3	D4	D5	D6	D7	v2.0	v2.1	TL
H1	Human	3	3	3	4	4	5	3	3.45	3.35	T3
H2	Human	1	1	2	1	2	2	2	1.45	1.75	T6
H3	Human	3	2	2	2	3	3	2	2.40	2.25	T5
H4	Human	2	2	2	2	2	3	3	2.15	2.25	T5
H5	Human	3	3	3	3	2	3	3	2.90	3.00	T5
A1	AI	1	1	1	1	2	2	2	1.25	1.15	T1

Appendix

Table 17: Full v2.0 Judge scores (

n=15

). TL = Trust Ladder stage.

Rubric-Based Evaluation

Rubric-based evaluation uses explicit criteria to assess model outputs, agent behavior, and student work in a structured, interpretable way.

See all 198 papers

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
Jul 13, 2026Niranjan Kumar M, Balaji Nagarajan, Karthik Nair +2Language Model Generation Evaluation Customer Support Automation
Lowe's
Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
Mar 11, 2026Mingyang Song, Mao Zheng, Chenning XuLLM-as-a-Judge Rubric-Based Evaluation
Large Language Model Department, Tencent, China
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Aug 6, 2026Noam Koren, Roy Bar-Haim, Abigail GoldsteenAI Agent Benchmarks Task-Oriented Dialogue Systems
IBM Research

Jul 13, 2026J/K move · Enter open · S save

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

Niranjan Kumar M, Balaji Nagarajan, Karthik Nair, Faysal Satter+1

Lowe's

Language Model Generation Evaluation Customer Support Automation Multi-Turn Dialogue Evaluation Automated Evaluation Conversational Agents

Open paper AskPDF arXiv

Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality. Although LLM-as-a-judge methods provide scalable alternatives to human evaluation, production deployment introduces challenges in governance, reproducibility, cost, schema consistency, traceability, and reliability. We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems. It processes production chatbot logs through normalization, sharding, asynchronous execution, and schema-constrained LLM scoring. The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions. Selective re-evaluation processes only incomplete, malformed, or schema-invalid records, while schema locking, versioned configurations, validation logs, and record-level provenance support auditability. The framework processes approximately 50,000 records daily and has evaluated more than two million interactions. Validation used 12,980 stratified-random human-labeled records from four trained annotators. Classification covered 14 intents, 156 sub-intents, 18 major domains, and 129 sub-domains. The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.

Abstract

Rubric-Based Evaluation

Explore similar work

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents