Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents

Kaiwen Luo; Ming Gao

cs.AISep 27, 2026

Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents

Authors: Kaiwen Luo, Ming Gao

Organizations: Meta

Abstract

Many LLM-as-a-judge evaluations score fixed outputs under a fixed task definition. Production multi-turn business agents instead require a maintained measurement system: correctness depends on business-specific facts and procedures, outcomes emerge across turns, and failures must be attributed to either agent capability or missing business knowledge before they are actionable. We present an integrated methodology spanning evaluation specification, modular LLM judges, intent-preserving user simulation, and human-in-the-loop governance. The specification defines conversation-level end states and actionable failure ownership. Atomic judges share versioned evidence and feed an explicit aggregation graph. The simulator is released only after task-preservation and stability checks. Independent human audits estimate measurement fidelity, renew tiered reference sets, and route disagreements to label correction, guideline revision, or judge improvement. Production studies show that system-level fidelity improved across repeated audits, that human reviewers and automated judges improved together under the shared feedback loop, and that their combined workflow had the strongest descriptive performance in both reported task-completion settings. Because the studies are observational and the human reference itself required revision, these findings demonstrate operational usefulness rather than causal or universal superiority. The contribution is a practical framework for making multi-turn agent measurement reliable, actionable, and maintainable as the evaluated system and its evidence evolve.

Figures & tables

Figure 1: The aggregation graph that implements the conversation-level specification. ACG denotes an agent-capability gap and BIG a business-information gap. The displayed Y/N values are raw predicate outcomes; downstream metrics use the metric-specific positive class defined above. Dashed nodes denote optional components outside the evaluated configuration.

Level	Dimension	Aggregation role
L5	factual hallucination	atomic
L5	procedure following	atomic
L4	action accuracy	atomic
L3	hallucination	OR of L4 + L5
L3	modality limitation	atomic
L3	missing data	atomic

Table 1: Structure of atomic and composite nodes in the aggregation graph.

Figure 2: The operational feedback system. Live judge outputs enter independent human review; audit results renew tiered golden sets; the golden sets test and allocate raters and benchmark judge revisions; fidelity dashboards and guardrails trigger investigation when the measurement system drifts. Dashed elements denote planned extensions.

Agent	judge only P / R / acc	human only P / R / acc	co-pilot P / R / acc	co-pilot $-$ judge $\Delta$ P / R / acc (pp)	co-pilot $-$ human $\Delta$ P / R / acc (pp)
Agent-A	66.7 / 53.8 / 75.6	51.3 / 73.1 / 67.9	71.4 / 76.9 / 82.1	+4.7 / +23.1 / +6.5	+20.1 / +3.8 / +14.2
Agent-B	55.8 / 68.6 / 75.0	56.1 / 51.4 / 75.0	67.6 / 71.4 / 81.7	+11.8 / +2.8 / +6.7	+11.5 / +20.0 / +6.7

Table 2: Task-completion fidelity of human, judge, and combined review workflows. Precision and recall treat task completion as the positive class. Takeaway: the combined workflow has the strongest descriptive point estimate in both reported task-completion settings. No confidence intervals or paired significance tests were available, so the table does not establish statistical superiority.

Figure 3: Coupled improvement of human reviewers and the LLM judge, measured against the same audit reference. Each panel is one evaluation dimension for Agent-A in a single locale; both curves are net detection rate (TPR $-$ FPR) on 14-day trailing windows of live traffic, over the first 138 daily readings of the audited period. Time is given as months elapsed since the first audited window. Heavy segments mark the mean of the first and last fourteen windows — the readings quoted in each panel subtitle — and the shaded band is the human–judge fidelity gap. All eight series end above where they start, and the gap narrows on every dimension: model capability 0.48 to 0.09, conversation failure 0.46 to 0.29, knowledge coverage 0.35 to 0.29, full resolution 0.24 to 0.15. The audit continues past the plotted interval; Section 9 records what the later readings do. These NDR readings are not comparable with the precision and recall in Table 2 , and a single continuous window from one agent and locale is observational: it shows co-movement under a shared feedback loop, not a causal effect of either arm on the other.

Explore similar work

Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents
Jun 9, 2026Sawyer Zhang, Alexander Wang, Sophie LeiLlm-As-A-Judge Blind Spots
Lumivate (Lumi)
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
May 18, 2026Leyao Wang, Yanan He, Peng Chen +5Llm-As-A-Judge Large Language Model Judges
Yale University · IBM Research
RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
May 20, 2026Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh +3Llm-As-A-Judge Chatbots
Layer 6 AI · University of Toronto

Jun 9, 2026, cs.CLJ/K move · Enter open · S save

Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents

Sawyer Zhang, Alexander Wang, Sophie Lei

Lumivate (Lumi)

Llm-As-A-Judge Blind Spots Defect

Open paper AskPDF arXiv

LLM-as-judge is the default instrument for evaluating conversational agents, yet its reliability is almost always reported as agreement with human ratings, not recall of real defects. We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems its built-in LLM judge catches, using exhaustive human transcript review as ground truth. Across three batches the judge surfaces well under a quarter of human-confirmed systematic problems -- 2 of 9 patterns (22%) in one batch, and its operational gate flagged zero of 100 rounds in a batch where humans confirmed 23 distinct defects and 7 new cross-cutting patterns. Our blind-spot taxonomy shows the failure is structured, not random: the judge catches turn-local issues (a fabricated statistic, a wrong language) but misses cross-turn state issues (confirm-gate lockout, cart hallucination, escalation lockout, stale referents). The mechanism: the scoring rubric exposes only three coarse axes (intent, brand-voice, personalization) and has no category for the behavioural dimensions -- state-tracking, guardrails, recovery -- where most defects cluster. The failure is routing, not perception: 113 of 114 rounds whose raw judge note describes a confirm-gate or cart-state defect are scored "brand voice", and none reach an operational failure -- the gate is wired to hangs and hard assertions, not the rubric -- so the 0% is a routing-and-wiring failure, not blindness. The consequence for prevalence estimation is sharp: when the apparent defect rate is zero the Rogan-Gladen correction degenerates -- no signal can recover the true rate -- while where the gate reports a nonzero rate the same estimator implies a 3-6x undercount under our measured sensitivity. For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.

Abstract

Explore similar work

Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents

Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator