Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems

Mukul Chhabra; Luigi Medrano; Arush Verma

cs.CLFeb 23, 2026

Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems

Authors: Mukul Chhabra, Luigi Medrano, Arush Verma

Organizations: Dell Technologies, Austin, TX, USA

Abstract

Enterprise Retrieval-Augmented Generation (RAG) assistants operate in multi-turn, case-based workflows such as technical support and IT operations, where evaluation must reflect operational constraints, structured identifiers (e.g., error codes, versions), and resolution workflows. Existing RAG evaluation frameworks are primarily designed for benchmark-style or single-turn settings and often fail to capture enterprise-specific failure modes such as case misidentification, workflow misalignment, and partial resolution across turns. We present a case-aware LLM-as-a-Judge evaluation framework for enterprise multi-turn RAG systems. The framework evaluates each turn using eight operationally grounded metrics that separate retrieval quality, grounding fidelity, answer utility, precision integrity, and case/workflow alignment. A severity-aware scoring protocol reduces score inflation and improves diagnostic clarity across heterogeneous enterprise cases. The system uses deterministic prompting with strict JSON outputs, enabling scalable batch evaluation, regression testing, and production monitoring. Through a comparative study of two instruction-tuned models across short and long workflows, we show that generic proxy metrics provide ambiguous signals, while the proposed framework exposes enterprise-critical tradeoffs that are actionable for system improvement.

Figures & tables

Panel of Figure 1 — Figure 1: End-to-end evaluation pipeline for case-aware LLM-as-a-Judge scoring of multi-turn RAG responses.

Metric	GPT-OSS (Short)	LLaMA (Short)	GPT-OSS (Long)	LLaMA (Long)
Hallucination (Grounding Fidelity)	0.6890	0.7431	0.7586	0.7132
Retrieval Correctness	0.7429	0.7647	0.7941	0.7578
Context Sufficiency	0.6526	0.6429	0.7411	0.6546
Answer Helpfulness	0.6943	0.6143	0.7724	0.6143
Answer Type Fit	0.7603	0.7274	0.8360	0.6825
Identifier Integrity	0.9421	0.9354	0.9559	0.9329

Table 1: Corrected retrieval-aware evaluation results. Scores reflect full judge evaluation with retrieved contexts provided.

Dataset	n	Mean (GPT)	Mean (Llama)	$\boldsymbol{\Delta}$	Wilcoxon $p$
Short Queries	70	0.7353	0.7202	+0.0151	0.6495
Long Queries	63	0.8099	0.7136	+0.0963	0.0011

Table 2: Paired Wilcoxon signed-rank test on per-case weighted aggregate scores.

\Delta

denotes GPT minus Llama.

Subset	Judge	$n$	Mean (GPT-oss)	Mean (Llama)	$\Delta$	Wilcoxon $p$
Short	GPT-4	70	0.7353	0.7202	+0.0151	0.6495
Short	Llama-3.3-70B	70	0.6639	0.6060	+0.0578	0.0005
Long	GPT-4	63	0.8099	0.7136	+0.0963	0.0011
Long	Llama-3.3-70B	63	0.7143	0.6589	+0.0554	0.0016

Table 3: Judge-robustness check on conversation-level weighted aggregate scores (

S_{\text{final}}

\Delta

denotes GPT-oss minus Llama.

Metric	Llama	GPT-oss
Faithfulness (Short)	0.8439	0.5799
Answer Relevancy (Short)	0.7404	0.8192
Faithfulness (Long)	0.8123	0.6005
Answer Relevancy (Long)	0.7276	0.8330

Table 4: Representative proxy metrics produce mixed signals and do not capture workflow alignment or issue targeting.

Panel of Figure 2 — Figure 2: Example metric weights for $S_{\text{final}}$ . Rendered as a wide figure to avoid cut-off in two-column layout.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Correlation heatmap across evaluation metrics.

Figure 4: Score distributions per metric.

Figure 5: Word count distribution of LLM judge justifications per metric.

Figure 6: Radar visualization of normalized mean enterprise metric scores for long diagnostic queries. While statistical significance is reported in Table 2 , this figure provides a qualitative view of performance divergence across workflow-aligned dimensions.

Figure 7: Weighted aggregate score ( $S_{final}$ ) vs. answer helpfulness.

Metric	Score field	Justification field
M1 Hallucination	hallucination_score	hallucination_justification
M2 Retrieval Correctness	retrieval_correctness_score	retrieval_correctness_justification
M3 Context Sufficiency	context_sufficiency_score	context_sufficiency_justification
M4 Answer Helpfulness	answer_helpfulness_score	answer_helpfulness_justification
M5 Answer Type Fit	answer_type_fit_score	answer_type_fit_justification
M6 Identifier Integrity	identifier_integrity_score	identifier_integrity_justification

Appendix

Table 5: Full evaluation schema used in batch processing.

Condition	$n$	Spearman $\rho$ on $S_{\text{final}}$
Short (Llama responses)	70	0.515
Short (GPT-oss responses)	70	0.577
Long (Llama responses)	63	0.691
Long (GPT-oss responses)	63	0.644

Appendix

Table 6: Inter-judge rank consistency between GPT-4 and Llama judges on weighted aggregate scores.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 352 papers

EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval
Aug 12, 2026Huiqi Miao, Xinbao Sun, Bo Wang +6Retrieval-Augmented Generation Instruction Following
Jiutian Research, China Mobile, Beijing, China
The RAT: A Unified Bayesian Model for RAG Evaluation
Aug 25, 2026Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak +1RAG Evaluation
Centre for Artificial Intelligence ZHAW School of Engineering
Overcoming the "Impracticality" of RAG: Proposing a Real-World Benchmark and Multi-Dimensional Diagnostic Framework
Apr 3, 2026Kenichirou Narita, Siqi Peng, Taku Fukui +3Retrieval-Augmented Generation RAG Evaluation
Artificial Intelligence Laboratories, Fujitsu Limited

Aug 12, 2026J/K move · Enter open · S save

EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval

Huiqi Miao, Xinbao Sun, Bo Wang, Fanyu Meng+5

Jiutian Research, China Mobile, Beijing, China

Retrieval-Augmented Generation Instruction Following LLM Reliability RAG Evaluation RAG Robustness

Open paper AskPDF arXiv

Enterprise RAG deployments face a critical reliability gap: while LLMs satisfy 80% of individual constraints, only 26.8% of responses meet all requirements simultaneously, revealing a 57-point orchestration gap. Existing benchmarks assume clean retrieval with simple queries, failing to capture production conditions where noisy documents and multi-dimensional constraints coexist. We introduce EnterpriseRAG, a benchmark of 983 expert-validated samples across six domains that systematically simulates three failure modes absent from prior work: retrieval noise, knowledge gaps, and factual conflicts, coupled with complex instructions. Evaluation of 13 state-of-the-art LLMs reveals a severe instruction adherence collapse, where high per-constraint satisfaction masks low holistic compliance. Critical findings expose deep barriers under knowledge gaps and factual conflicts, even with reasoning-enhanced inference, indicating production RAG requires explicit context-aware protocols and calibrated judgment. EnterpriseRAG provides a reproducible foundation for measuring and closing these gaps, directly informing deployment decisions for enterprise-scale RAG systems. We will release the benchmark and evaluation framework upon publication.

Abstract

RAG Evaluation

Explore similar work

EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval

The RAT: A Unified Bayesian Model for RAG Evaluation

Overcoming the "Impracticality" of RAG: Proposing a Real-World Benchmark and Multi-Dimensional Diagnostic Framework