Enterprise Retrieval-Augmented Generation (RAG) assistants operate in multi-turn, case-based workflows such as technical support and IT operations, where evaluation must reflect operational constraints, structured identifiers (e.g., error codes, versions), and resolution workflows. Existing RAG evaluation frameworks are primarily designed for benchmark-style or single-turn settings and often fail to capture enterprise-specific failure modes such as case misidentification, workflow misalignment, and partial resolution across turns. We present a case-aware LLM-as-a-Judge evaluation framework for enterprise multi-turn RAG systems. The framework evaluates each turn using eight operationally grounded metrics that separate retrieval quality, grounding fidelity, answer utility, precision integrity, and case/workflow alignment. A severity-aware scoring protocol reduces score inflation and improves diagnostic clarity across heterogeneous enterprise cases. The system uses deterministic prompting with strict JSON outputs, enabling scalable batch evaluation, regression testing, and production monitoring. Through a comparative study of two instruction-tuned models across short and long workflows, we show that generic proxy metrics provide ambiguous signals, while the proposed framework exposes enterprise-critical tradeoffs that are actionable for system improvement.
Figures & tables
Figure 1: End-to-end evaluation pipeline for case-aware LLM-as-a-Judge scoring of multi-turn RAG responses.
Metric
GPT-OSS (Short)
LLaMA (Short)
GPT-OSS (Long)
LLaMA (Long)
Hallucination (Grounding Fidelity)
0.6890
0.7431
0.7586
0.7132
Retrieval Correctness
0.7429
0.7647
0.7941
0.7578
Context Sufficiency
0.6526
0.6429
0.7411
0.6546
Answer Helpfulness
0.6943
0.6143
0.7724
0.6143
Answer Type Fit
0.7603
0.7274
0.8360
0.6825
Identifier Integrity
0.9421
0.9354
0.9559
0.9329
Table 1: Corrected retrieval-aware evaluation results. Scores reflect full judge evaluation with retrieved contexts provided.
Dataset
n
Mean (GPT)
Mean (Llama)
Δ
Wilcoxon p
Short Queries
70
0.7353
0.7202
+0.0151
0.6495
Long Queries
63
0.8099
0.7136
+0.0963
0.0011
Table 2: Paired Wilcoxon signed-rank test on per-case weighted aggregate scores. Δ denotes GPT minus Llama.
Subset
Judge
n
Mean (GPT-oss)
Mean (Llama)
Δ
Wilcoxon p
Short
GPT-4
70
0.7353
0.7202
+0.0151
0.6495
Short
Llama-3.3-70B
70
0.6639
0.6060
+0.0578
0.0005
Long
GPT-4
63
0.8099
0.7136
+0.0963
0.0011
Long
Llama-3.3-70B
63
0.7143
0.6589
+0.0554
0.0016
Table 3: Judge-robustness check on conversation-level weighted aggregate scores ( Sfinal ). Δ denotes GPT-oss minus Llama.
Metric
Llama
GPT-oss
Faithfulness (Short)
0.8439
0.5799
Answer Relevancy (Short)
0.7404
0.8192
Faithfulness (Long)
0.8123
0.6005
Answer Relevancy (Long)
0.7276
0.8330
Table 4: Representative proxy metrics produce mixed signals and do not capture workflow alignment or issue targeting.
Figure 2: Example metric weights for Sfinal . Rendered as a wide figure to avoid cut-off in two-column layout.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Correlation heatmap across evaluation metrics.
Figure 4: Score distributions per metric.
Figure 5: Word count distribution of LLM judge justifications per metric.
Figure 6: Radar visualization of normalized mean enterprise metric scores for long diagnostic queries. While statistical significance is reported in Table 2 , this figure provides a qualitative view of performance divergence across workflow-aligned dimensions.
Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.
Enterprise RAG deployments face a critical reliability gap: while LLMs satisfy 80% of individual constraints, only 26.8% of responses meet all requirements simultaneously, revealing a 57-point orchestration gap. Existing benchmarks assume clean retrieval with simple queries, failing to capture production conditions where noisy documents and multi-dimensional constraints coexist. We introduce EnterpriseRAG, a benchmark of 983 expert-validated samples across six domains that systematically simulates three failure modes absent from prior work: retrieval noise, knowledge gaps, and factual conflicts, coupled with complex instructions. Evaluation of 13 state-of-the-art LLMs reveals a severe instruction adherence collapse, where high per-constraint satisfaction masks low holistic compliance. Critical findings expose deep barriers under knowledge gaps and factual conflicts, even with reasoning-enhanced inference, indicating production RAG requires explicit context-aware protocols and calibrated judgment. EnterpriseRAG provides a reproducible foundation for measuring and closing these gaps, directly informing deployment decisions for enterprise-scale RAG systems. We will release the benchmark and evaluation framework upon publication.
Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate through the pipeline. We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness, factorized according to the pipeline's information flow. The model distinguishes task success. Whether the user received a correct answer (from generator success) and whether the generator behaved appropriately given the retrieval outcome. We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators, and show that the conditional decomposition reveals substantial behavioral differences between systems that appear equivalent under marginal metrics. We further analyze the annotation allocation problem, demonstrating that retrieval-success annotations are more informative than task-success annotations for estimating policy adherence, and provide an information-theoretic explanation for this asymmetry. Finally, we extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations, enabling practitioners to combine limited human judgments with cheaper automated assessments within a unified probabilistic model.
Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak +1
Centre for Artificial Intelligence ZHAW School of Engineering
Performance evaluation of Retrieval-Augmented Generation (RAG) systems within enterprise environments is governed by multi-dimensional and composite factors extending far beyond simple final accuracy checks. These factors include reasoning complexity, retrieval difficulty, the diverse structure of documents, and stringent requirements for operational explainability. Existing academic benchmarks fail to systematically diagnose these interlocking challenges, resulting in a critical gap where models achieving high performance scores fail to meet the expected reliability in practical deployment. To bridge this discrepancy, this research proposes a multi-dimensional diagnostic framework by defining a four-axis difficulty taxonomy and integrating it into an enterprise RAG benchmark to diagnose potential system weaknesses.