Overcoming the "Impracticality" of RAG: Proposing a Real-World Benchmark and Multi-Dimensional Diagnostic Framework

Kenichirou Narita; Siqi Peng; Taku Fukui; Moyuru Yamada; Satoshi Munakata; Satoru Takahashi

cs.CLApr 3, 2026

Overcoming the "Impracticality" of RAG: Proposing a Real-World Benchmark and Multi-Dimensional Diagnostic Framework

Authors: Kenichirou Narita, Siqi Peng, Taku Fukui, Moyuru Yamada, Satoshi Munakata, Satoru Takahashi

Organizations: Artificial Intelligence Laboratories, Fujitsu Limited

Abstract

Performance evaluation of Retrieval-Augmented Generation (RAG) systems within enterprise environments is governed by multi-dimensional and composite factors extending far beyond simple final accuracy checks. These factors include reasoning complexity, retrieval difficulty, the diverse structure of documents, and stringent requirements for operational explainability. Existing academic benchmarks fail to systematically diagnose these interlocking challenges, resulting in a critical gap where models achieving high performance scores fail to meet the expected reliability in practical deployment. To bridge this discrepancy, this research proposes a multi-dimensional diagnostic framework by defining a four-axis difficulty taxonomy and integrating it into an enterprise RAG benchmark to diagnose potential system weaknesses.

Figures & tables

Figure 1: An overview of evaluation framework of our benchmark dataset. Each test query is tagged with difficulty labels based on our original taxonomy of RAG difficulty, which help provide a more comprehensive evaluation of the RAG system.

Dataset	Dimensions of Evaluation	Multi-document	Multimodal	Multi-hop	Evidence Traceability
DomainRAG	Generation	true	false	true (48%)	none
ETRQA	Generation	true	false	true (100%)	none
HeteQA	Generation	true	false	true (18%)	text
JDocQA	Generation	true	true	true (15%)	image
Multihop-RAG	Generation & Retrieval	true	false	true (88%)	text
VisDoMBench	Generation & Retrieval	true	true	true (100%)	text & image

Table 1: An overview of the features of the latest RAG benchmarks.

Dimension	Factors	Elements	Data Format	Evaluation Content
Reasoning Complexity	Reasoning Depth	Multi-step Reasoning	bool	The amount of LLM logical steps
	Reasoning Operation	Quantitative Operation Negation Question Cause-and-Effect Comparison (and Conditional Judgment) Temporal	bool	Quality of LLM’s logical operation
	Type of Output Processing	-	Exclusive Selection (Summarization, Translation, List Generation)	LLM’s answer generation capability
Retrieval Difficulty	Evidence Location	Multi-document Multi-chunk Low Locality Remote Reference	bool	Retriever’s handling of information dispersion
	Data Scale	Document Volume Chunk Size	bool	Retriever’s scalability
	Query-Evidence Relationship	Abstraction Discrepancy Vocabulary Mismatch	bool	Retriever’s semantic search capability

Table 2: Difficulty Definition for Enterprise RAG

Statistic	Value
Total Documents	34
Total Pages	1699
Total Queries	100
Chunk Unit	page
Average Chunks per Doc	51.485
Avg. Evidence Sources per Query	2

Table 3: Proposed Dataset Information

item	value
query	How do airborne communications and satellite communications differ in terms of A. how radio waves propagate, B. the amount of information that can be transmitted, and C. the difficulty of the technologies used?
answer	A. In aviation communications, radio waves can bend around obstacles, while satellite communications exhibit strong line-of-sight characteristics. B. Satellite communications can transmit larger amounts of information. C. Satellite communications involve more complex technology.
Bounding Box	top: 33.09, left: 48.84,width: 34.05,height: 25.95…
chunk infomation	Information and Communica- tions in Japan White Paper 2024 (English Version) :p.133

Table 4: Example questions for the proposed dataset

item	value
Reasoning Complexity
Reasoning Depth	True(Multi-step Reasoning)
Quantitative Operation	False
Negation Question	False
Cause and Effect	False
Comparison (and Conditional Judgment)	True

Table 5: Difficulty Metadata Example

Figure 2: Examples of Evidence Chunk. (Source: Ministry of Internal Affairs and Communications (MIC), Japan. (2024). Information and Communications in Japan White Paper 2024 (English Version). p.133. ( https://www.soumu.go.jp/johotsusintokei/whitepaper/eng/WP2024/2024-index.html )).

RAG System	JDoCQA	proposed dataset
MRI-RAG	0.074	0.09
Agentic AI	0.149	0.170

Table 6: Overall Accracy Comparison

Dimension	MRI-RAG	Agentic AI
Reasoning Complexity	2.5%	5.0%(+2.5pt)
Retrieval Difficulty	0.0%	12.0%(+12.0pt)
Source Structure & Modality	10.0%	20.0%(+10.0pt)
Explainability Requirement	9.1%	17.2%(+8.1pt)

Table 7: Dimensional Diagnostic Metrics (D-value) Comparison

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 786 papers

EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval
Aug 12, 2026Huiqi Miao, Xinbao Sun, Bo Wang +6Retrieval-Augmented Generation Instruction Following
Jiutian Research, China Mobile, Beijing, China
Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems
Feb 23, 2026Mukul Chhabra, Luigi Medrano, Arush VermaRAG Evaluation Multi-Turn Dialogue Evaluation
Dell Technologies, Austin, TX, USA
XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
Dec 20, 2024Qili Zhang, Qianren Mao, Yangyifei Luo +15Retrieval-Augmented Generation RAG Evaluation
†Zhongguancun Laboratory · ∗Beihang University · ‡Nanyang Technological University +1

Aug 12, 2026J/K move · Enter open · S save

EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval

Huiqi Miao, Xinbao Sun, Bo Wang, Fanyu Meng+5

Jiutian Research, China Mobile, Beijing, China

Retrieval-Augmented Generation Instruction Following LLM Reliability RAG Evaluation RAG Robustness

Open paper AskPDF arXiv

Enterprise RAG deployments face a critical reliability gap: while LLMs satisfy 80% of individual constraints, only 26.8% of responses meet all requirements simultaneously, revealing a 57-point orchestration gap. Existing benchmarks assume clean retrieval with simple queries, failing to capture production conditions where noisy documents and multi-dimensional constraints coexist. We introduce EnterpriseRAG, a benchmark of 983 expert-validated samples across six domains that systematically simulates three failure modes absent from prior work: retrieval noise, knowledge gaps, and factual conflicts, coupled with complex instructions. Evaluation of 13 state-of-the-art LLMs reveals a severe instruction adherence collapse, where high per-constraint satisfaction masks low holistic compliance. Critical findings expose deep barriers under knowledge gaps and factual conflicts, even with reasoning-enhanced inference, indicating production RAG requires explicit context-aware protocols and calibrated judgment. EnterpriseRAG provides a reproducible foundation for measuring and closing these gaps, directly informing deployment decisions for enterprise-scale RAG systems. We will release the benchmark and evaluation framework upon publication.

Abstract

Retrieval-Augmented Generation

Explore similar work

EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval

Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems

XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation