cs.CLMar 6, 2026

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

Authors: Koki Itai, Shunichi Hasegawa, Yuta Yamamoto, Gouki Minegishi, Masaki Otsuki

Organizations: neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving documents from an external collection using a Retriever. In practice, Generators must integrate evidence from long contexts, perform multi-step reasoning, interpret tables, and abstain when evidence is missing. However, existing benchmarks for Generators provide limited coverage, with none enabling simultaneous evaluation of multiple capabilities under unified conditions. To bridge the gap between existing evaluations and practical use, we introduce LIT-RAGBench (the Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), which defines five categories: Integration, Reasoning, Logic, Table, and Abstention, each further divided into practical evaluation aspects. LIT-RAGBench systematically covers patterns combining multiple aspects across categories. By using fictional entities and scenarios, LIT-RAGBench evaluates answers grounded in the provided external documents. The dataset consists of 114 human-constructed Japanese questions and an English version generated by machine translation with human curation. We use LLM-as-a-Judge for scoring and report category-wise and overall accuracy. Across API-based and open-weight models, no model exceeds 90% overall accuracy. By making strengths and weaknesses measurable within each category, LIT-RAGBench serves as a valuable metric for model selection in practical RAG deployments and for building RAG-specialized models. We release LIT-RAGBench, including the dataset and evaluation code, at https://github.com/Koki-Itai/LIT-RAGBench.

Figures & tables

Figure 2: Example of a co-occurrence across evaluation categories ( θR\theta_{R} and θT\theta_{T} )
Model NameVersionDeveloper
API Models
GPT-5 †2025-08-07OpenAI
GPT-5-mini †2025-08-07OpenAI
GPT-5-nano †2025-08-07OpenAI
o3 †2025-04-16OpenAI
o4-mini †2025-04-16OpenAI
Table 1: Evaluated LLMs (API-based and open-weight, † denotes reasoning models)
Figure 3: Evaluation results for overall accuracy. Blue bars represent Japanese scores, and red bars represent English scores.
ModelIntegrationReasoningLogicTableMain (avg)Abstention
jaenjaenjaenjaenjaenjaen
API Models
GPT-50.8330.8330.8700.7390.8670.9000.8390.8390.8520.8280.9000.933
GPT-5-mini0.9170.7500.8260.6960.8670.8670.7740.8060.8460.7800.8830.917
GPT-5-nano0.7500.6670.5650.7390.7330.8330.6770.7100.6810.7370.7670.817
o30.8330.8330.9570.8700.9000.9000.8390.8060.8820.8520.8170.833
Table 2: Category-wise accuracy comparison across models for Japanese and English evaluations. The best and second-best performances in each column are highlighted in bold and underlined, respectively. In cases of ties, all models achieving the same best or second-best score are highlighted accordingly.
Modeljaenavg
Claude-Sonnet-40.1480.3700.259
Llama-3.1-8B-Instruct0.2040.2220.213
Llama-3.3-70B-Instruct0.0930.2220.157
Gemma-3-27B-Instruct0.1670.0740.120
Qwen3-235B-A22B-Thinking0.1300.1110.120
GPT-4.10.0560.1110.083
Table 3: Over-Abstention Rates on the Japanese and English Evaluation Sets for the Main Category. The column avg indicates the mean of the Japanese ( ja ) and English ( en ) rates.
More in this topic

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 341 papers

Explore similar work

CardsList
May 9, 2026

Generating Leakage-Free Benchmarks for Robust RAG Evaluation

Retrieval-augmented generation (RAG) is widely used to augment large language models (LLMs) with external knowledge. However, many benchmark datasets, designed to test RAG performance, comprise many questions that can already be answered from an LLM's parametric memory. This leads to unreliable evaluation. We refer to this phenomenon as knowledge leakage: cases where RAG tasks are solvable without retrieval. This issue worsens over time due to benchmark aging. As benchmarks are reused for training, their contents are increasingly absorbed into model parameters, making them less effective for evaluating retrieval. We introduce SeedRG, a semi-synthetic benchmark generation pipeline that mitigates knowledge leakage and addresses the issue of benchmark aging. Starting from a seed benchmark dataset, SeedRG extracts a reasoning graph from question-context pairs to capture their underlying reasoning structure, and then generates new examples via type-constrained entity replacement. This process produces structurally similar but novel instances that are unlikely to exist in the model's parametric knowledge, while preserving the original reasoning patterns. To ensure quality, we incorporate two verification steps: (1) a reasoning-graph consistency check to maintain task difficulty, and (2) a knowledge-leakage filter to exclude instances answerable without retrieval.

Jiayi Liu, Jiaxing Zhang, Bowen Jin +1

Department of Computer Science, Purdue University · New Jersey Institute of Technology · University of Illinois at Urbana-Champaign +1

Benchmark ContaminationSynthetic Benchmark GenerationRAG EvaluationSynthetic Data Generation for Language ModelsRAG Robustness
AskPDF
Jul 31, 2026

Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests. We propose Q-CARE, a query-agnostic and fully reference-free framework that enables fine-grained assessment by decomposing queries into sub-queries and answers into atomic claims. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework. Code and data are publicly available at https://github.com/DISL-Lab/Q-CaRE-COLM-26.

Jeonghwan Choi, Taewon Yun, Minjeong Ban +3

1Korea Advanced Institute of Science and Technology (KAIST) · 2Cluvion

Reference-Free EvaluationRAG EvaluationAutomated EvaluationIR Evaluation
AskPDF
Jul 26, 2026

RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models

Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatible configurations, and compare it with a baseline test-input generator. Across 72,000 test executions, RagTester detected 21,633 failures, 6.6% more than the baseline, and outperformed it in 20 of the 24 configurations. The detected failures include inaccurate retrieval, unsupported answers, incomplete use of retrieved context, and difficulties interpreting complex passages. These results show that coverage-oriented test generation can effectively expose failures caused by the interaction between retrieval and generation components and support the assessment of RAG configurations before deployment.

Ange Maiztegi, Jon Ayerdi, Miren Illarramendi +1

Mondragon University, Arrasate-Mondragon, Gipuzkoa, Spain

RAG EvaluationAutomated Test Generation
AskPDF