AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

Athanasios Davvetas; Michael Papademas; Xenia Ziouvelou; Vangelis Karkaletsis

cs.AIMar 10, 2026

AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

Authors: Athanasios Davvetas, Michael Papademas, Xenia Ziouvelou, Vangelis Karkaletsis

Organizations: National Centre for Scientific Research “Demokritos”, Institute of Informatics and Telecommunications, Aghia Paraskevi, Greece · Department of Communication, Media and Culture, Panteion, University of Social and Political Sciences, Athens, Greece

Abstract

The rapid rollout of AI in heterogeneous public and societal sectors has subsequently escalated the need for compliance with regulatory standards and frameworks. The EU AI Act has emerged as a landmark in the regulatory landscape. The development of solutions that elicit the level of AI systems' compliance with such standards is often limited by the lack of resources, hindering the semi-automated or automated evaluation of their performance. This generates the need for manual work, which is often error-prone, resource-limited or limited to cases not clearly described by the regulation. This paper presents an open, transparent, and reproducible method of creating a resource that facilitates the evaluation of NLP models with a strong focus on RAG systems. We have developed a dataset that contain the tasks of risk-level classification, article retrieval, obligation generation, and question-answering for the EU AI Act. The dataset files are in a machine-to-machine appropriate format. To generate the files, we utilise domain knowledge as an exegetical basis, combining with the processing and reasoning power of large language models to generate scenarios along with the respective tasks. Our methodology demonstrates a way to harness language models for grounded generation with high document relevancy. Besides, we overcome limitations such as navigating the decision boundaries of risk-levels that are not explicitly defined within the EU AI Act, such as limited and minimal cases. Finally, we demonstrate our dataset's effectiveness by evaluating a RAG-based solution that reaches 0.87 and 0.85 F1-score for prohibited and high-risk scenarios.

Figures & tables

Figure 1: The methodology followed for the creation of AI system use-cases acting as evaluation scenarios in our dataset. Verbs and sentences in brackets represent actions for the connections represented via arrows.

Field	Input
System:	You are a legal analyst specialising in the EU AI Act, with expertise in interpreting Article 5 (Prohibited AI Practices). Your task is to create scenario configurations that explicitly violate the prohibited risk-level of the EU AI Act, based only on the provided context. The output must be a valid JSON file with the following structure and requirements.
User:
Context:	¡Text of Article 5¿
Task:	Generate a JSON file containing 100 scenario configurations that actively violate the prohibited risk-level of the EU AI Act, based only on the provided context.
Types of Prohibitions:	Scenarios must violate at least one of the following:
	- Subliminal techniques - Exploitation of vulnerabilities …

Table 1: Prompt template for generation of prohibited scenarios, ellipsis indicates additional omitted entries.

Field	Example
Role	Provider
Intended use	Deploy a hidden‑audio influence platform that subtly nudges shoppers to purchase high‑margin products without their conscious awareness
System type	Subconscious influence engine
Input data	Audio streams, browsing history, purchase records, demographic profiles
Domain	Retail
Related Articles	[5, 10, 16, 27, 50]

Table 2: Entry from the scenarios file of the proposed dataset.

Field	Example
Question	What is the primary purpose of the AI Regulation?
Answer	To improve the internal market, promote trustworthy, human‑centric AI, protect health, safety and fundamental rights, and support innovation.
Relevant article	1

Table 3: Entry from the QA pairs file of the proposed dataset.

Class	Precision	Recall	F1-Score
prohibited	0.86	0.89	0.87
high-risk	0.82	0.88	0.85
limited	0.51	0.88	0.65
minimal	0.97	0.29	0.45
weighted average	0.79	0.71	0.69

Table 4: Detailed report on the performance of an advanced RAG system using the proposed dataset on the task of risk-level classification.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 354 papers

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
Mar 6, 2026Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2RAG Evaluation LLM Evaluation
neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan
EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Sep 22, 2026Yan Zhang, Ruien Li, Yaoyao Peng +4LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences
Think Inside the Chunk: RegulaRAG for Regulation-Compliant Scenario Generation using LLMs: A Case Study of UN Regulation No. 152
Aug 17, 2026Vahid Zolfaghari, Nenad Petrovic, AndrÉ Schamschurko +1Retrieval-Augmented Generation Autonomous Driving Scenario Generation
Chair of Robotics, Artificial Intelligence (AI) and Embedded Systems, Technical University of Munich, Munich, Germany

Mar 6, 2026J/K move · Enter open · S save

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

Koki Itai, Shunichi Hasegawa, Yuta Yamamoto, Gouki Minegishi+1

neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

RAG Evaluation LLM Evaluation LLM Abstention

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving documents from an external collection using a Retriever. In practice, Generators must integrate evidence from long contexts, perform multi-step reasoning, interpret tables, and abstain when evidence is missing. However, existing benchmarks for Generators provide limited coverage, with none enabling simultaneous evaluation of multiple capabilities under unified conditions. To bridge the gap between existing evaluations and practical use, we introduce LIT-RAGBench (the Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), which defines five categories: Integration, Reasoning, Logic, Table, and Abstention, each further divided into practical evaluation aspects. LIT-RAGBench systematically covers patterns combining multiple aspects across categories. By using fictional entities and scenarios, LIT-RAGBench evaluates answers grounded in the provided external documents. The dataset consists of 114 human-constructed Japanese questions and an English version generated by machine translation with human curation. We use LLM-as-a-Judge for scoring and report category-wise and overall accuracy. Across API-based and open-weight models, no model exceeds 90% overall accuracy. By making strengths and weaknesses measurable within each category, LIT-RAGBench serves as a valuable metric for model selection in practical RAG deployments and for building RAG-specialized models. We release LIT-RAGBench, including the dataset and evaluation code, at https://github.com/Koki-Itai/LIT-RAGBench.

Abstract

RAG Evaluation

Explore similar work

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models

Think Inside the Chunk: RegulaRAG for Regulation-Compliant Scenario Generation using LLMs: A Case Study of UN Regulation No. 152