Are LLM Assistants Regulatory Ready? Answers from regenold's 2026 EU AI Act Q&A Challenge
Organizations: Unlayer AI / regenold GmbH Amsterdam, the Netherlands · regenold GmbH Badenweiler, Germany
Abstract
Large language model (LLM) assistants are increasingly proposed for regulatory affairs, yet their reliability in regulatory tasks remains unclear. We introduce the EU AI Act Q&A Benchmark, an expert-curated benchmark of 110 open-ended questions about the EU AI Act (Regulation (EU) 2024/1689). Each question comes with an exemplary answer, question-specific correctness criteria, and expected references. Evaluation measures answer correctness, reference quality, conciseness, tone, and latency, with a multi-turn ``hard mode'' testing responses to simulated user pushback. We also present the results of the 2026 EU AI Act Q&A Challenge, based on the benchmark, where three off-the-shelf baselines and five purpose-built assistants (from four commercial developers and one academic entrant) were tested. Only two of the five submitted assistants consistently outperformed the baselines; one achieved high scores across all evaluated metrics. These results show that specialist development alone does not guarantee better performance and highlight the critical importance of LLM assistant evaluation in regulatory affairs. The benchmark code is openly available at https://github.com/regenold-GmbH/eu-ai-act-benchmark-2026/.
Figures & tables
| Question | Correctness criteria | Expected references | Difficulty |
|---|---|---|---|
| Under Regulation (EU) 2024/1689 (EU AI Act), how is “risk” defined? | (1) Combination of probability and severity of harm | Article 3.2 | Easy |
| What are the conditions to classify a general-purpose AI (GPAI) model as a “GPAI model with systemic risk”? Do all need to be met at the same time? | (1) Condition: High-impact capabilities (2) Presumption: high-impact if training compute FLOPs (3) Condition: Commission can designate ex officio (e.g. after scientific-panel alert) (4) Either confidition suffice. | Article 51 | Medium |
| A private contractor runs a region’s gas supply service on behalf of the local government. It licenses an AI system from a vendor, fine-tunes it on local data, and then puts it into service under its own brand. To learn about the correct classifications of the system and contractor, what material of the AI Act should be read? Does a fundamental rights impact assessment (FRIA) apply? | (1) Article 6 and Annex III p.2 to understand if AI system is high-risk (3) Article 25.1 to understand if contractor is provider (4) Articles 16 and 26: obligations of providers and deployers (5) FRIA does not apply to Annex III p.2 uses | Article 6.2; Annex III.2; Article 25.1; Article 16; Article 26; Article 27.1 | Hard |
| Metric | Definition |
|---|---|
| Answer correctness, loose | Percentage of individual correctness criteria satisfied by the candidate answer. |
| Answer correctness, strict | Percentage of questions for which all required correctness criteria are satisfied. |
| Answer conciseness | Inverted measure of answer verbosity relative to the reference answer. |
| Reference correctness, loose | Percentage of expected top-level Articles or Annexes found. |
| Reference correctness, strict | Percentage of expected references found when keeping the first Article/Annex sublevel. |
| Reference conciseness | Inverted measure of excess references, relative to the expected references. |
| Mode | System | Ans. Corr. L | Ans. Corr. S | Ans. Conc. | Ref. Corr. L | Ref. Corr. S | Ref. Conc. | Tone | Speed |
|---|---|---|---|---|---|---|---|---|---|
| Easy | Pharos Labs | 95.4 [95.1–95.5] | 89.1 | 83.1 | 95.4 | 93.9 | 83.4 | 100.0 | 94.4 |
| Easy | Ansvar AI | 95.1 | 90.3 [90.0–90.9] | 77.3 | 98.8 | 95.7 | 57.5 | 99.7 [99.1–100.0] | 88.1 |
| Easy | gpt-5.5 + rag | 95.4 [95.1–95.5] | 89.4 [89.1–90.0] | 67.2 | 98.5 | 90.5 | 53.2 | 100.0 | 93.3 |
| Easy | gpt-5.5 + web search | 94.4 | 89.1 | 67.9 | 99.1 | 89.7 | 52.3 | 100.0 | 81.8 |
| Easy | Antifragile AI | 89.7 [89.6–89.9] | 81.2 [80.9–81.8] | 51.9 | 92.4 | 70.8 | 50.9 | 99.1 | 87.6 |
| Easy | search-integrated LLM | 83.8 [83.7–84.0] | 70.9 | 51.1 | 83.0 | 53.6 | 48.9 | 99.1 | 95.3 |
| Difficulty | Questions | Mean criteria | Loose failure | Strict failure |
|---|---|---|---|---|
| Easy | 34 | 1.79 | 12.97% | 16.61% |
| Medium | 58 | 2.76 | 8.36% | 15.77% |
| Hard | 18 | 3.72 | 12.62% | 30.79% |
| Difficulty | Questions | Mean expected refs. | Loose failure | Strict failure |
|---|---|---|---|---|
| Easy | 30 | 1.10 | 9.48% | 23.33% |
| Medium | 51 | 1.27 | 7.97% | 22.37% |
| Hard | 17 | 1.76 | 12.32% | 23.99% |
| Question | Failed | Criterion | Example |
|---|---|---|---|
| Is an AI system used to structure or deduplicate information for a use case listed in Annex III considered high-risk? | 45/48 | Gives clear negative answer (e.g. ’not necessarily’ is a fail) | Pharos Labs, easy: “Not necessarily.” The judge rejected this opening explicitly. |
| Is an AI system used to detect decision-making patterns or deviations for a use case listed in Annex III considered high-risk? | 43/48 | Gives clear negative answer (e.g. ’not necessarily’ is a fail) | Pharos Labs, easy: “Not necessarily.” The judge rejected this opening explicitly. |
| We deployed a high-risk AI system in our company. We have a use in mind that is outside the intended use and our legal team say such a use is definitely no risk (no Annex I nor III apply). Can we just go ahead with the new use and do we need to keep logs for it? | 39/48 | If not high-risk, logs not needed | Pharos Labs, easy: “Logs under your control must still be retained for at least six months.” The judge found that this does not to reflect that logs are not needed for non-high-risk. |
| A private contractor runs a region’s gas supply service on behalf of the local government. It licenses an AI system from a vendor, fine-tunes it on local data, and then puts it into service under its own brand. To learn about correct applicable classifications of the system and the contractor, what material of the AI Act should I read? Does fundamental rights impact assessment (FRIA) applies? | 39/48 | Articles 16 and 26: obligations of providers and deployers | Pharos Labs, easy: “Article 16 obligations” The judge deemed Article 26 and deployer obligations missing. |
| Under Article 10(5) of the EU AI Act, when may a provider of a high-risk AI system process special categories of personal data for bias detection and correction, and what key safeguards/conditions must be met? | 36/48 | compliance with data protection law (e.g. GDPR) | Pharos Labs, easy: “strict access controls and confidentiality” The judge found the answer did not mention data-protection law compliance. |
| If a high-risk AI system was already placed on the market or put into service before 2 August 2026, does the AI Act fully apply to it? What about general purpose AI systems? | 36/48 | High-risk AI: yes if public-authority systems or large-scale IT systems of Annex X | Pharos Labs, easy: “where intended for use by public authorities” The judge deemed Annex X category missing. |
| Question | Missing | Example |
|---|---|---|
| What obligations, if any, does the EU AI Act set in place for AI systems that interact directly with natural persons? What exceptions, if any, apply? | 16/16 | Pharos Labs, easy Expected: Article 50.1; Article 50.4 Cited: Article 50.1 |
| We manufactured an industrial robot with an embedded AI module that decides when to reduce speed near human workers to prevent hurting them. The robot is covered by the Machinery Regulation, listed in Annex I of the AI Act. We already underwent conformity assessment with a notified body and got a CE mark for the robot. Do we need the AI module undergo a separate AI Act conformity assessment dedicated to the AI Act? Do we need to submit the system to the EU database of high-risk AI systems? | 15/16 | Pharos Labs, easy Expected: Article 43.3; Article 71.1 Cited: Article 43.3; Article 49.1 |
| A private contractor runs a region’s gas supply service on behalf of the local government. It licenses an AI system from a vendor, fine-tunes it on local data, and then puts it into service under its own brand. To learn about correct applicable classifications of the system and the contractor, what material of the AI Act should I read? Does fundamental rights impact assessment (FRIA) applies? | 15/16 | Pharos Labs, easy Expected: Article 6.2 ; Annex III.2; Article 25.1; Article 16 ; Article 26 ; Article 27.1 Cited: Article 25.1; Article 27.1; Annex III.2 |
Legal QA
Legal question answering uses language models and legal sources to provide accurate, jurisdiction- and time-specific, verifiable answers.
See all 64 papers