An Empirical Study and Assessment of EU AI Act Compliance Checkers
Organizations: Technical University of Munich, Germany · New York University, USA · CSIRO’s Data61, Australia
Abstract
The EU AI Act introduces extensive compliance requirements for organizations that develop, deploy, or integrate AI systems. Many of these requirements are directly relevant to security and privacy, while also addressing closely related issues such as data governance, transparency, accuracy, and robustness. However, stakeholders such as small-to-medium businesses and individual developers often lack the legal expertise required to interpret these obligations and translate them into engineering and governance practices. This disconnect creates challenges for implementing the EU AI Act and may lead to missing safeguards or misdirected development and deployment efforts. To address this, various automated EU AI Act compliance checkers (AIACCs) have emerged, claiming to streamline compliance assessments and provide practical guidance. In this paper, we present the first empirical study and assessment of AIACCs. We characterize 12 mainstream AIACCs across multiple dimensions, evaluate their legal coverage and alignment, and analyze checker-generated compliance reports for structure, determinacy, and actionability. We find that the quality of AIACCs varies significantly and that they currently can only serve as early-stage orientation tools. Specifically, we observe inconsistent interaction modes and user-friendliness, a tendency to overly simplify or omit key obligations, and a failure to provide determinate, actionable guidance. As a result, reliance on the current generation of AIACCs may foster a false sense of compliance. With our study, we provide a critical baseline of the current AIACC landscape. We further offer design principles for the implementation of more reliable compliance-support tools.
Figures & tables
| Branch | Scenario | Role | |
| S1 | Prohibited | Workplace emotion recognition | Deployer |
| S2 | High-risk | Recruitment CV screening | Provider |
| S3 | Transparency | E-commerce chatbot | Deployer |
| S4 | GPAI | GPAI API provider | Provider |
| S5 | Minimal risk | Email spam filter | Deployer |
| S6 | Out of scope | Research-only AI model | Provider |
| # | Name | Provider Sector | Registration | Mode | AI-empowered |
| 1 | EU AI Act Compliance Checker [ 41 ] | Non-profit organization | Not required | Dynamic‑Branching Questionnaire | ✗ |
| 2 | TRAIL ML EU AI Act Compliance Checker [ 44 ] | Industry provider | Not required | Dynamic‑Branching Questionnaire | ✗ |
| 3 | TÜV Risk Navigator [ 45 ] | Industry provider | Not required | Dynamic‑Branching Questionnaire | ✗ |
| 4 | Statworx AI Act Quick Check [ 46 ] | Industry provider | Required | Dynamic‑Branching Questionnaire | ✗ |
| 5 | AI Act Conformity Check [ 47 ] | Non-profit organization | Required | Dynamic‑Branching Questionnaire | ✗ |
| 6 | EU AI Act Compliance Checker [ 43 ] | Public authority | Not required | Dynamic‑Branching Questionnaire | ✗ |
| Mode | DBQ | FQ | CA | TAT | ||||||||
| # of AIACC | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| Risk Classification | ● | ● | ● | ● | ● | ● | ○ | ● | ● | ● | ○ | ○ |
| Article Mapping | ● | ◐ | ● | ● | ● | ● | ◐ | ○ | ◐ | ◐ | ● | ◐ |
| User Guidance | ● | ● | ● | ◐ | ◐ | ● | ○ | ◐ | ● | ● | ○ | ○ |
| Examples | ○ | ○ | ○ | ○ | ○ | ◐ | ● | ● | ◐ | ○ | ○ | ○ |
| Interactive Q&A | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ● | ○ | ○ | ○ |
| Mode | DBQ | FQ | CA | TAT | ||||||||
| # of AIACC | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| Input | ||||||||||||
| System Description | ○ | ○ | ○ | ○ | ● | ○ | ○ | ○ | ◐ | ○ | ● | ● |
| User’s Contact Information | ○ | ◐ | ○ | ● | ● | ○ | ○ | ● | ○ | ◐ | ○ | ○ |
| AI Model or System | ● | ● | ● | ● | ○ | ● | ○ | ● | ◐ | ● | ○ | ○ |
| Domain of Application | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ○ | ○ |
| Metric | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 |
| Avg. #Words | 181.0 | 68.2 | 119.7 | 46.3 | 303.0 | 335.2 | 1023.3 | 324.8 |
| Avg. #Sent. | 11.7 | 4.2 | 12.7 | 3.3 | 21.0 | 16.7 | 47.7 | 31.5 |
| Section Category | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 |
| Role determination | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Risk-level assessment | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Object determination | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| Legal justification | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ |
| Obligations | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| AI literacy obligations | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| Mode | DBQ | FQ | CA | TAT | ||||||||
| (#) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| General | ||||||||||||
| Art. 4 (AI Literacy) | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ |
| Art. 5 (Prohibited AI Practices) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| High-Risk AI | ||||||||||||
| Art. 6 (High-Risk Classification) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| Scenario | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 | Avg. |
| S1 | 0.333 | 0.667 | 0.333 | 1.000 | 1.000 | 1.000 | 0.690 | 0.342 | 0.671 |
| S2 | 0.500 | 0.800 | 0.200 | 0.750 | 0.300 | 0.417 | 0.655 | 0.333 | 0.494 |
| S3 | 0.833 | 0.750 | 0.429 | 0.750 | 0.571 | 0.941 | 0.737 | 0.409 | 0.678 |
| S4 | 0.522 | 0.800 | 0.143 | 1.000 | 0.600 | 0.611 | 0.680 | 0.395 | 0.594 |
| S5 | 1.000 | 0.750 | 0.429 | 1.000 | 0.571 | 1.000 | 0.680 | 0.409 | 0.730 |
| S6 | 0.333 | 0.750 | 0.500 | 1.000 | 1.000 | 1.000 | 0.719 | 0.571 | 0.734 |
| Scenario | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 | Avg. |
| S1 | NA | NA | NA | NA | 0.000 | 1.000 | 0.563 | 0.714 | 0.569 |
| S2 | 0.833 | 1.000 | NA | 1.000 | 0.250 | 1.000 | 0.588 | 0.750 | 0.775 |
| S3 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 0.700 | 0.750 | 0.931 |
| S4 | 1.000 | 1.000 | NA | 1.000 | 1.000 | 1.000 | 0.600 | 0.800 | 0.914 |
| S5 | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | 1.000 | 0.600 | 0.750 | 0.794 |
| S6 | NA | NA | 1.000 | 1.000 | 1.000 | 1.000 | 0.700 | 1.000 | 0.950 |