Is your AI Model Accurate Enough? The Difficult Choices Behind Rigorous AI Development and the EU AI Act
Organizations: Law & Tech Lab, Maastricht University, Maastricht, The Netherlands · University of Tübingen - Computational Law Lab, TÜbingen, Germany
Abstract
Technical and legal debates frequently suggest that "accuracy" is an objective, measurable, and purely technical property. We challenge this view, showing that evaluating AI performance fundamentally depends on context-dependent normative decisions. These techno-normative choices are crucial for rigorous AI deployment, as they determine which errors are prioritised, how risks are distributed, and how trade-offs between competing objectives are resolved. This paper provides a legal-technical analysis of the choices that shape how accuracy is defined, measured, and assessed, using the 2024 European Union AI Act -- which mandates an "appropriate level of accuracy" for high-risk systems -- as a primary case study. We identify and analyse four choices central to any robust performance evaluation: (1) selecting metrics, (2) balancing multiple metrics, (3) measuring metrics against representative data, and (4) determining acceptance thresholds. For each choice, we study its relationship to the AI Act's accuracy requirement and associated documentation obligations, show how its technical implementation embeds implicit or explicit assumptions about acceptable risks, errors, and trade-offs, and discuss the implications for the practical implementation of the AI Act by examples and related technical standards. By making the techno-normative dimensions of accuracy explicit, this paper contributes to broader interdisciplinary debates on AI governance and regulation, and offers specific guidance for regulators, auditors, and developers tasked with translating (legal) safety requirements into technical practice.
Figures & tables
| Metric | What It Measures (Definition) | What It Means (Implications) |
|---|---|---|
| Accuracy | The percentage of correct predictions out of the total number of predictions made. | Can be deceptive. In datasets where one outcome is rare (e.g., melanoma), a model can have high accuracy by simply ignoring the rare cases. |
| Precision | The quality of a positive prediction. When the AI claims something is true (e.g., “This is a risk”), how often is it actually true? | Minimises False Positives. High precision ensures the system does not flag innocent data or harmless events. Low precision can lead to user distrust, unnecessary blocks, or excessive use of resources. |
| Recall | The coverage of actual positives. Out of all the positive cases (e.g., melanoma), what percentage did the AI manage to find? | Minimises False Negatives. High Recall is essential for safety-critical AI (like melanoma detection) to ensure no dangers are missed, even if it causes some false alarms (by compromising on precision). |
| F1-Score (Aggregated Metric) | A combined score that balances Precision and Recall into a single number. | Balances Precision and Recall. Useful when the AI needs to balance caution ( Precision ) with safety ( Recall ) but hides the specific error type. |
| AUROC (Aggregated Metric) | A measure of the model’s ability to separate classes. It asks: “How likely is a random positive sample to be given a higher classification probability of being positive than a random negative sample ” | Goes beyond binary predictions . Unlike Accuracy , it considers the model’s continuous scores across different thresholds. However, it can be deceptive in cases of extreme class imbalance. |
AI Governance
Artificial intelligence governance studies how laws, institutions, and technical controls shape AI development, deployment, accountability, and social effects.
See all 384 papers