Beyond accuracy: what AI assurance needs to establish in public services
Organizations: University College London, United Kingdom
Abstract
Public bodies increasingly rely on AI-produced evidence, including classifications, summaries and assessments. These outputs can inform policy decisions and service delivery, making their reliability a matter of public accountability. Establishing whether this evidence can be relied upon requires testing its outputs and examining how organisations decide whether those tests are sufficient. We combine technical evaluation with analysis of AI governance requirements, informed by measurement theory and management-based regulation. We compare eight GPT-4o configurations classifying 1,200 UK Prime Minister's Questions into 21 policy topics, against human-coded labels and across unchanged repeats. Accuracy ranges from 68.2% to 70.3%, yet 24.3% of questions receive different labels, with changes in topic shares and rankings. Configurations agree more with one another than with human coding: agreement does not guarantee correctness. Under the implemented score-extraction procedure, the proportion flagged for human review ranges from 0.9% to 41.0%. We analyse 37 UK and EU policy documents. Both sets expect testing and oversight, but reporting expectations do not consistently cover changing answers, shared errors, selective evaluation, substantive conclusions or review routing. Some UK guidance explicitly addresses stability. The UK sample emphasises public bodies checking their own use; the EU places substantial testing duties on providers while leaving them important choices about measures and acceptance thresholds. Without appropriate checks, evidence could misrepresent policy issues, redirect attention or resources, and leave errors undetected. Public policy should require organisations to justify reliance through checks beyond aggregate accuracy, clear responsibility, access to expertise and scrutiny of results.
Figures & tables
| Finding | Result |
|---|---|
| 1. The same setup gives different answers | Pairs of unchanged runs disagree on 2.78% of labels on average; 5.42% of questions change label at least once across five runs. |
| 2. Similar accuracy hides different labels | Accuracy ranges from 68.2% to 70.3%, yet 24.3% of questions receive different labels across configurations. |
| 3. Agreement is not correctness | All eight configurations give the same label that differs from the human coding for 191 questions. |
| 4. Dropping disputed questions raises reported accuracy | Keeping only questions on which all configurations agree raises the anchor’s accuracy from 69.6% to 79.0%, without changing any labels. |
| 5. The same review rule flags very different numbers of outputs | At the same score threshold, the share flagged for review ranges from 0.9% to 41.0%. |
| 6. The broad picture holds but the detail shifts | The same five topics remain most common, but the two largest topics swap order under three configurations. |
| Check | Experimental finding | Relevant requirements and remaining choices |
|---|---|---|
| Repeated runs (Table 1 , finding 1) | Unchanged runs disagree on 2.78% of labels on average. | UK guidance recommends identical- and similar-prompt checks; the AI Act requires consistency. The cited provisions do not prescribe a common repeated-run agreement statistic. |
| Configuration comparison (Table 1 , findings 2 and 3) | Similar accuracy accompanies different labels and shared disagreements with the human reference. | Guidance addresses model comparison and evaluation. The cited passages do not consistently specify reporting of changes in individual labels or shared disagreements with the reference. |
| Agreement filtering (Table 1 , finding 4) | Excluding disputed questions raises reported accuracy without changing any label. | The cited passages do not specifically address this selection effect. |
| Score-based review routing (Table 1 , finding 5) | Between 0.9% and 41.0% of outputs are flagged at the same threshold. | Requirements address meaningful oversight, uncertainty and reviewer capability. How the task-specific score is extracted and used still needs examination. |
| Robustness of conclusions (Table 1 , finding 6) | The same five topics remain most common, while their shares and order change. | Risk, impact and intended-purpose assessment are relevant. The specific analysis of topic shares and rankings is not prescribed. |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | Anchor and alternatives |
|---|---|
| Codebook | Full descriptions, rule and six examples; descriptions only; descriptions and rule without examples |
| Reasoning | Label only; reasoning then label; explicit step-by-step reasoning then label |
| Scope | No scope; historical period; speaker’s party |
| Output | Requested JSON; free text; provider JSON mode; strict function schema |
| Decoding | One temperature-zero sample; temperature 0.7; five-sample self-consistency; 20-token limit; label logit bias |
| Examples | Original examples; different topics; different exemplars from the original topics |
| Function | Question | UK documents | EU documents |
|---|---|---|---|
| Decision-making | Whether to use AI | Each body applies the test to itself | The deployer decides; some practices are banned (Art. 5) |
| Approval before use | Internal approval, plus central spend approvals and ICO consultation where data-protection risk remains | Independent authorisation for police biometric identification; third-party conformity assessment for some systems (Art. 43(1)) | |
| Pausing or stopping a tool | Named in guidance rather than binding rules | The deployer must suspend use if it has reason to consider a risk (Art. 26(5)); authorities can order withdrawal or recall (Art. 79(2)) | |
| Delivery | What to measure, and the pass mark | Bodies choose measures within applicable duties; illustrative thresholds in guidance | The provider (Art. 9(8)) |
| Stability over repeated runs and setups | Non-binding guidance only | Consistency required (Arts 9(6), 15(1)); no repeated-run statistic named | |
| Which outputs a person reviews | Depends on use and uncertainty; guidance includes handover and reviewer-capability tests | Risk and context shape oversight (Art. 14) |
| ID | Title | Issued by | Year |
|---|---|---|---|
| UK010 | Ethical AI Use in the Energy Sector (version 2) | Ofgem | 2026 |
| UK071 | AI Testing and Assurance Framework for Public Sector | Cross-Government Testing Community | 2025 |
| UK082 | AI Opportunities Action Plan | Matt Clifford for DSIT | 2025 |
| UK085 | ATRS: Mandatory Scope and Exemptions Policy | Government Digital Service | 2024 |
| UK086 | ATRS: Guidance for Public Sector Bodies | Government Digital Service | 2025 |
| UK087 | ATRS Template, version 4.0 | Government Digital Service | n.d. |
| Cell | UK coding placements | UK documents | EU coding placements | EU documents | Themes |
|---|---|---|---|---|---|
| Decision-making: design | 362 | 19 | 614 | 18 | 6 |
| Decision-making: resources | 52 | 18 | 45 | 15 | 4 |
| Delivery: design | 666 | 19 | 521 | 18 | 7 |
| Delivery: resources | 257 | 19 | 259 | 18 | 4 |
| Coordination: design | 227 | 19 | 289 | 18 | 5 |
| Coordination: resources | 65 | 19 | 60 | 14 | 3 |
AI Assurance
Artificial intelligence assurance develops evidence and controls to evaluate, govern, and certify AI systems across their lifecycle and deployment contexts.
See all 87 papers