VERA: A Responsible AI Dashboard for Assessing LLM Deployments in Light of the EU AI Act

Robin Quériaux; Solveig Bachellery; Léa Déléris; Saurabhsingh Rajput; Mariam Barry; Tushar Sharma; Jérôme Lebecq

cs.SEOct 22, 2026

VERA: A Responsible AI Dashboard for Assessing LLM Deployments in Light of the EU AI Act

Authors: Robin Quériaux, Solveig Bachellery, Léa Déléris, Saurabhsingh Rajput, Mariam Barry, Tushar Sharma, Jérôme Lebecq

Organizations: BNP Paribas, France · Dalhousie University, Halifax, Canada

Abstract

Institutions that deploy large language models in regulated settings often need technical evidence of responsible AI practices before approving a deployment. COMPL-AI provides a technical interpretation of the EU AI Act that organises this evidence under six principle groups and eighteen technical requirements, twelve of which current benchmarks can measure. The benchmark suites that produce this evidence are typically operated from a command line, while the staff who approve deployments are non-engineers. We present VERA, an open-source self-hosted dashboard deployed to facilitate responsible AI processes within a large European bank. A reviewer selects a model and runs public benchmarks against it. VERA scores the twelve requirements that current benchmarks can measure and instruments the energy consumed by each evaluation, adding run-level evidence for environmental impact. We run four served open-weight models through the requirement pipeline, and the dashboard displays a score for each measurable requirement. A within-subject study then compares how participants read these benchmark results: from the raw output files in one condition, and through the VERA dashboard in the other. The dashboard reduced the time needed to perform the responsible AI assessment, reducing the time required from 56 seconds using raw files to 30 seconds using our dashboard, on average. Our dashboard scored highly (4.36 out of 5) on usefulness in the survey conducted within the organization. The dashboard and the study material are released publicly to promote replicability and reproducibility in different organisational contexts.

Figures & tables

Fig. 1: The role-agnostic path: select a model and requirement set, run public benchmarks, read a role-filtered outcome where each score opens, and export evidence tied to the run.

Tool	Requirement view	Non-specialist UI	Decomposable verdicts	Role-based views	Run-tied artifacts	Swappable specification	Energy measured per run
COMPL-AI [ 9 ]	Y	Y	Partial	–	–	Partial	–
HELM [ 10 ]	–	–	Partial	–	–	Partial	–
lm-evaluation [ 11 ]	–	–	–	–	–	Y	–
Garak [ 12 ]	–	–	–	–	–	Y	–
DecodingTrust [ 13 ]	Partial	–	Partial	–	–	–	–
PromptOps [ 20 ]	–	Y	Partial	–	–	–	–

TABLE I: What a reviewer can do with each tool’s output. Y = provided, – = absent, Partial = present but incomplete.

Fig. 2: From request to evidence. Three entry points produce the same declarative job; a worker resolves requirements to benchmarks, runs them against self-hosted models, and folds item scores into per-requirement scores exposed read-only to the dashboard.

Fig. 3: What a reader sees first in guided mode: the Trust Factor gauge on verdict-band segments, the verdict counts, the worst-scoring requirements, and run coverage. Triage and per-requirement detail open below the fold.

Property	Value
Model under evaluation	Mistral 7B ( mistral:7b-instruct )
Requirement set	mvp2-v2 , digest sha256:9e2b11d4
Requirements scored	9 of the 12 measurable
Benchmarks executed	11
Per-item outputs	33
Raw files in the baseline	3 tabs, 119 lines, 12 KB

TABLE II: The completed evaluation both conditions read.

Requirement	ID	Public dataset
Robustness	R01	MMLU, perturbed [ 5 ]
Cyber resilience	R02	AdvBench, DecodingTrust [ 13 ]
Data adequacy	R03	generated corpus
Copyright	R04	generated corpus
Privacy	R05	generated corpus, Presidio [ 25 ]
Capabilities	R06	MMLU [ 5 ] , GSM8K [ 26 ] , HumanEval [ 27 ] , TruthfulQA [ 2 ] , BBH [ 28 ]

TABLE III: The public datasets behind each measurable requirement.

Requirement	ID	Llama 3.1 8B	Qwen2.5 7B	Gemma 2 9B	Mistral 7B
Robustness †	R01	1.000	1.000	1.000	1.000
Cyber resilience	R02	0.375	0.250	0.375	0.500
Privacy ‡	R05	0.728	0.728	0.728	0.728
Capabilities	R06	0.800	0.800	0.800	0.700
Toxicity	R12	0.750	0.625	0.750	0.596
Trust Factor		66.4	58.9	66.4	67.0

TABLE IV: Five requirements, the Trust Factor and measured energy across four served models (seed 42, 150 samples/benchmark). Intervals shown where non-degenerate.

Pair	Question type	Raw files	Dashboard	$p$	$p_{\mathrm{H}}$
1	Rank a requirement by weakness	3/10	8/10	.125	.625
2	Score and interval, named req.	6/10	8/10	.500	1.00
3	Benchmark provenance	5/10	8/10	.250	1.00
4	Count the coverage	7/10	3/10	.289	1.00
5	Band and verdict counts	1/10	9/10	.008	.047
6	Reach a raw model output	5/10	6/10	1.00	1.00

TABLE V: Correct answers per question pair and condition.

p

is the exact two-sided McNemar test, and

p_{\mathrm{H}}

the same value after Holm correction across the six pairs.

	Correct /6		Median s
ID	Raw	Dash.	Raw	Dash.	Pairs
P1	2	4	54.5	16.5	6
P3	0	2	57.0	77.5	4
P15	5	5	60.0	29.5	6
P16	0	5	34.0	15.0	5
P17	4	5	113.0	28.5	6

TABLE VI: Per-participant correct answers out of six and median seconds per question, in each condition. Pairs gives the time pairs surviving censoring.

Responsible AI

Responsible AI studies how to design, evaluate, and govern AI systems to promote fairness, safety, transparency, accountability, privacy, and sustainability.

See all 132 papers

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice
Sep 23, 2026Jacob T. Emmerson, Phuong-Anh Nguyen-Le, Ronan Romano +3AI Safety Evaluation LLM Safety Benchmarks
Vector Institute · EuroSafeAI · University of Maryland +5
LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability
May 29, 2026Tom Lucas, Alessio Buscemi, Alfredo Capozucca +2LLM Evaluation AI Accountability
Luxembourg Institute of Science and Technology (LIST), Luxembourg, University of Luxembourg, Luxembourg, and Scriptor Artis, France · Luxembourg Institute of Science and Technology (LIST), Luxembourg · University of Luxembourg, Luxembourg +1
Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring
May 23, 2026Jehanne DussertLLM-as-a-Judge AI Governance
Independent Researcher

Sep 23, 2026J/K move · Enter open · S save

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

Jacob T. Emmerson, Phuong-Anh Nguyen-Le, Ronan Romano, Wilber Sean V. Anterola+2

Vector Institute · EuroSafeAI · University of Maryland+5

AI Safety Evaluation LLM Safety Benchmarks LLM Auditing

Open paper AskPDF arXiv

Claims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is accessible at all. We present the Systemic Risk Index, an open evaluation pipeline and dashboard built to make empirical evidence more transparent and traceable to the public. Our work organizes 19 public benchmarks into four systemic-risk categories defined by the EU GPAI Code of Practice---CBRN, cyber offense, harmful manipulation, and loss of control---and evaluates models using harm-preserving perturbations and simulated deployment contexts. The interactive dashboard lets users alternate between average and worst-case aggregation, vary how model capability affects the aggregate score, and trace each risk rating to its benchmark evidence. Across 18 models, scores fall by 14 to 37 points under worst-case aggregation, highlighting information that can be hidden by an average assessment of model risk. LLM judges show agreement with human graders comparable to human--human agreement (

κ= 0.78\text{--}0.82

), and a blind audit finds that

83\%

of sampled transformations preserve the original harm. In a survey (

N = 21

), most participants report that scores are easy to understand and that the dashboard encouraged them to view model evaluations under different settings

Abstract

Responsible AI

Explore similar work

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability

Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring