MERCI Cards: An LLM Evaluation and Deployment Framework for High-Stakes Domains

Aparna Komarla; Annalisa Szymanski

cs.AIOct 3, 2026

MERCI Cards: An LLM Evaluation and Deployment Framework for High-Stakes Domains

Authors: Aparna Komarla, Annalisa Szymanski

Organizations: University of Notre Dame Notre Dame, Indiana, USA

Abstract

As LLMs are increasingly deployed in high-stakes professional workflows, engineers and researchers require principled protocols to systematically track, monitor, and improve model performance across deployment cycles. We present a mathematical framework for iterative LLM evaluation and deployment, and demonstrate its application to AI systems used in criminal justice. Our framework formalizes LLM integration in high-stakes, high-risk, and resource-constrained domains across model selection, rubric design, evaluations and deployment via a weighted multi-objective optimization. We demonstrate that MERCI Cards can guide improvements of the system across deployment iterations, direct developer attention toward under-performing areas, and focus user attention on validation and error-correction in the LLM's outputs.

Figures & tables

Symbol	Meaning
$\mathcal{T}$ , $\mathit{TR}_{n}$ , $N$	Task; $n$ -th task result; number of results
$\tau$	Iteration index (suppressed when unambiguous)
$E$	Domain expert input
$G$ , $P$ , $C$ , $R$ , $W$	Configuration: model, prompt, context, rubric, weights
$D$ , $d$	Number of rubric dimensions; dimension index
$J$ , $\ell$	LLM judge; judge runs per task result

Table 1. Notation.

Decision	Symbol	Step
Scoring granularity per dimension	—	Setup
Aggregation of judge and human scores	$\mathrm{agg}$ , $\lambda$	Setup
Rubric dimensions	$R$	A
Dimension weights	$W$	A
Whether context is supplied, and whether it is withheld during selection	$C$	C
Candidate model set	$\mathcal{M}$	D

Table 2. Choices the framework leaves to the deploying team. The framework requires that each be recorded on the MERCI card with its rationale; it does not prescribe the value.

Panel of Figure 1 — Figure 1. Framework pipeline. The expert configures the task, rubric, and generation model, evaluation is set up, the task is run and scored, and a release decision is made. The quality vector $Q$ returns to the configuration stage.

Section	Field	Kind
Identity	System name, domain, iteration $\tau$	record
	Card version, date issued, prior card reference ( $\tau-1$ )	record
	Deploying team and accountable contact	record
Task	$\mathcal{T}$ : task description and what a task result is (e.g., answer, report, summary, drafted document)	record
	$N$ : number of inputs the task was run on; how they were selected	record
	Scoring granularity per dimension: continuous, discrete, or categorical, with the mapping onto $[0,1]$	choice

Table 3. MERCI Card specification. One card per iteration

\tau

. choice fields are decisions the deploying team makes: record the value and the reasoning. derived fields are computed from the run. record fields are factual. optional fields improve the card without being necessary to interpret the iteration.

Dimension	Tests whether the report
OR CI score	contextualizes odds ratios with CI width and whether the interval spans 1
RR CI score	does the same for relative risk
OR Smp score	contextualizes odds ratios with sample size
RR Smp score	does the same for relative risk
$\chi^{2}$ Smp score	does the same for chi-square
$p$ score	contextualizes $p$ -values against the significance threshold

Table 4. Examples of evaluation rubric dimensions

Generation model			Judge
$\mathcal{F}(G)$	native	$\rho$	$\mathcal{F}(J)$	$\rho$	$J_{d}$	$\hat{S}_{m}$	$t$ (s)	$c$ ($)
Opus	none	0	GPT	3	0.92	11.60	40	0.06
Opus	adapt / med	2	GPT	3	0.92	10.92	53	0.08
Opus	adapt / high	3	GPT	3	0.92	11.94	55	0.08
Opus	adapt / max	4	GPT	3	0.91	10.31	96	0.16
Sonnet	adapt / high	3	GPT	3	0.90	9.91	180	0.21

Table 5. Model selection sweep:

|\mathcal{M}|=11

configurations over

N=18

reports and

\ell=15

judge runs. Native settings appear with the normalized reasoning level

\rho

the team declared for them.

J_{d}

is the mean judge score across the nine dimensions;

\hat{S}_{m}

is the composite including cost and latency (

w_{\text{Cmp}}=5

w_{d}=1

otherwise,

w_{c}=w_{t}=1

). Sorted by

J_{d}

. Highlighted row is the selected

G

	Judge		Human		$\Delta$ from $\tau=0$
Dimension	$J_{d}$	sd	$H_{d}$	sd	$\Delta J_{d}$	$\Delta H_{d}$
OR CI score	0.94	0.03	0.89	0.16	$+$ 0.15	$+$ 0.09
OR Smp score	1.00	0.00	0.80	0.13	$+$ 0.51	$+$ 0.19
RR CI score	0.86	0.04	0.87	0.18	$+$ 0.10	$+$ 0.07
RR Smp score	1.00	0.00	0.80	0.13	$+$ 0.51	$+$ 0.19
Cmp score	0.74	0.01	0.76	0.35	$+$ 0.05	$-$ 0.02

Table 6. Scores per rubric dimension at

\tau=1

(

N=30

\ell=15

V=3

), with the change from the prior iteration. Judge sd is the mean standard deviation across the

\ell

runs per report; human sd is the inter-reviewer standard deviation.

Section	Field	Entry
Identity	Prior card	$\tau=0$
Identity	$N$ inputs	30 reports; 18 in the selection sweep
Task	Granularity	any value in the range $[0,1]$
Task	$\mathrm{agg}$	$Q_{d}=\tfrac{1}{2}(J_{d}+H_{d})$ for deployment evaluation, i.e. both humans and LLM judges are used; $Q_{d}=J_{d}$ for model selection, i.e. no humans are used
Rubric	$R$	$R(E,Q^{(0)})$ , $D=9$
	$W$	$w_{\text{Cmp}}=5$ , $w_{d}=1$ otherwise, $w_{c}=w_{t}=1$

Table 7. MERCI Card for

\tau=1

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

LLJ Cards: Best practices for the Use of LLMs as Judges
Sep 21, 2026Khaoula Chehbouni, Melina Medjdoub, Florian Carichon +2LLM Evaluation LLM-as-a-Judge
McGill University · Mila - Quebec AI Institute · Concordia University
Acceptance-Test-Driven Evaluation Protocols for Business-Centric LLM Systems
Jun 1, 2026Eric LiangLLM Evaluation Language Model Safety Evaluation
Oracle
Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation
Oct 15, 2026Jianchang Su, Dawei Xiang, Yifan Zhang +1LLM Security LLM Agent Evaluation
University of Connecticut

Sep 21, 2026J/K move · Enter open · S save

LLJ Cards: Best practices for the Use of LLMs as Judges

Khaoula Chehbouni, Melina Medjdoub, Florian Carichon, Golnoosh Farnadi+1

McGill University · Mila - Quebec AI Institute · Concordia University

LLM Evaluation LLM-as-a-Judge LLM Reliability Automated Evaluation

Open paper AskPDF arXiv

In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these systems have been widely adopted by researchers and practitioners across a broad range of measurement tasks, driven by their strong performance, scalability, and cost-effectiveness relative to human judgment. However, a growing body of work has shown that the use of LLJs raise concerns about their validity and reliability as evaluators. Existing efforts to address these challenges have largely focused on developing bias-mitigation techniques and refining prompting strategies. While these approaches represent an important step forward, they primarily offer technical fixes and leave a more fundamental challenge unaddressed: the lack of standardized, transparent, and reproducible evaluation practices. In this paper, we introduce LLJ Cards, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations. While LLJs offer a promising path toward scalable evaluation, their effective use requires grounding in rigorous evaluation principles to ensure validity, reliability, and reproducibility. LLJ Cards addresses this need by providing a structured framework for applying these principles in the design and reporting of automated evaluations.

Abstract

LLM Evaluation

Explore similar work

LLJ Cards: Best practices for the Use of LLMs as Judges

Acceptance-Test-Driven Evaluation Protocols for Business-Centric LLM Systems

Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation