AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation

Giulio Zeloni; Enrico Lo Conte; Salvatore Rionero; Giuseppe Santoro; Alessandro Rastelli; Fabio Sorrentino

cs.CLOct 1, 2026

AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation

Authors: Giulio Zeloni, Enrico Lo Conte, Salvatore Rionero, Giuseppe Santoro, Alessandro Rastelli, Fabio Sorrentino

Organizations: Protom Group S.p.A., Napoli, Italy

Abstract

Enterprises adopting retrieval-augmented generation (RAG) face a recurring operational decision: promote, revise, or block a system version. The evidence is incomplete and the metrics come from fallible LLM judges. We report on AGO AI Quality Gate (AGO), an evidence-first quality-gate framework deployed in industrial RAG assessment engagements. AGO integrates four key components: a four-state decision model that treats missing data and judge errors as explicit outcomes; layered scoring combining deterministic checks, local guardrails, and structured LLM evaluation; a stratified beta-binomial gate that quantifies regression risk probabilistically; and a mandatory meta-evaluation protocol to validate the LLM judge before it influences decisions. Since engagement data is proprietary, we evaluate the judge layer on RAGBench, a public benchmark of 100k annotated RAG traces across 12 datasets. On identical stratified test samples (N=1200 per judge), a low-cost judge (gpt-4.1-nano) detects non-adherent answers barely above chance (AUROC 0.603 [0.570, 0.634]), despite producing flawless protocol output, while gpt-4o reaches 0.783 [0.756, 0.807] -- yet its per-domain performance still ranges from 0.62 to 0.88. A fixed-seed gate study spanning regression, no change, and improvement quantifies unsafe promotion, false-alarm cost, and improvement throughput. Under regression, the decision-grade profile reduces unsafe promotion to 22.2%-35.1%, against 29.3%-41.8% for a naive gate. These results support the design choices that judge quality must be measured per engagement and that point estimates alone are not a release decision.

Figures & tables

Judge	AUROC adh.	MCC @ $0.8$	Spearman rel.	Cost
gpt-4.1-nano	$0.672$ [0.609, 0.732]	$0.284$ [0.176, 0.391]	$0.279$ [0.163, 0.386]	$0.06
gpt-4o	$0.787$ [0.731, 0.837]	$0.435$ [0.329, 0.539]	$0.217$ [0.103, 0.330]	$1.36

Table 1: Judge selection on the validation split (3 datasets,

N{=}300

per judge;

95\%

bootstrap CIs). MCC is computed at the deployed threshold

\theta{=}0.8

Judge	AUROC adh.	MCC @ $0.8$	Spearman rel.	Cost
gpt-4.1-nano	$0.603$ [0.570, 0.634]	$0.170$ [0.110, 0.221]	$0.366$ [0.313, 0.413]	$0.39
gpt-4o	$0.783$ [0.756, 0.807]	$0.433$ [0.386, 0.484]	$0.298$ [0.243, 0.351]	$9.42

Table 2: Test-split results over all 12 RAGBench datasets (

N{=}1200

per judge, identical instances;

95\%

bootstrap CIs; overall micro aggregation).

Figure 1: Per-dataset adherence AUROC of the selected judge (gpt-4o, filled circles; stratified test samples,

N{=}100

per dataset) alongside evaluator results published by the benchmark authors on full test splits (their Table 3 [ 7 ] ). Dashed line: chance. Datasets sorted by gpt-4o AUROC; the comparison across protocols is indicative (Sect. 5.1 ). Per-dataset MCC and Spearman values appear in Table 3 .

Dataset	AUROC (adherence)	MCC @ $0.8$	Spearman (relevance)
covidqa	$0.769$	$0.388$	$0.009$
cuad	$0.620$	$0.199$	$0.201$
delucionqa	$0.800$	$0.166$	$0.236$
emanual	$0.630$	$0.040$	$0.324$
expertqa	$0.883$	$0.572$	$0.400$
finqa	$0.787$	$0.446$	$0.233$

Table 3: Per-dataset test results for the selected judge (gpt-4o,

N{=}100

per dataset).

Regressed
$N$	Naive promote	AGO promote	AGO review	AGO block	Avg. $\rho$
20	41.8	33.0	47.3	19.7	0.532
30	37.4	35.1	41.3	23.6	0.538
50	38.1	34.5	40.8	24.7	0.546
100	38.4	29.9	42.5	27.6	0.571
200	29.3	22.2	41.6	36.2	0.638

Table 4: Gate operating characteristics for the decision_grade profile (rates in %).

Explore similar work

Sep 30, 2026cs.CR

RAGScope: A Leakage-Controlled, Cost-Aware Evidence-Gating Protocol for RAG Hallucination Triage

Retrieval-augmented generation (RAG) systems need inexpensive ways to route generated answers: accept low-risk outputs, review uncertain ones, and reserve strong verifiers for the expensive tail. We present RAGScope, a leakage-controlled protocol for evaluating local evidence gates that use only the task input, retrieved context, and answer text. The protocol combines context-grouped splits, fold-scoped preprocessing, group bootstrap intervals, deployment operating points, end-to-end runtime, and explicit source-shift stress tests. On three RAGTruth tasks, the enhanced gate RAGScope-E reaches 0.798 AUROC and 0.660 average precision (AP) in pooled grouped cross-validation. Its pooled AP exceeds ROUGE-L by 0.034 with a 95% context-group interval of [0.002, 0.064], although the AUROC gain is not significant and ROUGE-L remains stronger on data-to-text. At a top-10% review budget, RAGScope-E attains 0.748 precision; accepting the lowest-risk 50% yields 0.141 residual unfaithfulness. RAGScope-E runs in 6.22 ms/example on CPU, versus 145.75 and 223.07 ms/example for the tested DeBERTa-NLI and HHEM settings. A 14,900-example HaluBench stress test exposes the deployment boundary: an in-domain calibrated gate reaches 0.879 AUROC, but leave-source-out calibration averages only 0.466. Target-only calibration recovers to 0.675 AUROC with 100 labels per source and 0.685 with 200. Cheap evidence gates are therefore useful routing components, but learned calibration must be validated and adapted within the target domain.

Zeming Liu, Qibai Chen, Jingtao Zhang +1

Brown University, United States · Independent Researcher, United States · Georgia Institute of Technology, United States

Leakage Cost-Aware Cross Validation

Ask PDF

Sep 28, 2026cs.AI

RAGWarrant: Evidence-Preserving Governance for RAG Policy Promotion Under Quality, Cost, Latency, and Risk Constraints

Retrieval-augmented generation systems are extensively instrumented with metrics, benchmarks, traces, and automated judges, but these tools do not decide whether a proposed policy change is safe to release. We present RAGWarrant, an open-source promotion-control framework that treats deployment as a constrained evidence decision rather than a leaderboard choice. RAGWarrant normalizes evaluator outputs and operational telemetry, applies predeclared quality and hard-risk gates, assigns evidence-class claim ceilings, preserves negative outcomes, and emits auditable PROMOTE, BLOCK, REJECT, or INCONCLUSIVE decisions. We evaluate the framework across T2-RAGBench, MultiHop-RAG, CRAG, HotpotQA, synthetic reproduction, and bounded local generative experiments. On HotpotQA, operational savings were blocked because answer quality fell beyond the declared margin. A bounded CRAG study selected a lower-cost quality-tied policy, but related generative gains were unstable and a held-out guardrail failed closed. We claim an auditable promotion-control abstraction, not optimizer superiority, human validation, or production readiness. The tagged artifact reproduces from a fresh clone, runs as a hardened Docker job, accepts external evaluator exports, and verifies artifact integrity.

Richard Krueger, Lucas Krause, Zach Pocquette

Independent open-source research project

Agentic Retrieval-Augmented Generation Systems Governance

Ask PDF

Jul 31, 2026cs.AI

Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests. We propose Q-CARE, a query-agnostic and fully reference-free framework that enables fine-grained assessment by decomposing queries into sub-queries and answers into atomic claims. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework. Code and data are publicly available at https://github.com/DISL-Lab/Q-CaRE-COLM-26.

Jeonghwan Choi, Taewon Yun, Minjeong Ban +3

1Korea Advanced Institute of Science and Technology (KAIST) · 2Cluvion

Question-Answering Benchmarks

Ask PDF