cs.SEMar 13, 2026

Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications

Authors: Alexandre Cristovão Maiorano

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM applications are AI systems whose nondeterministic outputs and evolving model behavior make traditional testing insufficient for release governance. We present an automated self-testing framework that introduces quality gates with evidence-based release decisions (PROMOTE/HOLD/ROLLBACK) across five empirically grounded dimensions: task success rate, research context preservation, P95 latency, safety pass rate, and evidence coverage. We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system with specific marketing capabilities in active development, covering 38 evaluation runs across 20+ internal releases. The gate identified two ROLLBACK-grade builds in early runs and supported stable quality evolution over a four-week staging lifecycle while exercising persona-grounded, multi-turn, adversarial, and evidence-required scenarios. Statistical analysis (Mann-Kendall trends, Spearman correlations, bootstrap confidence intervals), gate ablation, and overhead scaling indicate that evidence coverage is the primary severe-regression discriminator and that runtime scales predictably with suite size. A human calibration study (n=60 stratified cases, two independent evaluators, LLM-as-judge cross-validation) reveals complementary multi-modal coverage: LLM-judge disagreements with the system gate (kappa=0.13) are attributable to structural failure modes - latency violations and routing errors - invisible in response text alone, while the judge independently surfaces content quality failures missed by structural checks, consistent with a multi-dimensional gate design. The framework, supplementary pseudocode, and calibration artifacts are provided to support AI-system quality assurance and independent replication.

Figures & tables

AspectOursSTELLARMA-Comm.
Primary goalRelease gatingFailure discoveryBug detection
Quality dims.5D gateLinguistic feats.Persona diversity
Release decisionP/H/RFailure setConsensus vote
Evidence cov.Formal dim.Not addressedNot addressed
Safety as dim.✓Partial (SafeQA)Adversarial persona
CI/CDNativeOfflineOffline
Table 1: Qualitative comparison with concurrent automated testing frameworks for LLM applications (STELLAR [ 27 ] ; MA-Committees [ 12 ] ).
Figure 1: High-level interaction between the static/dynamic Question Bank and the multi-agent LLM orchestrator during evaluation.
Figure 2: CI/CD integration pipeline. A merge to the main branch triggers automated build checkout, question bank loading, full test suite execution with OpenTelemetry trace collection, five-dimensional metric computation, and deterministic gate decision. HOLD and ROLLBACK events feed back into the question bank expansion loop, preventing test suite drift.
DimensionThresholdDirection
Task Success Rate≥80%\geq 80\%Higher
Research Context Preservation≥90%\geq 90\%Higher
P95 Latency<15,000<15{,}000 msLower
Safety Pass Rate≥95%\geq 95\%Higher
Evidence Coverage≥80%\geq 80\%Higher
Table 2: Quality gate dimensions and acceptance thresholds.
DimensionAutomated signalScoring ruleKnown limitation and mitigation
Task Success RatePer-test validation over route, semantic, and issue-free completion checks; consolidates task success and response utilitypassed-tests / total-testsRubric-design sensitivity; mitigated by issue taxonomy, question-bank snapshots, and human calibration (Sec. 6 )
Research Context PreservationMulti-turn research tests with conversation history and expected research routingcontext-preserved / research-tests-with-historySaturates at 100% in stable periods (observed throughout this study); mitigated by harder follow-up injection
P95 LatencyEnd-to-end wall-clock latency per test from evaluator trace95th percentile (ms)Inflated by suite growth and rate limits; mitigated by per-test runtime reporting
Safety Pass RateAdversarial/boundary behavior via guardrail routing and safety issue checkssafety-tests-passing / safety-testsMay miss novel attacks; mitigated by adversarial tier expansion and spot checks
Evidence CoverageRetrieval-required tests with explicit citation signal in response text or source IDsweb-tests-with-evidence / web-testsCitation heuristic can be gamed; mitigated by source-ID tracking and low-confidence audits
Table 3: Operationalization of the five quality dimensions used by the release gate. Task Success Rate consolidates the former separate Success and Helpful dimensions, which exhibited Spearman ρ=1.00\rho=1.00 across all 38 evaluation runs, indicating complete construct overlap.
Figure 3: Decision flowchart: PROMOTE, HOLD, or ROLLBACK based on five quality dimensions.
AspectU/IE2ESelf-Test
Deterministic logic✓✓–
API contracts✓✓–
UI flow validation–✓–
Hallucination detection––✓
Context preservation––✓
Safety alignment––✓
Table 4: Complementary coverage: traditional testing vs. automated self-testing.
RunTestsTask SuccessP95 (ms)ContextSafetyEvidenceDecision
159100.0%8487100%97.0%50%ROLLBACK
259100.0%12316100%97.0%50%ROLLBACK
35998.3%7970100%97.0%100%PROMOTE
413100.0%11522100%100.0%100%PROMOTE
559100.0%9401100%97.0%100%PROMOTE
613100.0%12397100%100.0%100%PROMOTE
Table 5: Complete evaluation trace results across 38 runs with 5 quality dimensions (Task Success Rate consolidates the legacy Success/Helpful overlap).
ScenarioPromoteHoldRollbackSev.
Full 5D Gate36020
No Evidence Coverage38002
No Safety Dimension36020
Task Success + Latency Only38002
Traditional CI only (no behavioral gate)38002
Table 6: Gate ablation analysis across 38 runs with the consolidated 5D gate. Severe-promoted counts indicate missed critical failures (lower is better).
 TaskSRP95SafetyEvid
TaskSR1.00-0.47 ∗0.160.02
P95-0.47 ∗1.000.03-0.27
Safety0.160.031.00-0.20
Evid0.02-0.27-0.201.00
Table 7: Spearman rank correlation matrix between variable quality dimensions (RC excluded as constant). ∗ p<0.05p<0.05 (two-sided, n=38n=38 ); coefficients without a marker are not statistically significant.
MetricMean95% CI
Task Success Rate (%)97.9[97.2, 98.6]
P95 Latency (ms)11542.3[10969.7, 12083.1]
Research Context Preservation (%)100.0[100.0, 100.0]
Safety Pass Rate (%)97.1[96.8, 97.4]
Evidence Coverage (%)96.4[92.3, 99.5]
Table 8: Bootstrap 95% confidence intervals for metric means (B=10,000).
MetricTrendτ\taup-valueSig.
Task Success Rate (%)decreasing-0.3200.0038Yes
P95 Latency (ms)increasing0.3740.0010Yes
Research Context Preservation (%)no trend0.0001.0000No
Safety Pass Rate (%)no trend-0.1320.2037No
Evidence Coverage (%)no trend-0.0240.7306No
Table 9: Mann-Kendall trend test for monotonic trends ( α=0.05\alpha=0.05 ).
Figure 4: Success rate across 38 evaluation runs. Green markers indicate PROMOTE decisions; red markers indicate ROLLBACK. The dashed line marks the 80% acceptance threshold.
Figure 5: Distribution of P95 latency by suite phase labels (D13, C59, C86-88, C106, C133). The increasing trend ( τ=0.374\tau=0.374 ) correlates with growing test-suite complexity.
Figure 6: Joint evolution of the five gate dimensions across 38 evaluation runs. Task success trend reflects increasing test-suite difficulty; evidence coverage isolates severe regressions; latency growth tracks suite expansion while remaining under threshold.
MetricMeanMed.SDMinMaxIQR
Task Success97.998.42.291.5100.03.6
P95 Latency11542.311715.51769.37970.014631.01777.2
Research Ctx. Pres.100.0100.00.0100.0100.00.0
Safety Pass97.197.01.096.0100.00.0
Evidence Coverage96.4100.011.650.0100.00.0
Table 10: Descriptive statistics across evaluation runs.
SuiteRunsMed. sMed./Test msP95 ms
13261.14702.711960
501191.93837.68915
599241.14086.79850
833548.56608.514331
869409.64763.211393
884391.04443.111604
Table 11: Operational overhead by suite size. Total tests vs run duration: Pearson r=0.92r=0.92 ( p=0.000p=0.000 ), slope +5996+5996 ms/test.
ComparisonNAgree.κ\kappa
Human Evaluator 1 vs 26083.3%0.359
LLM Judge vs System Gate6063.3%0.132
Human 1 vs System Gate6066.7%0.167
Human 2 vs System Gate6063.3%0.000
LLM Judge vs Human 16080.0%0.444
LLM Judge vs Human 26073.3%0.149
Table 12: Evaluator alignment study: n=60n=60 stratified test cases (20 system-failed, 40 system-passed), evaluated blind by two independent human evaluators and an LLM-as-judge using a judge model from a different capability tier within the same model family (partial mitigation of shared-bias risk; same-family bias cannot be fully excluded) [ 31 ] . Human evaluator columns completed. κ\kappa interpretation follows Landis & Koch [ 13 ] ; see Section 6 for disaggregated failure-mode analysis.
More in this topic

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

Explore similar work

CardsList
Mar 28, 2026

LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications

We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow. The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract, then aggregates workflow success, policy compliance, groundedness, retrieval hit rate, cost, and p95 latency into scenario-weighted readiness scores with Pareto frontiers. We evaluate the harness on ticket-routing workflows and BEIR grounding tasks (SciFact and FiQA) with full Azure matrix coverage (162/162 valid cells across datasets, scenarios, retrieval depths, seeds, and models). Results show that readiness is not a single metric: on FiQA under sla-first at k=5, gpt-4.1-mini leads in readiness and faithfulness, while gpt-5.2 pays a substantial latency cost; on SciFact, models are closer in quality but still separable operationally. Ticket-routing regression gates consistently reject unsafe prompt variants, demonstrating that the harness can block risky releases instead of merely reporting offline scores. The result is a reproducible, operationally grounded framework for deciding whether an LLM or RAG system is ready to ship.

Alexandre Cristovão Maiorano

Lumytics

LLM EvaluationRAG EvaluationLLM Safety Evaluation
AskPDF
Jun 1, 2026

Acceptance-Test-Driven Evaluation Protocols for Business-Centric LLM Systems

Large language model (LLM) applications are increasingly expected to satisfy deterministic institutional requirements while relying on probabilistic generative components. This mismatch makes ordinary post-hoc benchmarking insufficient for systems that must be safe, reliable, auditable, and economically useful. This paper contributes an evaluation-protocol extension for operational LLM systems grounded in acceptance-test-driven development, safety engineering, and business-centric validation. The extension translates stakeholder goals into executable behavioral contracts, release gates, monitoring signals, and evidence artifacts before prompt, model, retrieval, or agent changes are accepted. It adapts the red-green-refactor discipline of test-driven development to a red-train-green lifecycle: first define failing acceptance tests for desired behavior, then improve the LLM system through prompt changes, retrieval design, fine-tuning, guardrails, or data augmentation, and finally release only when multidimensional gates are satisfied. The contribution is a governance-oriented metric stack, reference architecture, and empirical protocol for comparing acceptance-test-driven LLM development against prompt-first and benchmark-after workflows.

Eric Liang

Oracle

LLM EvaluationLanguage Model Safety Evaluation
AskPDF
Oct 1, 2026

AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation

Enterprises adopting retrieval-augmented generation (RAG) face a recurring operational decision: promote, revise, or block a system version. The evidence is incomplete and the metrics come from fallible LLM judges. We report on AGO AI Quality Gate (AGO), an evidence-first quality-gate framework deployed in industrial RAG assessment engagements. AGO integrates four key components: a four-state decision model that treats missing data and judge errors as explicit outcomes; layered scoring combining deterministic checks, local guardrails, and structured LLM evaluation; a stratified beta-binomial gate that quantifies regression risk probabilistically; and a mandatory meta-evaluation protocol to validate the LLM judge before it influences decisions. Since engagement data is proprietary, we evaluate the judge layer on RAGBench, a public benchmark of 100k annotated RAG traces across 12 datasets. On identical stratified test samples (N=1200 per judge), a low-cost judge (gpt-4.1-nano) detects non-adherent answers barely above chance (AUROC 0.603 [0.570, 0.634]), despite producing flawless protocol output, while gpt-4o reaches 0.783 [0.756, 0.807] -- yet its per-domain performance still ranges from 0.62 to 0.88. A fixed-seed gate study spanning regression, no change, and improvement quantifies unsafe promotion, false-alarm cost, and improvement throughput. Under regression, the decision-grade profile reduces unsafe promotion to 22.2%-35.1%, against 29.3%-41.8% for a naive gate. These results support the design choices that judge quality must be measured per engagement and that point estimates alone are not a release decision.

Giulio Zeloni, Enrico Lo Conte, Salvatore Rionero +3

Protom Group S.p.A., Napoli, Italy

LLM-as-a-JudgeRAG Evaluation
AskPDF