LLM applications are AI systems whose nondeterministic outputs and evolving model behavior make traditional testing insufficient for release governance. We present an automated self-testing framework that introduces quality gates with evidence-based release decisions (PROMOTE/HOLD/ROLLBACK) across five empirically grounded dimensions: task success rate, research context preservation, P95 latency, safety pass rate, and evidence coverage. We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system with specific marketing capabilities in active development, covering 38 evaluation runs across 20+ internal releases. The gate identified two ROLLBACK-grade builds in early runs and supported stable quality evolution over a four-week staging lifecycle while exercising persona-grounded, multi-turn, adversarial, and evidence-required scenarios. Statistical analysis (Mann-Kendall trends, Spearman correlations, bootstrap confidence intervals), gate ablation, and overhead scaling indicate that evidence coverage is the primary severe-regression discriminator and that runtime scales predictably with suite size. A human calibration study (n=60 stratified cases, two independent evaluators, LLM-as-judge cross-validation) reveals complementary multi-modal coverage: LLM-judge disagreements with the system gate (kappa=0.13) are attributable to structural failure modes - latency violations and routing errors - invisible in response text alone, while the judge independently surfaces content quality failures missed by structural checks, consistent with a multi-dimensional gate design. The framework, supplementary pseudocode, and calibration artifacts are provided to support AI-system quality assurance and independent replication.
Figure 1: High-level interaction between the static/dynamic Question Bank and the multi-agent LLM orchestrator during evaluation.
Figure 2: CI/CD integration pipeline. A merge to the main branch triggers automated build checkout, question bank loading, full test suite execution with OpenTelemetry trace collection, five-dimensional metric computation, and deterministic gate decision. HOLD and ROLLBACK events feed back into the question bank expansion loop, preventing test suite drift.
Dimension
Threshold
Direction
Task Success Rate
≥80%
Higher
Research Context Preservation
≥90%
Higher
P95 Latency
<15,000 ms
Lower
Safety Pass Rate
≥95%
Higher
Evidence Coverage
≥80%
Higher
Table 2: Quality gate dimensions and acceptance thresholds.
Dimension
Automated signal
Scoring rule
Known limitation and mitigation
Task Success Rate
Per-test validation over route, semantic, and issue-free completion checks; consolidates task success and response utility
passed-tests / total-tests
Rubric-design sensitivity; mitigated by issue taxonomy, question-bank snapshots, and human calibration (Sec. 6 )
Research Context Preservation
Multi-turn research tests with conversation history and expected research routing
context-preserved / research-tests-with-history
Saturates at 100% in stable periods (observed throughout this study); mitigated by harder follow-up injection
P95 Latency
End-to-end wall-clock latency per test from evaluator trace
95th percentile (ms)
Inflated by suite growth and rate limits; mitigated by per-test runtime reporting
Safety Pass Rate
Adversarial/boundary behavior via guardrail routing and safety issue checks
safety-tests-passing / safety-tests
May miss novel attacks; mitigated by adversarial tier expansion and spot checks
Evidence Coverage
Retrieval-required tests with explicit citation signal in response text or source IDs
web-tests-with-evidence / web-tests
Citation heuristic can be gamed; mitigated by source-ID tracking and low-confidence audits
Table 3: Operationalization of the five quality dimensions used by the release gate. Task Success Rate consolidates the former separate Success and Helpful dimensions, which exhibited Spearman ρ=1.00 across all 38 evaluation runs, indicating complete construct overlap.
Figure 3: Decision flowchart: PROMOTE, HOLD, or ROLLBACK based on five quality dimensions.
Aspect
U/I
E2E
Self-Test
Deterministic logic
✓
✓
–
API contracts
✓
✓
–
UI flow validation
–
✓
–
Hallucination detection
–
–
✓
Context preservation
–
–
✓
Safety alignment
–
–
✓
Table 4: Complementary coverage: traditional testing vs. automated self-testing.
Run
Tests
Task Success
P95 (ms)
Context
Safety
Evidence
Decision
1
59
100.0%
8487
100%
97.0%
50%
ROLLBACK
2
59
100.0%
12316
100%
97.0%
50%
ROLLBACK
3
59
98.3%
7970
100%
97.0%
100%
PROMOTE
4
13
100.0%
11522
100%
100.0%
100%
PROMOTE
5
59
100.0%
9401
100%
97.0%
100%
PROMOTE
6
13
100.0%
12397
100%
100.0%
100%
PROMOTE
Table 5: Complete evaluation trace results across 38 runs with 5 quality dimensions (Task Success Rate consolidates the legacy Success/Helpful overlap).
Scenario
Promote
Hold
Rollback
Sev.
Full 5D Gate
36
0
2
0
No Evidence Coverage
38
0
0
2
No Safety Dimension
36
0
2
0
Task Success + Latency Only
38
0
0
2
Traditional CI only (no behavioral gate)
38
0
0
2
Table 6: Gate ablation analysis across 38 runs with the consolidated 5D gate. Severe-promoted counts indicate missed critical failures (lower is better).
TaskSR
P95
Safety
Evid
TaskSR
1.00
-0.47 ∗
0.16
0.02
P95
-0.47 ∗
1.00
0.03
-0.27
Safety
0.16
0.03
1.00
-0.20
Evid
0.02
-0.27
-0.20
1.00
Table 7: Spearman rank correlation matrix between variable quality dimensions (RC excluded as constant). ∗ p<0.05 (two-sided, n=38 ); coefficients without a marker are not statistically significant.
Metric
Mean
95% CI
Task Success Rate (%)
97.9
[97.2, 98.6]
P95 Latency (ms)
11542.3
[10969.7, 12083.1]
Research Context Preservation (%)
100.0
[100.0, 100.0]
Safety Pass Rate (%)
97.1
[96.8, 97.4]
Evidence Coverage (%)
96.4
[92.3, 99.5]
Table 8: Bootstrap 95% confidence intervals for metric means (B=10,000).
Metric
Trend
τ
p-value
Sig.
Task Success Rate (%)
decreasing
-0.320
0.0038
Yes
P95 Latency (ms)
increasing
0.374
0.0010
Yes
Research Context Preservation (%)
no trend
0.000
1.0000
No
Safety Pass Rate (%)
no trend
-0.132
0.2037
No
Evidence Coverage (%)
no trend
-0.024
0.7306
No
Table 9: Mann-Kendall trend test for monotonic trends ( α=0.05 ).
Figure 4: Success rate across 38 evaluation runs. Green markers indicate PROMOTE decisions; red markers indicate ROLLBACK. The dashed line marks the 80% acceptance threshold.
Figure 5: Distribution of P95 latency by suite phase labels (D13, C59, C86-88, C106, C133). The increasing trend ( τ=0.374 ) correlates with growing test-suite complexity.
Figure 6: Joint evolution of the five gate dimensions across 38 evaluation runs. Task success trend reflects increasing test-suite difficulty; evidence coverage isolates severe regressions; latency growth tracks suite expansion while remaining under threshold.
Metric
Mean
Med.
SD
Min
Max
IQR
Task Success
97.9
98.4
2.2
91.5
100.0
3.6
P95 Latency
11542.3
11715.5
1769.3
7970.0
14631.0
1777.2
Research Ctx. Pres.
100.0
100.0
0.0
100.0
100.0
0.0
Safety Pass
97.1
97.0
1.0
96.0
100.0
0.0
Evidence Coverage
96.4
100.0
11.6
50.0
100.0
0.0
Table 10: Descriptive statistics across evaluation runs.
Suite
Runs
Med. s
Med./Test ms
P95 ms
13
2
61.1
4702.7
11960
50
1
191.9
3837.6
8915
59
9
241.1
4086.7
9850
83
3
548.5
6608.5
14331
86
9
409.6
4763.2
11393
88
4
391.0
4443.1
11604
Table 11: Operational overhead by suite size. Total tests vs run duration: Pearson r=0.92 ( p=0.000 ), slope +5996 ms/test.
Comparison
N
Agree.
κ
Human Evaluator 1 vs 2
60
83.3%
0.359
LLM Judge vs System Gate
60
63.3%
0.132
Human 1 vs System Gate
60
66.7%
0.167
Human 2 vs System Gate
60
63.3%
0.000
LLM Judge vs Human 1
60
80.0%
0.444
LLM Judge vs Human 2
60
73.3%
0.149
Table 12: Evaluator alignment study: n=60 stratified test cases (20 system-failed, 40 system-passed), evaluated blind by two independent human evaluators and an LLM-as-judge using a judge model from a different capability tier within the same model family (partial mitigation of shared-bias risk; same-family bias cannot be fully excluded) [ 31 ] . Human evaluator columns completed. κ interpretation follows Landis & Koch [ 13 ] ; see Section 6 for disaggregated failure-mode analysis.