Performance regression testing is essential in large-scale continuous-integration (CI) systems, yet executing full performance suites for every commit is prohibitively expensive. Prior work on performance regression prediction and batch testing has shown independent benefits, but each faces practical limitations: predictive models are rarely integrated into CI decision-making, and conventional batching strategies ignore commit-level heterogeneity. We unify these strands by introducing a risk-aware framework that integrates machine-learned commit risk with adaptive batching. Using Mozilla Firefox as a case study, we construct a production-derived dataset of human-confirmed regressions aligned chronologically with Autoland, and fine-tune ModernBERT, CodeBERT, and LLaMA-3.1 variants to estimate commit-level performance regression risk, achieving up to 0.694 ROC-AUC with CodeBERT. The risk scores drive a family of risk-aware batching strategies, including Risk-Aged Priority Batching and Risk-Adaptive Stream Batching, evaluated through realistic CI simulations. Across thousands of historical Firefox commits, our best overall configuration, Risk-Aged Priority Batching with linear aggregation (RAPB-la), yields a Pareto improvement over Mozilla's production-inspired baseline. RAPB-la reduces total test executions by 32.4%, decreases mean feedback time by 3.8%, maintains mean time-to-culprit at approximately the baseline level, reduces maximum time-to-culprit by 26.2%, and corresponds to an estimated annual infrastructure cost savings of approximately $491K under our cost model. These results demonstrate that risk-aware batch testing can reduce CI resource consumption while improving diagnostic timeliness. To support reproducibility and future research, we release a complete replication package containing all datasets, fine-tuning pipelines, and implementations of our batching algorithms.
Figures & tables
Fig. 1: Example of a structured diff used as input to the ModernBERT model, reproduced from the DRS-OSS framework [ 13 ] . Custom tokens (e.g., <FILE> , <ADDED> , <REMOVED> ) encode the hierarchical structure of the change while reducing raw diff verbosity.
Split
Clean
Regressor
Positive %
Train
7328
71
0.96%
Evaluation
1126
12
1.05%
Test
2832
15
0.53%
TABLE I: Risk model dataset statistics: class counts and positive ratios in each chronological split.
Model
ROC-AUC
Recall@10%
Recall@30%
LLaMA-3.1 8B
0.681
0.231
0.615
ModernBERT
0.683
0.23
0.538
CodeBERT
0.694
0.077
0.692
TABLE II: Test-set performance of fine-tuned regression-risk predictors.
Strategy
Total Tests
Mean Feedback Time (hr)
Mean TTC (hr)
Max TTC (hr)
TWSB
387,422
6.6
6.0
12.4
ET
12,636,342
4,999.3
3,118.0
6,749.2
TABLE III: Comparison between Exhaustive Testing (ET) and Mozilla’s batching strategy (TWSB).
Batching Strategy
Total Tests
Mean Feedback Time (hr)
Mean TTC (hr)
Max TTC (hr)
Infra Usage Time (hr)
Cost ($K)
Cost savings (%)
TWSB (Baseline)
387,422
6.6
6.0
12.4
112,108
390
-
TWB
317,363
6.5
6.7
8.7
93,052
324
16.9
TWB-s
259,426
7.0
7.5
14.1
76,145
265
32.1
FSB
199,156
9.9
7.9
13.5
58,370
203
47.9
FSB-s
168,796
9.9
7.9
13.5
49,614
173
55.6
RASB
275,707
7.9
6.0
10.5
80,808
281
27.9
TABLE IV: Results comparison for batching strategies on the test split, together with estimated infrastructure cost over the 3-month simulation window. Estimated infra usage time is reported for the bare-metal test-runner infrastructure. Cost is computed using an effective infrastructure-wide rate of $3.48 per hour of infrastructure usage. “Cost savings” denotes the percentage reduction in estimated cost relative to TWSB.
When machine learning classifiers are retrained, inputs correctly classified by the previous model version may be misclassified by the updated version, creating regression faults that are costly to detect because verifying predictions against ground truth may require human annotation, expert review, or expensive simulation rather than inexpensive model inference. Test input prioritization addresses this problem by ranking inputs so that a limited verification budget reveals as many regression faults as possible. Existing approaches rely predominantly on single-model confidence scores and do not exploit how predictions, decision boundaries, and local neighborhoods change between model versions. We propose RiskBlend, a classifier-agnostic prioritization framework that combines four complementary risk signals: historical failure patterns, prediction shift, decision-boundary shift, and neighborhood change. These signals are combined using validation-learned APFD-squared weighting. Across four datasets, five classifiers, four regression-update scenarios, and 15 random seeds, totaling 1,200 experimental configurations, RiskBlend achieves the highest average APFD in all 80 dataset-classifier-scenario combinations, with improvements of up to 0.32 APFD over the strongest baseline. Confidence-based methods remain competitive primarily for linear classifiers on sparse categorical features, which we attribute to feature-space geometry. The results show that cross-version behavioral signals provide important complementary information for prioritizing regression faults in machine learning systems.
Madhusudan Srinivasan, Namith Nishal Raphae
Department of Computer Science East Carolina University Greenville, USA
Autonomous AI agents are deployed at unprecedented scale, yet no principled methodology exists for verifying that an agent has not regressed after changes to its prompts, tools, models, or orchestration logic. We present AgentAssay, the first token-efficient framework for regression testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining rigorous statistical guarantees. Our contributions include: (1) stochastic three-valued verdicts (PASS/FAIL/INCONCLUSIVE) grounded in hypothesis testing; (2) five-dimensional agent coverage metrics; (3) agent-specific mutation testing operators; (4) metamorphic relations for agent workflows; (5) CI/CD deployment gates as statistical decision procedures; (6) behavioral fingerprinting that maps execution traces to compact vectors, enabling multivariate regression detection; (7) adaptive budget optimization calibrating trial counts to behavioral variance; and (8) trace-first offline analysis enabling zero-cost testing on production traces. Experiments across 5 models (GPT-5.2, Claude Sonnet 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4), 3 scenarios, and 7,605 trials demonstrate that behavioral fingerprinting achieves 86% detection power where binary testing has 0%, SPRT reduces trials by 78%, and the full pipeline achieves 100% cost savings through trace-first analysis. Implementation: 20,000+ lines of Python, 751 tests, 10 framework adapters.
Test-time compute has emerged as an effective paradigm for improving large language model capability at inference time. Existing allocation strategies primarily prioritize tasks according to difficulty, uncertainty, or expected performance gain, implicitly treating prediction errors as equally costly. This assumption is often misaligned with real deployment, where failures can differ substantially in their downstream tasks. To address this limitation, this paper introduces consequence-aware test-time compute allocation by formulating a cost-weighted scheduling problem where the priority of a task is its failure consequence with the marginal gain of additional compute. In practice, however, marginal gain is difficult to predict before execution, so we propose a deployable scheduler that uses consequence as the routing signal. The scheduler predicts task consequence from pre-solution inputs and allocates the available premium compute to the corresponding top-ranked tasks. Experiments on the SWE bench Lite show that consequence provides information beyond task difficulty and can be predicted before solving. Under a fixed compute budget, consequence-aware routing achieves the best high-consequence task success, while overall accuracy remains competitive. A controlled within-model experiment further confirms the same advantage when only inference attempts are reallocated.
Liang He, Jingbo Wen, Haoyu Wang +4
Shanghai Institute of Optics and Fine Mechanics · The University of Sydney