Test Case Prioritization: A Snowballing Literature Review and TCPFramework with Approach Combinators

Tomasz Chojnacki; Lech Madeyski

cs.SEFeb 26, 2026

Test Case Prioritization: A Snowballing Literature Review and TCPFramework with Approach Combinators

Authors: Tomasz Chojnacki, Lech Madeyski

Organizations: Wroclaw University of Science and Technology, Wroclaw, Poland

Abstract

Context: Test case prioritization (TCP) is a technique widely used by software development organizations to accelerate regression testing. Objectives: We aim to systematize existing TCP knowledge and to propose and empirically evaluate a new TCP approach. Methods: We conduct a snowballing review (SR) on TCP, implement a~~comprehensive platform for TCP research (TCPFramework), analyze existing evaluation metrics and propose two new ones (\rAPFDc{} and ATR), and develop a~~family of ensemble TCP methods called approach combinators. Results: The SR helped identify 324 studies related to TCP. The techniques proposed in our study were evaluated on the RTPTorrent dataset, consistently outperforming their base approaches across the majority of subject programs, and achieving performance comparable to the current state of the art for heuristical algorithms (in terms of \rAPFDc{}, NTR, and ATR), while using a distinct approach. Conclusions: The proposed methods can be used efficiently for TCP, reducing the time spent on regression testing by up to 2.7%. Approach combinators offer significant potential for improvements in future TCP research, due to their composability.

Figures & tables

Panel of Figure 1 — Figure 1: Workflow of the search process and flowchart of paper counts at each stage.

Review stage	Evaluated entries	Identified papers
	(duplicates included)	(new and unique)
Initial papers	—	4
Start set	440	∗ 3
Iteration #1: Backward	274	104
Iteration #1: Forward	222	60
Iteration #2: Backward	6 734	111

Table 1: The total number of new papers included in each search process stage.

Figure 3: Number of identified works in different publication years.

Year	Study reference	Years covered	Identified works
2012	Singh et al. (2012)	1997 – 2011	65
2013	Catal and Mishra (2013)	2001 – 2011	120
2017	de S. Campos Junior et al. (2017)	1999 – 2016	90
2018	Khatibsyarbini et al. (2018)	1999 – 2016	80
2020	de Castro-Cabrera et al. (2020)	2017 – 2019	320
2020	Prado Lima and Vergilio (2020)	2009 – 2019	35

Table 2: Overview of other surveys encountered during the SR.

Research decision	Related SR publications	How SR informed the decision
Dataset: RTPTorrent	( Mattis et al., 2020 )	SRRQ1 revealed dataset fragmentation (SIR, GSDTSR, Defects4j, Paint Control, BigFaultMatrix, etc.); RTPTorrent is the only dataset with a dedicated methodological paper addressing data quality issues.
Baseline: ROCKET	( Marijan et al., 2013 ) , ( Marijan, 2023 ) , ( Prado Lima and Vergilio, 2020 )	SRRQ2: ROCKET is the second most widely used TCP baseline after random ordering; performs comparably to or better than many ML-based methods.
Baseline: DBP	( Zhou et al., 2021 )	Identified through SR as a recent heuristic method meeting inclusion criteria for baseline selection.
Baseline: DFE	( Kim and Porter, 2002 ) , ( Mattis et al., 2020 )	Used as a non-trivial baseline in RTPTorrent; represents simple exponential-smoothing TCP approaches.
Approach combinators (RQ1)	( Huang et al., 2021 ) , ( Cao et al., 2022 ) , ( Mondal and Nasre, 2021 ) , ( Hassan et al., 2024 )	SR identified a gap: prior attempts at combining TCP methods were limited to ML-based approaches (requiring training) or simple voting schemes; no training-free combinators existed.
Metric: NTR	( Prado Lima and Vergilio, 2022 ) , ( Zhao et al., 2023 )	SR identified NTR as suitable for evaluating CI applicability; inspired the proposed ATR metric addressing its limitations.

Table 3: Mapping of SR findings to research decisions.

Figure 4: Taxonomies of TCP (on a basis of ( Yoo and Harman, 2012 ) ) and ML-based TCP (on a basis of ( Pan et al., 2021 ) ) approaches.

Measure	Computable	Normalized ∗	Awareness
Measure	Computable	Normalized ∗	Failure	Cost	Severity	Constraint
APFD ( Rothermel et al., 1999 )	failed cycles
$\text{APFD}_{C}$ ( Elbaum et al., 2001 )	failed cycles
$\text{APFD}_{C}$ (simple)	failed cycles
NAPFD ( Qu et al., 2007 )	failed cycles
$r\text{APFD}$ ( Zhao et al., 2023 )	failed cycles

Table 4: Comparison of described TCP effectiveness metrics.

Figure 5: Summary of the general workflow of a mixer.

Model	Existing methods	Justification
P1	FoldFailsOrder ExeTimeOrder RecentnessOrder	We want to prioritize cases based on three criteria: historical failures, execution efficiency, and expected result variability. We thus promote cases that failed in previous executions (FoldFailsOrder), are quick to execute (ExeTimeOrder), and are likely to produce unstable results due to their recent introduction (RecentnessOrder).
P2	FailDensityOrder ExeTimeOrder RecentnessOrder	FailDensityOrder tends to perform well in later testing cycles; however, it is unstable in early cycles, as it relies on the ratio of two estimated averages. We thus want to interpolate it with prioritizers that demonstrate stable performance early on. In our preliminary experiments, ExeTimeOrder and RecentnessOrder both have this quality.
P3	FoldFailsOrder ExeTimeOrder CodeDistOrder	The total strategy of FoldFailsOrder yields large clusters of cases with identical scores, which limits its effectiveness. We hypothesize that breaking these ties is beneficial. Prioritizers that produce distinct scores are best for tiebreaking. Both ExeTimeOrder and CodeDistOrder satisfy this property and are thus well-suited.

Table 5: Existing TCP methods used within the proposed sample models.

Subject program	Count of CI cycles	Failure percentage	Average number of test cases	Total time spent in CI
LittleProxy	427	6.6%	25.6	59 h
HikariCP	1 577	5.3%	14.1	76 h
jade4j	931	9.1%	38.5	52 h
wicket-bootstrap	904	2.7%	42.9	53 h
titan	941	10.4%	41.7	1 215 h
dynjs	935	1.7%	73.3	229 h

Table 6: The description of selected RTPTorrent repositories.

Figure 6: Diagram of various static TCP cycle time measures.

Number	Question	Metrics
RQ1:	Can approach combinators beat their sub-approaches in terms of effectiveness?	—
• RQ1.1:	Can mixers beat their sub-approaches in terms of effectiveness?	$r\text{APFD}_{C}$
• RQ1.2:	Can interpolators beat their sub-approaches in terms of effectiveness?	$r\text{APFD}_{C}$
• RQ1.3:	Can tiebreakers beat their base approaches in terms of effectiveness?	$r\text{APFD}_{C}$
RQ2:	What is the performance of the proposed approaches?	—
• RQ2.1:	What is the effectiveness of the proposed approaches?	$r\text{APFD}_{C}$

Table 7: The summary of stated research questions.

Subject program	RQ1.1: Mixers vs. sub-methods
Subject program	FoldFails	Recent	ExeTime	P1.1	P1.2	P1.3
LittleProxy	.697	.375	.501	.584	.639	.628
HikariCP	.749	.530	.666	.719	.846	.831
jade4j	.661	.805	.657	.652	.857	.681
wicket-bootstrap	.748	.727	.495	.756	.833	.833
titan	.896	.431	.287	.844	.856	.856

Table 8: Average

r\text{APFD}_{C}

per dataset for RQ1.1 and RQ1.2.

Figure 7: Boxplot of average

r\text{APFD}_{C}

values for RQ1.1 and RQ1.2.

Subject program	RQ1.3: ExeTime breaking
Subject program	TotalFails	ExeTime	P3.1
LittleProxy	.678	.501	.716
HikariCP	.583	.666	.596
jade4j	.606	.657	.617
wicket-bootstrap	.744	.495	.898
titan	.888	.287	.899

Table 9: Average

r\text{APFD}_{C}

per dataset for RQ1.3.

Figure 8: Boxplot of average

r\text{APFD}_{C}

values for RQ1.3.

Subject program	RQ2: Baselines
	B1	B2	B3	B4	B5	B6	B7	B8
	( Rothermel et al., 1999 )	( Zhou et al., 2021 )	( Rothermel et al., 1999 )	( Fazlalizadeh et al., 2009 )	( Kim and Porter, 2002 )	—	( Marijan et al., 2013 )
LittleProxy	.453	.424	.558	.470	.659	.694	.584	.426
HikariCP	.570	.462	.446	.527	.744	.775	.793	.535
jade4j	.226	.485	.566	.769	.661	.887	.570	.842
wicket-bootstrap	.009	.529	.514	.741	.892	.747	.855	.783

Table 10: Average

r\text{APFD}_{C}

per dataset for baselines and RQ2.1.

Figure 9: Boxplot of average

r\text{APFD}_{C}

values for baselines and RQ2.1.

Subject program	RQ2.2: Approaches (NTR)
	B3	B5	B7	P1.2	P2	P3.1
	( Rothermel et al., 1999 )	( Kim and Porter, 2002 )	( Marijan et al., 2013 )	ours
LittleProxy	.440	.521	.513	.564	.617	.570
HikariCP	.608	.634	.796	.835	.635	.590
jade4j	.807	.750	.863	.900	.904	.630
wicket-bootstrap	.472	.807	.768	.747	.746	.824

Table 11: Average NTR and ATR per dataset for RQ2.2.

Figure 10: Boxplot of average NTR and ATR values for RQ2.2.

Figure 11: Critical difference plot for RQ2.1

r\text{APFD}_{C}

of different approaches.

Automated Software Testing

Automated software testing uses tools and algorithms to generate, run, and evaluate tests that find defects and prevent regressions.

See all 134 papers

Risk-Aware Batch Testing for Performance Regression Detection
Mar 31, 2026Ali Sayedsalehi, Peter C. Rigby, Gregory MierzwinskiSoftware Engineering Automated Software Testing
Department of Computer Science and Software Engineering, Concordia University, Montreal, Quebec, Canada. · Mozilla, Potton, Quebec, Canada.
CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing
Aug 7, 2026Yan Zhou, Yue Ouyang, Kaiyang Zheng +1Cost-Aware Inference Test-Time Scaling
School of Mathematics and Statistics, Changsha University of Science and Technology, Changsha, China · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.
RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing
Aug 27, 2026Madhusudan Srinivasan, Namith Nishal RaphaeAutomated Software Testing
Department of Computer Science East Carolina University Greenville, USA

Mar 31, 2026J/K move · Enter open · S save

Risk-Aware Batch Testing for Performance Regression Detection

Ali Sayedsalehi, Peter C. Rigby, Gregory Mierzwinski

Department of Computer Science and Software Engineering, Concordia University, Montreal, Quebec, Canada. · Mozilla, Potton, Quebec, Canada.

Software Engineering Automated Software Testing

Open paper AskPDF arXiv

Performance regression testing is essential in large-scale continuous-integration (CI) systems, yet executing full performance suites for every commit is prohibitively expensive. Prior work on performance regression prediction and batch testing has shown independent benefits, but each faces practical limitations: predictive models are rarely integrated into CI decision-making, and conventional batching strategies ignore commit-level heterogeneity. We unify these strands by introducing a risk-aware framework that integrates machine-learned commit risk with adaptive batching. Using Mozilla Firefox as a case study, we construct a production-derived dataset of human-confirmed regressions aligned chronologically with Autoland, and fine-tune ModernBERT, CodeBERT, and LLaMA-3.1 variants to estimate commit-level performance regression risk, achieving up to 0.694 ROC-AUC with CodeBERT. The risk scores drive a family of risk-aware batching strategies, including Risk-Aged Priority Batching and Risk-Adaptive Stream Batching, evaluated through realistic CI simulations. Across thousands of historical Firefox commits, our best overall configuration, Risk-Aged Priority Batching with linear aggregation (RAPB-la), yields a Pareto improvement over Mozilla's production-inspired baseline. RAPB-la reduces total test executions by 32.4%, decreases mean feedback time by 3.8%, maintains mean time-to-culprit at approximately the baseline level, reduces maximum time-to-culprit by 26.2%, and corresponds to an estimated annual infrastructure cost savings of approximately $491K under our cost model. These results demonstrate that risk-aware batch testing can reduce CI resource consumption while improving diagnostic timeliness. To support reproducibility and future research, we release a complete replication package containing all datasets, fine-tuning pipelines, and implementations of our batching algorithms.

Abstract

Automated Software Testing

Explore similar work

Risk-Aware Batch Testing for Performance Regression Detection

CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing

RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing