Effects of Training Data Quality on Classifier Performance

Alan F. Karr; Regina Ruane

cs.LGFeb 25, 2026

Effects of Training Data Quality on Classifier Performance

Authors: Alan F. Karr, Regina Ruane

Organizations: Fraunhofer USA Center Mid-Atlantic, Riverdale MD 20737 · Computational Social Science Laboratory, University of Pennsylvania, Philadelphia, PA 19104

Abstract

We describe extensive numerical experiments assessing and quantifying how classifier performance depends on the quality of the training data, a frequently neglected component of the analysis of classifiers. More specifically, in the scientific context of metagenomic assembly of short DNA reads into "contigs," we examine the effects of degrading the quality of the training data by multiple mechanisms, and for four classifiers -- Bayes classifiers, neural nets, partition models and random forests. We investigate both individual behavior and congruence among the classifiers. We find breakdown-like behavior that holds for all four classifiers, as degradation increases and they move from being mostly correct to only coincidentally correct, because they are wrong in the same way. In the process, a picture of spatial heterogeneity emerges: as the training data move farther from analysis data, classifier decisions degenerate, the boundary becomes less dense, and congruence increases.

Figures & tables

Figure 1: Effect of SNP degradation ( Karr et al.,, 2022 ) on the entropy of 26964 virus genomes.

	Decision
Source	Adeno	COVID	SARS	Sum
Adeno	1580	114	272	1966
COVID	58	1738	200	1996
SARS	253	184	1470	1907
Sum	1891	2036	1942	5869

Table 1: Confusion matrix for the Bayes classifier and undegraded training data

\mathcal{T}

. The correct classification rate is 81.58%.

	Adeno	COVID	SARS	Sum
Adeno	1626	40	300	1966
COVID	51	1724	221	1996
SARS	532	253	1122	1907
Sum	2209	2017	1643	5869

Table 2: Confusion matrix for the neural net classifier and undegraded training data

\mathcal{T}

. The correct classification rate is 76.20%.

	0	1	2
	Adeno	COVID	SARS	Sum
Adeno	1491	93	382	1966
COVID	130	1640	226	1996
SARS	333	256	1318	1907
Sum	1954	1989	1826	5869

Table 3: Confusion matrix for the partition model classifier and undegraded training data

\mathcal{T}

. The correct classification rate is 75.81%.

	0	1	2	Sum
Adeno	1815	62	89	1966
COVID	57	1896	43	1996
SARS	117	114	1676	1907
Sum	1989	2072	1808	5869

Table 4: Confusion matrix for the random forest classifier and undegraded training data

\mathcal{T}

. The correct classification rate is 91.78%.

	Decision
Source	Adeno	COVID	SARS	Sum
Adeno	1566	96	338	2000
COVID	58	1734	208	2000
SARS	253	232	1515	2000
Sum	1877	2062	2061	6000

Figure 2: Confusion matrices for the validation dataset

\mathcal{V}

for the four classifiers trained on the undegraded training data

\mathcal{T}

. Top left: Bayes classifier. Top right : neural net. Bottom left: partition model. Bottom right: random forest

Figure 3: SNP degradation: number of correctly classified elements of $\mathcal{V}$ as a function of SNP_Probability.

Figure 4: SNP degradation: classifier predictions as a function of SNP_Probability. Upper left: Bayes classifier. Upper right: neural net. Lower left: partition model. Lower right: random forest.

Panel of Figure 5 — Figure 5: SNP degradation: Boundary Status distribution as function of SNP_Probability. Upper left: Bayes classifier. Upper right: neural net. Lower left: Partition model. Lower right: random forest. In each of these, $\mbox{BS}=0$ is the green line, $\mbox{BS}=1$ is the yellow line, and $\mbox{BS}=2$ is the red line.

Figure 6: SNP degradation: classifier congruence as a function of SNP_Probability.

Figure 7: SNP degradation: classifier congruence as a function of SNP_Probability for the replicated version of the experiment from Section 4.1 .

Panel of Figure 8 — Figure 8: Length 200 reads: classifier predictions as a function of SNP_Probability. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Figure 9: Length 200 reads: classifier congruence as a function of SNP_Probability.

Panel of Figure 10 — Figure 10: Initially degraded training data: classifier predictions as a function of SNP_Probability. Top: SNP_Probability for initial degradation is 0.9. Bottom: SNP_Probability for initial degradation is 0.5. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 11 — Figure 11: Initially degraded training data: classifier congruence as a function of SNP_Probability. Left: SNP_Probability for initial degradation is 0.9. Right: SNP_Probability for initial degradation is 0.5.

Panel of Figure 12 — Figure 12: Selective SNP degradation: 3-way classifier congruence as a function of Sel_Probability and SNP_Probability. Left: heatmap. Right: 3D surface.

Panel of Figure 13 — Figure 13: SNP degradation not applied to Adeno reads: classifier predictions as a function of SNP_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 14 — Figure 14: SNP degradation not applied to COVID reads: classifier predictions as a function of SNP_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 15 — Figure 15: SNP degradation not applied to SARS reads: classifier predictions as a function of SNP_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 16 — Figure 16: SNP degradation not applied to one set of reads: classifier congruence as a function of SNP_Probability. Left: not applied to Adeno reads in training data. Center: not applied to COVID reads in training data. Right: not applied to SARS reads in training data.

Panel of Figure 17 — Figure 17: SNP degradation applied only to Adeno reads: classifier predictions as a function of SNP_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 18 — Figure 18: SNP degradation applied only to COVID reads: classifier predictions as a function of SNP_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 19 — Figure 19: SNP degradation applied only to SARS reads: classifier predictions as a function of SNP_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 20 — Figure 20: SNP degradation applied only to one set of reads: classifier congruence as a function of SNP_Probability. Left: applied only to Adeno reads in training data. Center: applied only to COVID reads in training data. Right: applied only to SARS reads in training data.

Panel of Figure 21 — Figure 21: Mislabeled training data: classifier predictions as a function of Mislabel_PROBABILITY. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Figure 22: Mislabeled training data: classifier congruence as a function of Mislabel_PROBABILITY.

Panel of Figure 23 — Figure 23: Reversed training data: classifier predictions as a function of the probability of reversing reads in the training dataset $\mathcal{T}$ . Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Figure 24: Reversed training data: classifier congruence as a function of probability of reversing reads in the training dataset $\mathcal{T}$ .

Panel of Figure 25 — Figure 25: Reduced Adeno reads: classifier predictions as a function of Removal_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 26 — Figure 26: Reduced: COVID reads: classifier predictions as a function of Removal_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 27 — Figure 27: Reduced SARS reads: classifier predictions as a function of Removal_Probability. Left: neural net. Center: partition model. Right: random forest. Green: prediction = Adeno. Red: prediction = COVID. Blue: prediction = SARS.

Panel of Figure 28 — Figure 28: Reduced training data reads: classifier congruence as a function of Removal_Probability. Left: reduced Adeno reads. Center: reduced COVID reads. Right: reduced SARS reads.

Add_Probability	Pred_Adeno	Pred_COVID	Pred_SARS	Pred_EColi	Pred_H5N1	Pred_HPV	Pred_Norovirus	Pred_PGingivalis
0.000	2015	2232	1753	0	0	0	0	0
0.001	1945	2192	0	0	0	1863	0	0
0.005	279	2249	0	0	0	867	2605	0
0.010	0	2467	0	0	0	0	1335	2198
0.050	0	2873	0	0	0	0	2923	204
0.100	0	2610	0	0	0	0	3390	0

Figure 29: Superfluous training data: classifier predictions as a function of Add_Probability. Top: neural net. Center: partition model. Bottom: random forest.

Panel of Figure 30 — Figure 30: Superfluous training data: boundary status distributions as a function of Add_Probability. In each of these, $\mbox{BS}=0$ is the green line, $\mbox{BS}=1$ is the yellow line, and $\mbox{BS}=2$ is the red line.

Figure 31: Superfluous training data: classifier congruence as a function of Add_Probability.

Sel_Prob	SNP_Frac	SNP_Prob	NN_PM	NN_RF	PM_RF	NN_PM_RF
0.050	0.050	0.050	4165	4809	4549	3822
0.150	0.050	0.050	4191	4695	4541	3770
0.850	0.050	0.050	2334	2770	2410	1135
0.950	0.050	0.050	2722	3171	2777	1555
0.950	0.150	0.050	2582	3734	2569	1704
0.050	0.850	0.050	4229	4872	4623	3917

Table 5: Mixed degradation: experimental design and congruence among neural net, partition model and random forest.

Response	Predictors				Adjusted $R^{2}$
	Intercept	Sel_Prob	SNP_Frac	SNP_Prob
NN_PM	4305.3 (***)	-1383.1 (***)	336.6	-365.6	0.6403
NN_RF	4818.8 (***)	-1157.4 (***)	344.1	-232.8	0.4468
PM_RF	4670.7 (***)	-1669.5 (***)	515.6	-418.9	0.6794
NN_PM_RF	3964.7 (***)	-1934.0 (***)	576.3 (.)	-462.3	0.6374

Table 6: Estimated coefficients and adjusted

R^{2}

-values for linear models of congruence as a function of Sel_Pobability, SNP_Fraction and SNP_Probability. Labels for statistical significance are the standard ones.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 32: SNP degradation, Bayes classifier: Neighbor Similarity as a function of SNP_Probability.

Figure 33: SNP degradation, neural net classifier: Neighbor Similarity as a function of SNP_Probability.

Figure 34: SNP degradation, partition model classifier: Neighbor Similarity as a function of SNP_Probability.

Figure 35: SNP degradation, random forest classifier: Neighbor Similarity as a function of SNP_Probability.

Training Data Curation

Training data curation improves the quality, diversity, organization, and reliability of datasets used to train machine-learning models.

See all 155 papers

DeMix: Debugging Training Data with Mixed Data Error Types by Investigating Influence Vectors
Jun 10, 2026Jiale Deng, Yanyan Shen, Xiaogang Shi +1Training Data Attribution
Shanghai Jiao Tong University Shanghai, China · Tiktok San Jose, CA, USA
TaxDistill: Improving Metagenomic Taxonomic Annotation via Distilled Genomic Foundation Models
May 22, 2026Rongye Ye, Lun Li, Zheng Luo +3Learning with Noisy Labels Knowledge Distillation
National Genomics Data Center, China National Center for Bioinformation, Beijing 100101, China · Beijing Institute of Genomics, Chinese Academy of Sciences, Beijing 100101, China · University of Chinese Academy of Sciences, Beijing 100049, China +1
Analytical study of the optimal combination of binary classifiers based on classifiers-induced partitioning of the training set
Jul 16, 2026Jean-Marc Brossier, Olivier LafitteEnsemble Learning Binary Classification
CNRS, Univ. Grenoble Alpes, Grenoble-INP, GIPSA-lab, Grenoble, France. · Universit´e Sorbonne Paris Nord, LAGA, UMR 7539. IRL CNRS-CRM 3457. Universit´e de Montr´eal. Canada.

Jun 10, 2026J/K move · Enter open · S save

DeMix: Debugging Training Data with Mixed Data Error Types by Investigating Influence Vectors

Jiale Deng, Yanyan Shen, Xiaogang Shi, Junjun Chai

Shanghai Jiao Tong University Shanghai, China · Tiktok San Jose, CA, USA

Training Data Attribution

Open paper AskPDF arXiv

High-quality training data is essential for the success of machine learning models. However, real-world datasets often contain mixed types of errors arising from systematic flaws in data preparation pipelines, including label errors, feature errors, and spurious correlations. Effective debugging of training data requires both detecting erroneous samples and identifying their specific error types to enable targeted repair, yet existing data cleaning and attribution methods fail to adequately address this dual requirement. In this paper, we propose DeMix, a novel framework that simultaneously diagnoses erroneous samples and their error types. Our key insight is that different error types produce distinct patterns on model behavior. DeMix captures such error-specific patterns by influence vectors that characterize how each training sample affects model predictions across all validation samples. We formulate training data debugging as a multi-label classification problem where a classifier is developed to predict error types directly from influence vectors. We further introduce an intervention-based learning strategy that guides the classifier to capture invariant rationales specific to each error type, ensuring the learned classifier generalizes effectively. Empirical evaluations on 11 tasks across tabular data prediction, recommendation systems, and LLM alignment demonstrate that DeMix significantly outperforms state-of-the-art approaches, achieving a 22.61% improvement in data debugging F1-score and a 9.32% gain in task model performance after data repair. Code is available at: https://github.com/SJTU-DMTai/DeMix.

Abstract

Training Data Curation

Explore similar work

DeMix: Debugging Training Data with Mixed Data Error Types by Investigating Influence Vectors

TaxDistill: Improving Metagenomic Taxonomic Annotation via Distilled Genomic Foundation Models

Analytical study of the optimal combination of binary classifiers based on classifiers-induced partitioning of the training set